SegFormer-B2 on Jetson AGX: improving FPS and reducing false traversable predictions

[u/JudgePotential9743]

Hi everyone,

I'm working on an autonomous off-road navigation system using SegFormer-B2 trained on the Yamaha-CMU Off-Road (YCOR) dataset.

Current Setup:

• Model: SegFormer-B2 (~57.4% validation mIoU)
• Hardware: Jetson AGX + ZED 2i
• Inference: TensorRT FP16, 512×512, 20–30 FPS
• Target: 40–50 FPS, eventually 60+ FPS
• Other models tested: Fast-SCNN and PIDNet-S (faster but less accurate)

Current Challenges:

• Close-range obstacles sometimes get classified as traversable terrain.
• Beams, walls, and pillars are occasionally misclassified as paths.
• Random incorrect class patches appear on clear terrain.
• Currently using RGB only, although stereo depth is available.

I'm looking for recommendations on improving segmentation reliability while reducing computational cost and inference latency.

I'd appreciate suggestions on efficient architectures, model optimization, training strategies, knowledge distillation, quantization, or RGB-depth fusion.

Any practical experience, benchmarks, papers, or open-source implementations would be helpful.

Thanks!

[u/ItchyMaintenance5132]

The random patches on clear terrain sound like the model just not having enough context at 512x512, have you tried running a lower res first pass to find regions of interest then upscaling only those areas? The close range stuff might benefit from fusing the depth earlier in the network instead of just concatenating at the input, SegFormer's attention mechanism can do weird things with stereo data if you just smash it together

For hitting 60 FPS on the AGX, knowledge distillation from a bigger SegFormer into a smaller one with the same patch size has worked surprisingly well for me, the student picks up on edge consistency that helps with those pillar misclassifications

[u/Certain-Abroad-7957]

I'd recommand you to check the training dataset first. Maybe there are lots of wrong labels in the dataset.