Skip to content

Fix YOLOv7 bounding-box restoration after anisotropic resize - #16

Open
LAlongago wants to merge 1 commit into
KTH-RPL:mainfrom
LAlongago:fix/yolov7-box-coordinate-restoration
Open

LAlongago wants to merge 1 commit into
KTH-RPL:mainfrom
LAlongago:fix/yolov7-box-coordinate-restoration

Conversation

@LAlongago

Copy link
Copy Markdown

The detector first resizes the image to (image_size, int(image_size * 0.7)), then applies letterboxing. Its previous call to scale_coords(..., orig_shape) assumed an aspect-preserving resize directly from the original image. That inverse does not match the actual preprocessing, so boxes can shift sideways or collapse at the image boundary before being passed to SAM.

For a 640 × 640 observation and the default image size, preprocessing produces a 640 × 448 image with no letterbox padding. A recorded chair detection [499.5, 218.625, 554.5, 288.0] was restored to [576, 312, 639, 411]; the correct result is [500, 312, 554, 411].

This change restores coordinates in two stages: undo letterboxing with its returned ratio/padding, then undo the preceding resize with independent width/height factors. Existing rounding, clipping and filtering remain in place. Model input pixels, weights, confidence/NMS settings and navigation logic are unchanged.

Validation:

  • Added 9 CPU regression tests exercising the real detect() preprocessing and restoration with mocked inference output/device transfer. They cover actual failing detections, image-edge collapse, landscape/portrait/odd-sized images, letterbox padding, clipping and empty output. The original implementation fails 7/9; the fix passes 9/9.
  • Command: python -m unittest discover -s tests -p 'test_yolov7_coordinates.py' -v.
  • Reran the first three official HM3D multi-object validation episodes with seed 0, the original PyTorch backend and the same environment compatibility changes used for the baseline:
Episode (zero-based) / target Before After
0 / toilet Success, 622 observations Success, 296 observations
0 / plant Timeout, 1500 observations Misdetect, 39 observations
1 / chair Misdetect, 104 observations Misdetect, 104 observations
2 / TV Success, 85 observations Success, 80 observations
2 / plant Misdetect, 520 observations Misdetect, 376 observations

Overall progress remains 2/9 planned targets and 0/3 complete sequences; this small slice does not demonstrate improved navigation success. Later targets can start from different poses after the preceding target's trajectory changes.

A full mapper replay of episode 1 isolates the geometric improvement: at observation index 102, the corrected SAM prompt yields 2663 chair pixels out of a 2692-pixel mask (98.9%), compared with 0 chair pixels in the original 2134-pixel mask. It still fails because the new candidate's adjusted similarity (1.07924) is below the existing replacement threshold (1.10323), so the old candidate is retained and the distance-based stop fires. Candidate replacement and stopping are separate concerns and are unchanged in this PR.

Only the detector coordinate restoration and its regression tests are included.

Undo letterboxing using its recorded ratio and padding, then scale x and y
independently back to the original camera image. Preserve inference pixels
and final clipping. Add CPU regression cases for square, non-square, padded,
edge and empty detections.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant