sam2 tracker for multiple object using yolo detection prompts in firs…

[adityarauniyar]

Motivation:

• The existing YOLO MOT ML backend integration in Label Studio primarily supports BoT-SORT and ByteTrack. While effective, they fall short in cases of complex occlusions or challenging re-identification scenarios.
• This pipeline improves tracking robustness by combining YOLO's precision in detection with SAM2.1's tracking capabilities.

SAM2.1 Prompting with YOLO11 Detections:

• At the initial frame, SAM2.1 is prompted using YOLO11 detections (bounding boxes).
• SAM2.1 now supports tracking multiple objects of varying categories detected by YOLO.

Tracking Across Frames:

• Tracks objects across frames using SAM2.1's features.
• Known limitation: New objects appearing later (not present in the initial frame) are not tracked, as SAM2.1 is only prompted in the first frame.

YOLO for Detection:

• Utilizes Ultralyitcs' built-in methods for object detection.
• Configurations (e.g., confidence threshold, model variants) are dynamically read from the frontend.

Bounding Box Support for Label Studio:
Outputs bounding boxes for video labeling, aligning with Label Studio's requirements for video annotations.

Implementation Approach:
Class Inheritance:

• Created a new class VideoRectangleWithSam2Tracker, inheriting from VideoRectangle in the YOLO ML backend example.

Overridden predict_region():

• Replaced default bounding box prediction logic with SAM2.1 tracker integration.
• Added support for a custom XML tag attribute (model_sam_tracker) to toggle the SAM2.1-based tracker.