About SAM2 Prompt

[ShuoShenDe]

Dear IDEA-Research Team,

It is not strictly correct to say that SAM2 only supports points as prompts. Based on my review of the SAM2 source code yesterday, it also supports masks as prompts. To verify this, I created Grounded-SAM2-Tracking (https://github.com/ShuoShenDe/Grounded-Sam2-Tracking.git) as an example, which validated the feasibility of using grounded-sam plus SAM2 for video tracking. Additionally, I have implemented continuous segmentation and tracking using Grounded-Segment-Anything and SAM2, ensuring the stability of object IDs.

As the code for this project is still being modified, do not hesitate to contact me if you have any concerns or feedback.

Best regards,
Shuo Shen

[rentainhe]

Dear Shuo Shen:

Thank you very much for correcting my mistakes. We previously overlooked SAM's video segmentation with its ability to support mask prompts. I have just carefully read your code, and I must say it's very well-implemented. It includes segmentation, determining new object IDs across all segment images (ensuring consistent IDs), as well as data dumping and visualization. Everything is done very nicely. If you don't mind, would it be possible for us to reference your code in our README? We would like to sincerely invite you to collaborate with us and see if you have time to implement a similar idea in this repository. If your schedule doesn't allow it, we would also like to update our code following a similar approach in the future.

Best regards,

Tianhe Ren

[ShuoShenDe]

Dear Tianhe Ren,

Thank you for your kind words and the invitation to collaborate. I am honored that you wish to reference my code in your README. Please feel free to do so.

I would be delighted to contribute as a collaborator and add to this project. For further details on our collaboration, please feel free to reach out to me via email at [email protected].

Best regards,
Shuo Shen

[rentainhe]

Dear Shuo Shen

Thank you very much for your willingness to collaborate with us. We would be delighted to have you merge your code implementation into this repository. Feel free to submit a pull request if you have the time, and we will merge your code as soon as possible.

Best regards,

Tianhe Ren

[ronghanghu]

Hi @rentainhe, thanks for the great work! Chiming in to provide more details regarding SAM 2 prompting:

Regarding mask prompts in SAM 2:

• SAM 2 also supports using masks as a prompt -- We have an interface add_new_mask for it in the video predictor class SAM2VideoPredictor

Regarding box prompts in SAM 2:

• SAM 2 also supports using bounding box as a prompt -- We just added an example in the video predictor notebook in Add interface for box prompt in SAM 2 video predictor facebookresearch/sam2#174 to provide a box prompt example.

Maybe we could directly use the box prompt or mask prompt instead of Uniform Point Sampling in this case?