Refining masks after propagation?

[Zarxrax]

I was wondering if anyone has had success with refining masks by adding additional points after running propagation?
In my attempts, it is not working well, if at all.

While the video_predictor_example notebook provides an example of how it should work, its difficult to see what its actually doing since the additional refining point is on such a small area and it gets almost fully covered by the star. But when I change the model used in the example to the tiny model, it's easier to see that nothing is really happening.

Putting a negative point on frame 150:

After running propagation again, there is no effect on frame 150 from the negative prompt:

[sabry9999]

I have the same issue. I'm using the video predictor notebook on Colab, and after applying a refinement on a specific frame, it doesn't show the refinement when I run the mask propagation.

[heyoeyo]

While the refinement capability is a neat idea, it seems it's always going to be unpredictable, since you're sort of asking the model to give you the 'average' of multiple prompts, and it's not always clear what that should be.

So for predictable results, it might better to 'restart' the tracking on the frame that needs refinement, using a prompt that captures the object in full (as opposed to partial prompts like the example, that don't fully specify what should be segmented).

Restarting might be a bit annoying to implement, but you can mimic the behavior by setting the max_cond_frames_in_attn to 1 (this can be added to the yaml config if you don't want to modify the code), which tells the model to only use the most recent prompt (instead of trying to combine them all).
For some reason the code includes an assertion to force this setting to be at least 2 prompts. You can comment that line out (without any obvious consequence) or otherwise set it to 2 and then double-add the 'refinement' prompt to get the same effect.