ControlNet & Inpainting task

[msariqs]

Hello, first of all congratulations for your amazing work and thank you for sharing it ! :)

I am interested in the MAISI Inpainting Task you mentioned in your paper (https://arxiv.org/html/2409.11169v2). Do you plan to share the weights ? Or do you plan to share the code/method to train / infer with the ControlNet for this specific task ? I don't fully understand how inpainting can be done using the ControlNet in terms of I/O, prepreocessig etc...

Thanks in advance,
Malik

[KumoLiu]

Hi @msariqs, did you refer to the tutorial and the bundle?
https://github.com/Project-MONAI/tutorials/tree/main/generation/maisi
https://github.com/Project-MONAI/model-zoo/tree/dev/models/maisi_ct_generative

[msariqs]

Hi @KumoLiu, yes I refer to this work

[msariqs]

Any update on this issue ?

[aylward]

The tumor in-painted demo discussed in the paper is demonstrated in this jupyter notebook:
https://github.com/Project-MONAI/tutorials/blob/main/generation/maisi/maisi_inference_tutorial.ipynb

[msariqs]

This notebook demonstrates how we can use MAISI to generate a pair of mask and image. Correct me if I'm wrong but it doesn't demonstrate the task of tumor inpainting (i.e. adding a tumor to an existing image).

[KumoLiu]

Hi @guopengf, could you please share more details here? Thanks in advance!

[guopengf]

Hi, we have not released the inpainting model, but we are considering including it in a future release. For this task, we follow the setup of Difftumor (https://arxiv.org/pdf/2402.19470v1). The input to ControlNet consists of the masked image and the corresponding tumor mask.