Getting all NAN masks when training on custom data

[sankarip]

I'm tuning tuning the obj365 pth with my custom data using the command torchrun --master_port=7777 --nproc_per_node=1 train.py -c configs/dfine/custom/objects365/dfine_hgnetv2_s_obj2custom.yml --use-amp --seed=0 -t dfine_s_obj365.pth

I'm getting an error: assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
This comes from the masks being all NAN. Any ideas on how to fix this? I've tried lowering learning rate, but that had no effect.

[DmitriiSavin]

Hi!
I have experienced a similar issue after a couple of training epoches.
Removing --use-amp from the command fixed it

#43