Learning rate value causes Infinite/nan loss value

[seo-95]

Hi,
first of all, thank you for sharing the code to reproduce your results. I was trying to train the model from scratch on VG using your configuration R-50-updn.yaml and I have found a potential problem. With the learning rate of 0.02 you have in your configs (both base grids and base bottom-up) the training script produces infinite/nan loss values causing the training to diverge.
The training works after decreasing the learning rate to 0.002.
I was wondering if there is something I am missing.
Thank you!

[endernewton]

Have you retried after it failed? sometimes the randomness can cause that.