WorkNCCL(SeqNum=5125, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807734 millisecond

[hudingding]

训练环境：双卡A100，torch版本 2.0.1
参数：batch-size = 4， base-size = 1536*1536

[hudingding]

Epoch: [0] [ 0/50108] eta: 1 day, 20:54:44 lr: 0.000000 loss: 63.1246 (63.1246) loss_bbox: 1.3422 (1.3422) loss_bbox_aux_0: 1.4365 (1.4365) loss_bbox_aux_1: 1.4062 (1.4062) loss_bbox_aux_2: 1.3247 (1.3247) loss_bbox_aux_3: 1.3386 (1.3386) loss_bbox_aux_4: 1.3422 (1.3422) loss_bbox_dn_0: 1.2075 (1.2075) loss_bbox_dn_1: 0.8915 (0.8915) loss_bbox_dn_2: 0.6473 (0.6473) loss_bbox_dn_3: 0.5657 (0.5657) loss_bbox_dn_4: 0.5573 (0.5573) loss_bbox_dn_5: 0.5573 (0.5573) loss_bbox_dn_pre: 0.8235 (0.8235) loss_bbox_enc_0: 1.1189 (1.1189) loss_bbox_pre: 1.2835 (1.2835) loss_ddf_aux_0: 0.0767 (0.0767) loss_ddf_aux_1: 0.0335 (0.0335) loss_ddf_aux_2: 0.0059 (0.0059) loss_ddf_aux_3: 0.0008 (0.0008) loss_ddf_aux_4: 0.0000 (0.0000) loss_ddf_dn_0: 0.5174 (0.5174) loss_ddf_dn_1: 0.1419 (0.1419) loss_ddf_dn_2: 0.0319 (0.0319) loss_ddf_dn_3: 0.0040 (0.0040) loss_ddf_dn_4: 0.0000 (0.0000) loss_ddf_dn_5: 0.0000 (0.0000) loss_fgl: 0.0078 (0.0078) loss_fgl_aux_0: 0.0058 (0.0058) loss_fgl_aux_1: 0.0063 (0.0063) loss_fgl_aux_2: 0.0080 (0.0080) loss_fgl_aux_3: 0.0078 (0.0078) loss_fgl_aux_4: 0.0078 (0.0078) loss_fgl_dn_0: 0.7864 (0.7864) loss_fgl_dn_1: 0.8262 (0.8262) loss_fgl_dn_2: 0.9184 (0.9184) loss_fgl_dn_3: 0.9502 (0.9502) loss_fgl_dn_4: 0.9526 (0.9526) loss_fgl_dn_5: 0.9526 (0.9526) loss_giou: 3.3034 (3.3034) loss_giou_aux_0: 3.3141 (3.3141) loss_giou_aux_1: 3.2965 (3.2965) loss_giou_aux_2: 3.2983 (3.2983) loss_giou_aux_3: 3.3022 (3.3022) loss_giou_aux_4: 3.3034 (3.3034) loss_giou_dn_0: 1.7992 (1.7992) loss_giou_dn_1: 1.8009 (1.8009) loss_giou_dn_2: 1.7354 (1.7354) loss_giou_dn_3: 1.6483 (1.6483) loss_giou_dn_4: 1.6331 (1.6331) loss_giou_dn_5: 1.6331 (1.6331) loss_giou_dn_pre: 1.7120 (1.7120) loss_giou_enc_0: 3.4042 (3.4042) loss_giou_pre: 3.3482 (3.3482) loss_vfl: 0.0023 (0.0023) loss_vfl_aux_0: 0.0004 (0.0004) loss_vfl_aux_1: 0.0005 (0.0005) loss_vfl_aux_2: 0.0011 (0.0011) loss_vfl_aux_3: 0.0004 (0.0004) loss_vfl_aux_4: 0.0011 (0.0011) loss_vfl_dn_0: 0.3264 (0.3264) loss_vfl_dn_1: 0.2465 (0.2465) loss_vfl_dn_2: 0.3514 (0.3514) loss_vfl_dn_3: 0.4320 (0.4320) loss_vfl_dn_4: 0.3920 (0.3920) loss_vfl_dn_5: 0.3892 (0.3892) loss_vfl_dn_pre: 0.3577 (0.3577) loss_vfl_enc_0: 0.0037 (0.0037) loss_vfl_pre: 0.0028 (0.0028) time: 3.2267 data: 2.1058 max mem: 8541
[E ProcessGroupNCCL.cpp:828] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=5125, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802732 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:828] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=5125, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802739 milliseconds before timing out.