Detr: 500 epoch replication for Resnet 50

[gaopengcuhk]

I replicate the Resnet 50 with 2 img per GPU, 8 GPU. However, my learning curve is quite different from yours.

[gaopengcuhk]

When I train models with 500 epochs, I observe overfitting.

[gaopengcuhk]

python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py \
--lr_drop 400 --epochs 500 \
--coco_path /path/to/coco

[szagoruyko]

looks strange, I will try to replicate

[gaopengcuhk]

Any progress on this? @szagoruyko

[szagoruyko]

150 epochs done, still 8 days to go

[szagoruyko]

@gaopengcuhk below are training logs for R50 DETR on 4 nodes (blue) vs 1 node (orange), there is indeed overfitting, but the model reached 42/62 AP - close to 4 node training. Not sure what's going on, perhaps need to adjust hyperparameters like learning rate or grad clip value, will investigate if we have bandwidth

[linlililin123456]

@szagoruyko may i ask the batch size of 4 node training, is this log https://dl.fbaipublicfiles.com/detr/logs/detr-r50_log.txt tranined with 4 node of total batch size 64?