Detr: Could you pls give a precise number for the training time

[twangnh]

Hi! @fmassa @szagoruyko, We are currently compare to detr on the training cost, so we may need explicit training time, on the paper I found only the message :
Training the baseline model for 300 epochs on 16 V100 GPUs takes 3 days, with 4 images per GPU (hence a total batch size of 64).
I'm wondering if you could provide a detailed training time (in hours) for the 500epoch detr-r50 model and detr-r50-dc5 model? Thanks and looking forward to your reply!

[fmassa]

Hi,

Different models used different configurations / number of GPUs for training, so an exact comparison of time might not be ideal as the cost for communication between nodes depends on the number of machines used, and they might have been trained on slightly different flavors of V100 GPUs (with 16GB or 32GB).
I'm putting here the numbers for the models we released.

With that being said, here are the numbers requested (approximate up to a couple of hours)

detr-r50

• Trained on 32 GPUs
• batch size 2 (per GPU)
• Training + eval time for 500 epochs: 85h

detr-r50-dc5

• Trained on 64 GPUs
• batch size 1 (per GPU)
• Training + eval time for 500 epochs: 99h

I believe I've answered your question, and as such I'm closing this issue, but let us know if you have further questions.

[twangnh]

thank you for sharing the statistics!