Detr: Extremely slow multi-node training due to clip_max_norm

[YiF-Zhang]

First thanks for the excellent work! I have been using torch.distributed.launch to lauch training on a two-node cluster, each with 8 GPUs, and I found the training is extremely slow (~7x slower than single-node setting). It's mainly caused by clip_max_norm. The environment is as follows:

• PyTorch 1.3.0

• CUDA 10.2

I understand that it is supposed to train the model on a Slurm cluster using PyTorch later than 1.5.0, but I have no access to both of them. I wonder whether there is a solution for this problem using the vanilla torch.distributed.launch launcher and PyTorch 1.3.0, and does these two factors actually affect the training time?

Best regards.

[YiF-Zhang]

Update: after disabling the gradient clip, now it's the optimizer.step() taking most of the iteration time (~1.6s per-iter). Can you please provide the iteration time in your multi-node experients under Slurm cluster?

[szagoruyko]

hi @YiF-Zhang looks like you are timing CPU kernel caalls, CUDA calls are asynchronous so timing needs to be done with sync points. Can you try adding torch.cuda.synchronize() before timers and check https://pytorch.org/tutorials/recipes/recipes/profiler.html