Yet-another-efficientdet-pytorch: How to improve the performance of the DataLoader

[mnslarcher]

Hi @zylo117,

Debugging my code on a toy dataset composed by a subset of my real dataset (high resolution images like 4kx4k) I notice this:

[05/20 15:24:26] eta: 0:05:18 - epoch: 5/20 - batch: 6/13 - loss_cls: 0.1730 - loss_box_reg: 0.0546 - total_loss: 0.2276 - time: 0.5427 - data_time: 0.0067 - lr: 0.009203 - max_mem: 9540M
[05/20 15:24:26] eta: 0:05:18 - epoch: 5/20 - batch: 7/13 - loss_cls: 0.1894 - loss_box_reg: 0.0524 - total_loss: 0.2418 - time: 0.6647 - data_time: 0.0093 - lr: 0.009168 - max_mem: 9540M
[05/20 15:24:27] eta: 0:05:19 - epoch: 5/20 - batch: 8/13 - loss_cls: 0.2114 - loss_box_reg: 0.0525 - total_loss: 0.2639 - time: 0.6349 - data_time: 0.0122 - lr: 0.009133 - max_mem: 9540M
[05/20 15:24:40] eta: 0:05:32 - epoch: 5/20 - batch: 9/13 - loss_cls: 0.2108 - loss_box_reg: 0.0540 - total_loss: 0.2648 - time: 12.9244 - data_time: 12.4863 - lr: 0.009097 - max_mem: 9540M
[05/20 15:24:40] eta: 0:05:32 - epoch: 5/20 - batch: 10/13 - loss_cls: 0.1715 - loss_box_reg: 0.0498 - total_loss: 0.2213 - time: 0.4807 - data_time: 0.0067 - lr: 0.009060 - max_mem: 9540M
[05/20 15:24:41] eta: 0:05:33 - epoch: 5/20 - batch: 11/13 - loss_cls: 0.3043 - loss_box_reg: 0.0517 - total_loss: 0.3561 - time: 0.5490 - data_time: 0.1081 - lr: 0.009023 - max_mem: 9540M
[05/20 15:24:42] eta: 0:05:34 - epoch: 5/20 - batch: 12/13 - loss_cls: 0.3021 - loss_box_reg: 0.0718 - total_loss: 0.3740 - time: 1.2406 - data_time: 0.8068 - lr: 0.008984 - max_mem: 9540M
[05/20 15:24:53] eta: 0:05:45 - epoch: 5/20 - batch: 13/13 - loss_cls: 0.1586 - loss_box_reg: 0.0419 - total_loss: 0.2004 - time: 10.9414 - data_time: 10.5151 - lr: 0.008946 - max_mem: 9540M
[05/20 15:25:07] val. epoch: 5/20 - val_loss_cls: 0.5549 - val_loss_box_reg: 0.0634 - total_val_loss: 0.6183 - time: 3.4075 - data_time 3.2518
[05/20 15:25:21] eta: 0:06:13 - epoch: 6/20 - batch: 1/13 - loss_cls: 0.3767 - loss_box_reg: 0.0670 - total_loss: 0.4437 - time: 13.5906 - data_time: 12.9661 - lr: 0.008906 - max_mem: 9540M
[05/20 15:25:21] eta: 0:06:13 - epoch: 6/20 - batch: 2/13 - loss_cls: 0.1587 - loss_box_reg: 0.0377 - total_loss: 0.1964 - time: 0.7764 - data_time: 0.3319 - lr: 0.008866 - max_mem: 9540M
[05/20 15:25:22] eta: 0:06:14 - epoch: 6/20 - batch: 3/13 - loss_cls: 0.1561 - loss_box_reg: 0.0451 - total_loss: 0.2012 - time: 0.4491 - data_time: 0.0067 - lr: 0.008826 - max_mem: 9540M
[05/20 15:25:22] eta: 0:06:14 - epoch: 6/20 - batch: 4/13 - loss_cls: 0.1431 - loss_box_reg: 0.0495 - total_loss: 0.1925 - time: 0.4902 - data_time: 0.0066 - lr: 0.008784 - max_mem: 9540M
[05/20 15:25:36] eta: 0:06:28 - epoch: 6/20 - batch: 5/13 - loss_cls: 0.1278 - loss_box_reg: 0.0353 - total_loss: 0.1631 - time: 13.5882 - data_time: 13.0189 - lr: 0.008743 - max_mem: 9540M
[05/20 15:25:37] eta: 0:06:29 - epoch: 6/20 - batch: 6/13 - loss_cls: 0.1124 - loss_box_reg: 0.0310 - total_loss: 0.1434 - time: 0.6269 - data_time: 0.0094 - lr: 0.008700 - max_mem: 9540M

As you can see, more or less every four iterations there is one iteration that takes a lot of time. Most of time is time to get the next batch from the data loader (time = iteration time, data_time = time before and after a next(data_loader_iter)).

Do you have any suggestion about how to improve this?

I remember that probably there is a way to prefetch the data or maybe there is some other smart solution that I can try.

Thanks,
M

[zylo117]

It may has something to do with caching. If your system or HDD/ SSD/ NAS has some built-in caching mechanism, the io speed may vary.
Can you replicate this on other servers/ environments?
Try setting num_workers to 0 and see if the time per batch is stable. If not, it should be related to filesystem.

[zylo117]

I'm suffering similar issue. The training stuck when epoch starts, and takes some time to start training again.
Then I monitored the disk IO and find out the IO speed is almost 0, but the CPU usage is maxed.
So if it's not caching, it's multiprocess overhead. Stupid dataloader respawns processes everytime it runs out of data and tries to reload data.
When I set num_workers to 0, this problem goes away, so it should be related to multiprocessing of dataloader.

[zylo117]

Maybe reimplement a dataloader with multithread instead of multiprocess would help

[zylo117]

For the time being, I think finding the balance among num_workers, batchsize and speed would be helpful.

[mnslarcher]

Thanks @zylo117, as always you give me great insights! I’ll try to continue your analysis and see if there is a solution to this and if just tuning num_workers reduce the problem. I’ll keep you updated if I’ll find something useful