Pytorch-lightning: non intuitive batch_size in ddp

[jloveric]

Is there a way in PyTorchLightning to set your desired batch size, say 512 and then have the effective batch size per processor (which is normally batch_size*num_gpus) be computed automatically? Right now your effective batch size scales with the number of gpus so these calculations must be computed outside of pytorchlightning (as far as my tests have shown)... This seems like something that PL could/should be able to handle. You'd likely also have to set a maximum processor batch_size so that it could determine an accumulate_grad_batches so as not to use too much memory.

[williamFalcon]

great suggestion!

as a researcher this kind of magic scares me... i'd prefer to be explicit. but would love to hear from the community
@ananthsub @tullie

[awaelchli]

I find the current way more intuitive and super helpful actually.

I can easly choose the right batch size by training initially on only one gpu to figure out which size fills the memory nicely (or use the automatic batch size finder).
Then, if I want to train on X gpus, I can just set gpus=X in the trainer and scale the learning rate roughly by X.