Mlpack: Synchronized Batch Normalization implementation

[mrityunjay-tripathi]

Hello, everyone
I think the synchronized batch norm layer is not already implemented. Right? I can continue with the implementation. If I am not wrong I have to put it in src/mlpack/methods/ann/layers folder?
The reference to the paper is here.

[kartikdutt18]

Hi @mrityunjay-tripathi, I think implementing it would be okay however I thought that synchronized Batch Normalization had main use with GPUs (for parallel training) and currently mlpack doesn't support gpu. I might be wrong so could you correct me if I'm wrong.
However a member of this organization would provide better organization.
Thanks.

[zoq]

At this point mlpack does not support multi-node training, so I think we don't have actual use for the layer. Let me know what you think.

[mrityunjay-tripathi]

Agree. On further reading the paper I come to know that it helps in speeding up the multi-network models. But still, I can't come to the conclusion that whether the sync batch norm will perform better in terms of speed than the default batch norm using parallel computing on CPU. Correct me if I got something wrong.

And whether mlpack plans to provide multi-node training support in the future?

[zoq]

At some point I wrote a implementation for Downpour SGD: https://github.com/mlpack/mlpack/pull/1117 but this is just one part. I'm not sure if we would see a runtime improvement for sync batch norm on the CPU, if you like you could test it out, but perhaps there are other layer that you are interested in as well.

[mrityunjay-tripathi]

Ok, I will test it side by side while working on other things in mlpack. You may or may not close this issue for now, as you wish.

[zoq]

Let's close this for now, we can reopen it later if anything comes up.