Xla: Proper way to create copies of models for xmp.spawn?

[hrbigelow]

❓ Questions and Help

I notice from here that the MNIST() constructor is called once per spawned process. This seems to imply that, if you were creating these models using random weight initialization, then care must be taken to actually get identical models.

What is the recommended practice? Do people just use a copy.deepcopy, or is there something more "torch"-like?

Thanks!

[jysohn23]

Typically we set the seed like we did here another example here, which is enough to guarantee same initialization of weight as long as its done close to actual initialization. If you start branching differently between processes from the time of setting the seed and initializing the weights, the weights may be different.

[hrbigelow]

Ahh I see, I missed that. Thanks very much Jin Young!

[jysohn23]

Feel free to open if you have other questions, but closing for now.