Ray: Reproducibility issue: `ray.init()` messes up `random.seed(SEED)`

[dburov190]

What is the problem?

Simply running ray.init() screws up seeding performed prior to that when using random standard library. Interestingly enough, this does not happen with np.random.

_ray: 0.8.6_
_python: 3.8.3_
_OS: macOS 10.15.4_

Reproduction

Code:

import random
import ray

SEED = 42

random.seed(SEED)
rand1 = random.random()

random.seed(SEED)

ray.init()

rand2 = random.random()

print("rand1:", rand1)
print("rand2:", rand2)

Output:

# ray initialization output...
rand1: 0.6394267984578837
rand2: 0.025010755222666936

[richardliaw]

Hm, interesting - is it typically expected that frameworks do not perturb the global random state?

[dburov190]

I would assume so... this is at least a very non-trivial behavior, and should be warned about?

Edit: the issue is that a typical code development cycle is:
(1) write serial code
(2) make sure everything is reproducible, and the results are as expected
(3) try to make faster by parallelization if needed

So, in (3) after you use some framework to speed up the code, you expect the result to be the same, and this is an unpleasant surprise... (:

[richardliaw]

Absolutely agree; thanks for illuminating that.

[simon-mo]

Thanks for pointing this out.

The reason is that ray uses random internally generate and find available ports open bind to.
https://github.com/ray-project/ray/blob/master/python/ray/services.py#L123-L124

This fix won't be trivial though. It's hard for me to visualize what kind of behavior should Ray promise:

1. random.seed(SEED) is called
2. ray.init is called and spawn multiple worker processes.
3. Should we restore the SEED in the process you called ray.init in?
4. In the same time, should we restore the SEED in all worker processes?

[richardliaw]

I think every Ray import (per process) should hold and use its own
RandomState.

On Sun, Aug 16, 2020 at 9:27 PM Simon Mo [e-mail] wrote:

Thanks for pointing this out.

The reason is that ray uses random internally generate and find available
ports open bind to.

https://github.com/ray-project/ray/blob/master/python/ray/services.py#L123-L124

This fix won't be trivial though. It's hard for me to visualize what kind
of behavior should Ray promise:

1. random.seed(SEED) is called
2. ray.init is called and spawn multiple worker processes.
3. Should we restore the SEED in the process you called ray.init in?
4. In the same time, should we restore the SEED in all worker
   processes?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ray-project/ray/issues/10145#issuecomment-674648275,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABCRZZK7DJU2ZB7YZNIACXTSBCWRXANCNFSM4QBAC7RQ
.

[dburov190]

Thanks for quick responses, guys!

A quick google search tells me that it's impossible to save and restore seeds (or a bad idea), and even though you can save and restore the _state_ (https://stackoverflow.com/a/32172816), it seems the "most correct" way is to have your own random-number generator that does not affect the global state of random.random, numpy.random etc.

So, I guess, something like

import random

def new_port():
    custom_rng = random.Random()
    return custom_rng.randint(10000, 65535)

SEED = 42

random.seed(SEED)
rand1 = random.random()

random.seed(SEED)

port_number = new_port()

rand2 = random.random()

print("rand1:", rand1)
print("rand2:", rand2)
print("port_number:", port_number)

with the output:

rand1: 0.6394267984578837
rand2: 0.6394267984578837
port_number: 39199 # this will be different on each execution

This would completely solve the issue with seeds in the main process (that calls ray.init); I'm not sure what's gonna happen within the worker processes... but that, to me, seems less of an issue since when you wrote your serial code you were not relying on workers anyway (and the parallelizable bits of code are most likely deterministic; if not, like in MCMC, well, it's your job to explicitly select seeds that you want for each worker).