Ax: [Question] Is there a way to set a random state for reproducibility?

[Irynei]

Hi!

I didn't find a possibility to set random seed in the docs, maybe I missed something.
Can you please tell if there is a way how to make results reproducible? Thanks.

[kkashin]

@Irynei - what level are you looking for reproducibility at? The whole optimization loop or a specific generator?

You can set the seed for (quasi)-random generators, such as the Sobol generator (see https://ax.dev/api/models.html#ax.models.random.sobol.SobolGenerator.seed).

[Irynei]

I was looking for the whole optimization loop, in the optimize method, ideally.

As far as I understand a (quasi)-random generator is chosen somehow under the hood?
So I can only set seed when I am using something similar to what explained here https://ax.dev/tutorials/gpei_hartmann_developer.html, right?

[bletham]

In that example (https://ax.dev/tutorials/gpei_hartmann_developer.html) there are two main sources of randomness: The scrambling in the Sobol generator used for the initialization, and the randomness in the (quasi) Monte Carlo approximation used in the acquisition function.

Like @kkashin says, the first can easily be handled by making the change to

sobol = Models.SOBOL(search_space=exp.search_space, seed=1000)

And yes, this will require using the more explicit API rather than optimize, whose purpose is to do a bunch of stuff under the hood for you.

Most of the randomness will come from that (the initialization). The actually Bayesian optimization phase will still have some randomness in that the acquisition function is evaluated with a Monte Carlo approximation. The seed for that is set here:

https://github.com/facebook/Ax/blob/eebb8c7b13ab7a28502d3f00459328c52057a2eb/ax/models/torch/botorch_defaults.py#L162

So, you can just run

torch.manual_seed(1000)

before every call to gpei.gen.

As a side note, for the purposes of benchmarking / performance evaluation I'd recommend setting the number of reps high enough that the seed doesn't matter. But of course there are other settings where reproducibility of a single run would be useful, and this should give that.

[Irynei]

@bletham, thank you for a great explanation! It's clear to me now.

[kkashin]

I'm going to close out this issue for now, given that you're unblocked. We will look into making it possible to pass a seed directly to the optimize function so that reproducibility is easier to achieve in the future, and will follow up with developments.