Ax: [Question] Documentation for Modelbridge.Factory Models

[nwrim]

I have been looking through the Ax documentation, such as this one to find specifics about the "out-of-shelf" models that Ax supports, such as ax.modelbridge.factory.get_GPEI. I was wondering where I could find some details about these models that Ax offers, such as what Kernel it is using (I am guessing ARD Matérn 5/2 kernels as it is the default in BoTorch, but I was not able to find it stated in Ax). Could you point me to where this documentation is (or is it in preparation)? Sorry for a such trivial question.

Also, I read from few papers published by Facebook that showed great work in Noisy Expected Improvement (and qNEI). Are there any plans in implementing GP+NEI and GP+qNEI as "out-of-shelf" models in Ax like get_GPEI (or is get_GPEI actually using NEI - I read that ax.models.torch.botorch.BotorchModel uses NEI as default acquisition function, but GPEI just said "EI" in documentation and tutorials)?

[qingfeng10]

Hey @nwrim, thanks for your interest! get_GPEI uses all the defaults when constructing the BotorchModel. The defaults can be found in __init__. Therefore, if you want to dig into what GP model, or acquisition function it uses, you can find in ax.models.torch.botorch.botorch_defaults. It uses the same SingleTaskGP as in BoTorch.

To use the model or acquisition function you want, get_botoch is a good choice that you can pass your own acqf_constructor. To handle noisy observations, we use qNEI which is the implementation of the Noisy Expected Improvement.

Let me know if you have other questions!

[nwrim]

Thanks for the response, @qingfeng10! Somehow, the links you posted all show "Facebook Sign in" page. I don't think I can access it. Could you check the URLs again?

[qingfeng10]

I updated the links. Do they work now?

[lena-kashtelyan]

Hi @nwrim, for default model components we use (get_GPEI returns the default setup), you can check specifically:

• which BoTorch surrogate model is used here: https://github.com/facebook/Ax/blob/master/ax/models/torch/botorch_defaults.py#L47,
• which acquisition function is used here: https://github.com/facebook/Ax/blob/master/ax/models/torch/botorch.py#L240 (it's NEI, Noisy Expected Improvement).~~

Edit: the part of my response above is now obsolete since @qingfeng10 updated links above.

---

(or is get_GPEI actually using NEI - I read that ax.models.torch.botorch.BotorchModel uses NEI as default acquisition function, but GPEI just said "EI" in documentation and tutorials)?

Yep it is, as linked above, great catch!

We have some documentation on the models we offer off-the-shelf (which you linked above), which states incorrectly that our default GPEI uses EI and not NEI as you point out. We will make a note to change that!

[nwrim]

Thanks for all the help! The links (both from @qingfeng10 and @lena-kashtelyan) work well!

[zeyueN]

Hey, thanks for the great work and also the links above! Piggybacking on this since it's somewhat related. When will the next release be that includes _get_acquisition_function? The current v0.1.20 seems to have qNEI hardcoded.

And I guess a more general question is that, why is there no user-facing function that makes swapping out acquisition functions easier? It seems all the utilities are already there.

[ldworkin]

@zeyueN , we'll push a new version in the next week or two. Will the addition of that function address your use case, or do you need anything else? What is it you're looking to accomplish? (cc @lena-kashtelyan, sounds like a modular BoTorch use case.)

[zeyueN]

@ldworkin Thanks for the reply. For simple named botorch acquisition functions, yes it partially addresses it. What I wanted to do is a quick way of swapping and testing different acquisition functions. Even with the _get_acquisition_function released, it only allows named botorch acquisition functions; replacing with custom botorch acquisition function is still a bit cumbersome since I have to copy the code in _get_acquisiton_funtion and transform the objective transform myself.

I guess a deeper question is if there's a way for tuning the greediness of the search.

[lena-kashtelyan]

@zeyueN, you might be interested in https://github.com/facebook/Ax/issues/363 –– the functionality discussed in that issue is still in beta-stage but it lets one use a custom BoTorch acquisition function and/or surrogate model (and mix-and-match those subcomponents) as part of one modeling setup in Ax (@ldworkin mentioned this above as 'modular BoTorch' –– our recent effort to allow for easy combining of various BoTorch subcomponents in Ax).

[lena-kashtelyan]

I guess a deeper question is if there's a way for tuning the greediness of the search.

That seems worthy of a separate issue and a bit more detail so folks on the modeling and optimization side can chime in. If you wouldn't mind opening one and explaining what you mean in a bit more detail, that would be great, @zeyueN!

[zeyueN]

Thanks for the pointer @lena-kashtelyan . Went through the tutorial and I think the new features addresses most of my questions.

In terms of greediness tuning, I can see the new GenerationStrategy can sort of do that by switching acquisitions functions midway. I guess I'm imagining a more "dynamic", condition-based, step-switching mechanism (e.g. switch to next step after detecting no improvements for k steps).

I'll provide my feedback for these in a new issue after playing around with them for a bit.

[lena-kashtelyan]

Went through the tutorial and I think the new features addresses most of my questions.

Awesome! Feel free to chime in on #363 if you have any questions; the new setup is still beta and all feedback is very welcome.

In terms of greediness tuning, I can see the new GenerationStrategy can sort of do that by switching acquisitions functions midway. I guess I'm imagining a more "dynamic", condition-based, step-switching mechanism (e.g. switch to next step after detecting no improvements for k steps).

Very interesting to hear this; we actually have been planning to support something like what you describe, a dynamic generation strategy, but we do not yet have a timeline for that project.

I'll provide my feedback for these in a new issue after playing around with them for a bit.

It will be great to hear your thoughts on this, please do!