Ax: Question: max number of objectives for multiobjective optimization

[NazyS]

Working on multiobjective opt. I discovered that there is max number of objectives for MultiObjectiveOptimizationConfig() class in ax.core.optimization_config which is defined with a variable MAX_OBJECTIVES = 4. It is used only in a single method of above mentioned class _validate_optimization_config() which is raising an error in case of extra objectives.

I would like to ask why it's there and if there are any particular reasons to restrict a number of objectives for such optimization.

P.S. I tried to overcome such restriction by making custom class inheriting MultiObjectiveOptimizationConfig() and overwriting _validate_optimization_config() method and managed to start optimization loop (having 13 objectives), however after few iterations (3-5) such model just hangs without any error messages. So this max objectives number is there for a reason.
Decreasing my objectives to 3-4 within the same model works great.

[sdaulton]

Thanks for raising this. There is not fundamental reason for this limit. I'll put up a PR to remove it. The results of the optimizing more than 4 objectives become harder to reason about, so if you are open to sharing, I would be curious to hear about your use case and how you evaluate the results.

[NazyS]

Thanks for your reply

We're optimizing the model based on MC generator with up to 8 hyperparameters in order to describe a number of distributions (13 for now). Single loss is just a mean linear/squared relative deviation between each target and our output. With single object optimization (minimization of mean loss among all targets) we obtained sufficient results but also we managed to greatly improve them using MOO.
Optimization with 13 objectives failed, but having only 3 worked great. In this case each objective was a mean loss among a group of single losses picked according to some external factors, however, the effectiveness of such an approach is up for debate and we're open to any suggestion.
Since you say there is no reason to limit a number of objectives, we're going to increase it and see if we'll have any improvements. But then I do not understand why 13 obj. optimization fails after a few successful iterations.

How to evaluate results of such MOO in general is also up for discussion. For now we've only judged results visually: differences in behavior of obtained distributions and their approach to confidence interval were kind of obvious. Just simply looking at a mean loss can distinguish bad and good fit but cannot highlight the best one.
Our target data is, actually, experimental data and hyperparameters have some underlying physical meaning so their obtained values are also taken into account.

[Balandat]

Since you say there is no reason to limit a number of objectives, we're going to increase it and see if we'll have any improvements. But then I do not understand why 13 obj. optimization fails after a few successful iterations.

Well, while there is no reason to put a hard limit on the objectives as there was in code before, there are definitely going to be scalability issues with a large number of objectives. In higher output dimensions the Pareto front can become very complex and contain a ton of points, and the decomposition that is computed to compute hypervolume also blows up. You can use approximate decompositions (which we do), but that may only go so far. Depending on the particular problem that may happen sooner or later. See https://arxiv.org/abs/2006.05078 for some more details on this.

If you really need to work in high-dim output space, then using qParEGO (see also the paper linked above) may be a better approach.

[sdaulton]

+1, I would recommend using qParEGO for 13 objectives instead of qEHVI for the reasons @Balandat listed above.

[eytan]

Yes, so that is as easy as using get_MOO_PAREGO() instead.

On Tue, Dec 15, 2020 at 9:29 AM Sam Daulton [e-mail]
wrote:

+1, I would recommend using qParEGO for 13 objectives instead of qEHVI for
the reasons @Balandat https://github.com/Balandat listed above.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/facebook/Ax/issues/454#issuecomment-745366739, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAW34NNBMEUMO7H7ZRHHX3SU56G5ANCNFSM4UVF5ZXQ
.

[eytan]

@NazyS, I am assuming this resolved your issue. Feel free to re-open if you are still having difficulties.