Hydra: Specify the schema for returning the result of optimization.

[shagunsodhani]

We want to have a schema, that all HPO plugins can use, for returning the result of optimization.

Related to #430

[jrapin]

Following hydra example and simplified for my use case, the file currently looks like this with nevergrad:

nevergrad:
  batch_size: 4
  db: mnist
  dropout: 0.381
  lr: 0.094

optimizer: nevergrad

I'm fine with @omry proposition:

@dataclass
class SweeperResult
  name : str = MISSING
  best_parameters: Dict[str, Any] = MISSING # optimal parameters found
  best_achieved_result: float = MISSING     # optimal return value obtained
  sweeper_cfg : Any = MISSING               # The hydra.sweeper.params node

I would tend to rename best_parameters to recommendation because that fits better with nevergrad, but anything will do.

A small detail though: sometimes nevergrad (probably the same with botorch?) recommendation has never been evaluated, so the best achieved result may not be available, or at least not be directly related to the recommendation. Another option is to always recommend an evaluated set of parameters. I think this seems more logical in this use case so I would fo for it, but that would however require some more lines of codes (nothing dramatic though :D)

[omry]

Thanks @jrapin.
We have been running into those exact considerations with Ax and decided for now to ignore the predicted result and just print and store the result of the best evaluation.

With Nevergrad, do you ever see a significant improvement between the best evaluation and the un-evalulated final recommendation (Or in other words : the best un-evaluated model prediction)?

[jrapin]

I know that my colleague Olivier works on such cases :D, this happens mostly on noisy functions, the mean of the best points in the optimizer population of points is usually a much better estimate of the solution. I doubt it would apply to cases that are relevant for hydra.

As long as we can eventually add another field I don't think we need to care for that, most users will probably want to see the best run only, if some need more information we can deal with this later

[omry]

No cases are relevant to Hydra, it's general purpose.
In fact RL training, which often uses Hydra is very noisy.

I think we can already add both predicted best numbers and predicted best result.

[jrapin]

I would tend to go for something along those lines (slightly different names and structure), but anything should work for me:

name  # eg: nevergrad?
best_evaluation
   value: xxx
   parameters:
      xxxx
      xxxx
recommendation
   xxxx
   xxxx
sweeper_config  # not sure what that is supposed to be

[omry]

sweeper_config is the hydra.sweeper node (or maybe just hydra.sweeper.params).
The intention here is to retain optimization parameters.
I don't mind skipping it if you don't think it's useful.

[jrapin]

The intention here is to retain optimization parameters.
I don't mind skipping it if you don't think it's useful.

Isn't it already copied somewhere?

[omry]

Yes, it's in hydra.yaml. you are right, not much of a point in duplicating it.

[shagunsodhani]

What is the difference between best_evaluation and recommendation?

Building on @jrapin 's suggestion,

name  # eg: nevergrad?
nevergrad (or can be called `result` etc) 
    best_params
        param: value
        .....
    best_result
        ....

The additional information that Ax/Nevergrad wants to return can be nested within the Ax/Nevergrad namespace. The word best_params etc can be set to something else.

[jrapin]

@shagunsodhani your exemple is difficult to read, are the indents missing ? :D

What is the difference between best_evaluation and recommendation?

best evaluation is the best job that has been evaluated, so a set of parameters and the corresponding value. Recommendation would be the recommended best set of parameters, which may not have been evaluated. As mentioned above this can happen for noisy settings for some algorithms, but the use cases are pretty slim.

[shagunsodhani]

So then nevergrad could use recommendation to populate the best_params field (they can be named something else as well). And the additional information that Ax/Nevergrad wants to return can be nested within the Ax/Nevergrad namespace. Does that sound good?

[omry]

If we think that information is not generally applicable to all hyper parameter optimization packages then yes.

[shagunsodhani]

Does this sound good to all?

name:  # eg: nevergrad?
nevergrad:
    best_params:
        param: value
        .....
    <Any thing else that nevergrad wants to specify>
        ....

[omry]

no.
keep common things outside of the specific namespace. it's just going to be a pain to access them this way.

[jrapin]

So then nevergrad could use recommendation to populate the best_params field (they can be named something else as well). And the additional information that Ax/Nevergrad wants to return can be nested within the Ax/Nevergrad namespace. Does that sound good?

The issue with the recommendation is that it has not been necessarily evaluated during the optimization, so we have no value to provide for it. And providing the best value for the set of best parameters seems useful (there seem to be no place for in in your configuration currently)

Also, why encapsulate everything in the optimizer namespace? If the aim is to have something generic, that seems to add some unnecessary complexity: when you want to recover the parameters you would need first to recover the name of the sweeper... while it does not really matter in the end.

I think we should agree on necessary base fields that need to be filled in a certain way, then all plugins can add additional base fields if they have additional information. Does that make sense?

[omry]

we can easily add a standard configuration parameter evaluate_recommendation: False that will enable the last evaluation for users that wants it.
In this case, we can have mandatory recommendation in the file, as well as an optional different field for the evaluation.

optimization:
  name: nevergrad # for information purposes only
  recommended_params:
    x: 0.1
    y: 0.2
    z: foo 

  predicted_output: 0.001

  # we can intentionally keep this here as null to help users discover
  # how to turn it on by looking for evaluated_output in the docs.
  evaluated_output: null

(Above is just a suggestion, I don't know if those are good names).

[jrapin]

I'm really not in favor of having a mandatory recommendation field that is a corner case unexpected by the users, and probably useful for 0.1% of them, if not less.

What seems expected is the best job that was performed and it's value.
If one plugin provides an additional (non-mandatory) non-evaluated recommendation, then it should explain why in the doc, the users should be able to understand and pick what is the most interesting for them.

Also, the settings where having a recommendation is useful is when the function is noisy, hence evaluate_recommendation is really dangerous, since to have an actual estimation you would need several runs, and you actually don't know how many.

[omry]

I actually agree with you that users are expecting best_evaluated_params and best_evaluated_result.
I am also fine dropping the evaluate_recommendation idea.

One thing we can add in addition is a full log:

optimization:
   run_log:
     - params: {x: 10, y:20, z: 30}
       evaluated_score: 0.2
     - ...

This can potentially be expensive if it's very large but worth considering.

Can you come up with a concrete proposition?

[omry]

@shagunsodhani can you clarify exactly what is the issue here? there is no description and the issue title is vague.

[shagunsodhani]

We want to have a schema, that all HPO plugins can use, for returning the result of optimization.

Regarding #810, the ask is that we support saving the "state" (and not just the final output) of the optimization process.

[omry]

Can you update the description of the task?

[shagunsodhani]

Reviving this thread. Can we switch to the following schema for results

@dataclass
class SweeperResult
  name : str = MISSING
  best_evaluated_parameters: Dict[str, Any] = MISSING # optimal parameters found (that were run and evaluated)
  recommended_parameters: Dict[str, Any] = MISSING # parameters that are recommended to be the best. For Ax, they will be same as `best_evaluated_parameters`, for Nevergrad, they can be different.
  best_evaluated_result: float = MISSING     # optimal return value obtained (using `best_evaluated_parameters`)

These are the base/common fields. We can have plugin-specific fields as needed later.

Regarding adding a full log, I think we can hold it off for now and first wrap up the common output schema.

[omry]

Seems reasonable but my context on this matter is now zero.
How does it compare to what we discussed before?

[shagunsodhani]

I had lost context as well so I went through all the previous comments and modified the original schema to reflect the followup discussion. As a side note, the motivation of reviving this thread is the discussion we had on Monday about HPO plugin. Standardizing the output schema is easier than standardizing the input schema and can be done alongside (or even before)work on input schema.

[omry]

I don't think this is nearly as important as the parameters config.