Ax: Optimize doesn't work with evaluation function that returns value of some NumPy types

[Irynei]

I noticed that when the evaluation_function returns value of some NumPy type(I was able to reproduce with np.int64, np.int32, np.float32), optimize method crashes with the following exception:

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py", line 206, in optimize
loop.full_run()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py", line 150, in full_run
self.run_trial()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py", line 141, in run_trial
trial.fetch_data()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/base_trial.py", line 259, in fetch_data
trial_index=self.index, metrics=metrics, **kwargs
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py", line 205, in _fetch_trial_data
return self.eval_trial(self.trials[trial_index])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py", line 119, in eval_trial
not_none(trial.arm).parameters, None
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py", line 192, in evaluation_function_outer
"Evaluation function returned an invalid type. The function must "
Exception: Evaluation function returned an invalid type. The function must either return a dictionary of metric names to mean, sem tuples or a single mean, sem tuple, or a single mean.

My environment:

numpy==1.16.1
ax-platform==0.1.2

Script to reproduce:

import numpy as np
from ax import optimize
from ax.utils.measurement.synthetic_functions import branin

best_parameters, values, experiment, model = optimize(
    parameters=[
        {
            "name": "x1",
            "type": "range",
            "bounds": [-5.0, 10.0],
        },
        {
            "name": "x2",
            "type": "range",
            "bounds": [0.0, 10.0],
        },
    ],
    evaluation_function=lambda p: np.float32(branin(p["x1"], p["x2"])),
    minimize=True,
)

[dkatz23238]

Interesting. Have you tried with torch.Tensor data types?

[lena-kashtelyan]

@Irynei, this is intentional behavior –– the evaluation function is strictly typed, and it's return type is a union of a float, a tuple of two floats, and a dictionary of a string to such tuples, as documented in SimpleExperiment. We use python builtins for primitive types and convert from numpy primitives where necessary. However, you're right that for the evaluation function, we should allow numpy types –– will add that support shortly.

Is it much trouble to convert back from numpy to python primitive in the evaluation function, for now? You could use our utility ax.utils.common.typeutils.numpy_type_to_python_type, if you'd like.

[Irynei]

@lena-kashtelyan, thanks for your explanation. I was expecting the evaluation function to work with numpy types, so for me, that was a surprising behavior.
No, It's not hard to convert them for now, but thanks for the suggestion.

[lena-kashtelyan]

Great! I'll let you know once the fix for this is on master.

[Irynei]

Cool, thank you!

[ldworkin]

Fix is in the latest version!