Ax: `ScalarizedObjective` serialization and storage

[BCJuan]

Hello,

Ax version: 0.1.6

I am trying to save an experiment which looks like:

    experiment = Experiment(
        name="experiment_building_blocks",
        search_space=search_space(),
    )

    optimization_config = OptimizationConfig(
        objective=ScalarizedObjective(
            metrics=[AccuracyMetric(datasets, dataloaders,
                                    device, datasetsizes,
                                    epochs, name="accuracy"),
                     WeightMetric(datasets, bits, name="weight")],
            weights=[0.5, 0.5],
            minimize=True,
        ),
    )

    experiment.optimization_config = optimization_config
    experiment.runner = MyRunner()

As seen it is an Scalarized Objective with two metrics. When I try to save the experiment by registering both the runner and the metrics:

    register_metric(AccuracyMetric)
    register_metric(WeightMetric)
    register_runner(MyRunner)

    save(exp, path.join(root, name))

Then the following error appears:

optimization_config Traceback (most recent call last):
  File "main.py", line 172, in <module>
    root=parsed.root)
  File "main.py", line 113, in main
    save_data(exp, name, root)
  File "main.py", line 152, in save_data
    save(exp, path.join(root, name))
  File "/home/kostal/anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/json_store/save.py", line 22, in save_experiment
    json_experiment = object_to_json(experiment)
  File "/home/kostal/anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/json_store/encoder.py", line 74, in object_to_json
    print(k, v)
  File "/home/kostal/anaconda3/envs/deep/lib/python3.7/site-packages/ax/core/optimization_config.py", line 138, in __repr__
    objective=repr(self.objective),
  File "/home/kostal/anaconda3/envs/deep/lib/python3.7/site-packages/ax/core/objective.py", line 44, in __repr__
    self.metric.name, self.minimize
  File "/home/kostal/anaconda3/envs/deep/lib/python3.7/site-packages/ax/core/objective.py", line 84, in metric
    raise NotImplementedError("ScalarizedObjective is composed of multiple metrics")

Looks like that in the source of Scalarized Objective it has no __repr__method. I have tried to implement it as:

    def __repr__(self) -> str:
        base_str = 'Objective('
        for k, v in zip(self.metrics, self.weights):
            base_str += ' metric_name="{}", metric_weight="{}",'.format(
                                    k.name, v)
        base_str += ' minimize={})'.format(self.minimize)
        return base_str

With no luck because another appears, as this is not registered as an ENCODER_REGISTRY for JSON encoding.

Thanks in advance.

[sdsingh]

Hi there, apologies for this error. It appears that ScalarizedObjective is missing its serialization logic. PR coming soon!

[BCJuan]

HI again,

Thank you for solving the issue, but there exists the same problem when trying to save with SQL storage.

  File "/home/.../Documents/kostal_projects/implementation/main.py", line 128, in main
    save_experiment(exp)

  File "/home/..../anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/sqa_store/save.py", line 22, in save_experiment
    _save_experiment(experiment=experiment, encoder=encoder)

  File "/home/.../anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/sqa_store/save.py", line 37, in _save_experiment
    new_sqa_experiment = encoder.experiment_to_sqa(experiment)

  File "/home/..../anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/sqa_store/encoder.py", line 94, in experiment_to_sqa
    experiment.optimization_config

  File "/home/.../anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/sqa_store/encoder.py", line 354, in optimization_config_to_sqa
    objective_sqa = self.objective_to_sqa(objective=optimization_config.objective)

  File "/home/.../anaconda3/envs/deep/lib/python3.7/site-packages/ax/storage/sqa_store/encoder.py", line 311, in objective_to_sqa
    metric = objective.metric

  File "/home/.../anaconda3/envs/deep/lib/python3.7/site-packages/ax/core/objective.py", line 84, in metric
    raise NotImplementedError("ScalarizedObjective is composed of multiple metrics")

NotImplementedError: ScalarizedObjective is composed of multiple metrics

[sdsingh]

Good catch, let me add support for that.

[jelena-markovic]

Hi @BCJuan, storing scalarized objective has just been added (commit: 933bf4f44dd9287f8a8f6bbff182258bbd2f3499, branch: master, date: March 10th, 2020). Feel free to test it and let us know if there are more issues!

[Jakepodell]

fix landed in new release