Ludwig: Hyperopt metric_score uses last epoch instead of best epoch

[michaelbzhu]

hyperopt() generates a hyperopt_statistics.json file which contains training_stats and the metric_score from each sample

I'm optimizing for validation accuracy, but I noticed that the final metric_score that gets reported in the json file uses the validation accuracy from the last epoch during training rather than the best epoch during training

Below is the output from my hyperopt_statistics.json file

"hyperopt_results": [
        {
            "eval_stats": {
                "combined": {
                    "loss": 1.4781785011291504
                },
                "label": {
                    "accuracy": 0.47138965129852295,  // this is the final reported accuracy
...
            },
            "metric_score": 0.47138965129852295,   // this value comes from HyperoptExecutor.get_metric_score(self, eval_stats) which just copies the value above
            "parameters": {
                "training.learning_rate": 0.0006019209790229743,
                "utterance.cell_type": "gru",
                "utterance.num_layers": 1,
                "utterance.state_size": 495
            },
            "training_stats": {
                "validation": {
                    "combined": {
                        "loss": [
                            1.3932547569274902,
                            1.2642898559570312,
                            1.3837428092956543,
                            1.2704368829727173,
                            1.3504513502120972,
                            1.3695340156555176,
                            1.6437498331069946,
                            1.589107632637024,
                            1.4781785011291504
                        ]
                    },
                    "label": {
                        "accuracy": [
                            0.40962761640548706,
                            0.440508633852005,
                            0.4423251450061798,
                            0.47320616245269775,  // this is the best validation accuracy from training
                            0.47320616245269775,
                            0.440508633852005,
                            0.4523160755634308,
                            0.40690281987190247,
                            0.47138965129852295   // this value from the last epoch is what's actually reported above
                        ],
...

Is this intended because the logger output from LudwigModel.train() uses the best validation accuracy so I thought hyperopt would have similar behavior?

If this is a bug I can try to help fix it as I have a general idea of where this behavior comes from in the codebase

Environment:

• Run in Google Colab
  
  
  
  • Python version 3.6.9
  
  
  • Ludwig version 0.3.1
  
  

[w4nderlust]

This is not expected behavior, you are correctly pointing out that the best score should be returned.

There could be a couple explanations for this. In particular if the model is not saved during the training part of an experiment, the model at the last epoch is used for evaluation, so I suspect this is what's happening. So if your skip_save_model=True that could explain the behavior.

Anyway, will look into it and thank you for spotting this!

[w4nderlust]

This PR is a first attempt at fixing the issue: https://github.com/uber/ludwig/pull/1031