Autokeras: AutoModel fit does not save the best model for export later
Bug Description
After finishing AutoModel fit when I try to export the model it complains about missing file. The error message is as below:
ValueError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /ak_vanilla/trial_24d08bd5d9cf85fdcf31a67e75367d72/checkpoints/epoch_20/checkpoint: Not found: /ak_vanilla/trial_24d08bd5d9cf85fdcf31a67e75367d72/checkpoints/epoch_20; No such file or directory
When looking into the directory the folder for epoch_0 is available followed by epoch_21, epoch_22, ... epoch_30 (max epoch). However, epoch_20 is missing. I am not sure why this behaviour occurs.
screenshot of the directory: https://prnt.sc/t6qacf
Bug Reproduction
import numpy as np
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.python.keras.utils.data_utils import Sequence
import autokeras as ak
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train[:100]
y_train = y_train[:100]
print(x_train.shape) # (60000, 28, 28)
print(y_train.shape) # (60000,)
print(y_train[:3]) # array([7, 2, 1], dtype=uint8)
# Initialize the image regressor.
reg = ak.ImageRegressor(
overwrite=True,
max_trials=10)
# Feed the image regressor with training data.
reg.fit(x_train, y_train, epochs=30)
mdl = reg.export_model()
Data used by the code:
Loading default mnist_data (shown in the code)
Expected Behavior
export_model() should export the Keras model. AutoModel.fit() should save all the epochs during training.
Setup Details
Ubuntu 18.04
Python 3.6.9
autokeras==1.0.3
keras-tuner==1.0.2rc0
sklearn==0.23.1
numpy==1.18.4
pandas==1.0.5
tensorflow==2.2.0
SivamPillai
All 12 comments
I was having similar issues with version 1.0.3
chenmin1968
on 26 Jun 2020
OK, on further investigation, I have found that the error comes from keras-tuner in the save_model function of tuner class:
def save_model(self, trial_id, model, step=0):
epoch = step
self._checkpoint_model(model, trial_id, epoch)
# TODO: save the top epoch checkpoints instead of last ones.
epoch_to_delete = epoch - self._save_n_checkpoints
best_epoch = self.oracle.get_trial(trial_id).best_step
if epoch > self._save_n_checkpoints and epoch_to_delete != best_epoch:
self._delete_checkpoint(
trial_id, epoch_to_delete)
Here the model checks, if the number of epochs is more than the maximum number and starts deleting older models. The if condition ensures best_model is not deleted. However, the problem comes from the fact that best_epoch is always None. As a result, that condition is useless and old models (older than self._save_n_checkpoints=10) including best model gets deleted.
Will need advice on how to make sure oracle appropriately updates best_epoch here so we can skip deleting the best model. A simple hack to make this work (if storage is not an issue) is to simply comment out the delete portion (that will save all the epochs)
def save_model(self, trial_id, model, step=0):
epoch = step
self._checkpoint_model(model, trial_id, epoch)
# TODO: save the top epoch checkpoints instead of last ones.
epoch_to_delete = epoch - self._save_n_checkpoints
best_epoch = self.oracle.get_trial(trial_id).best_step
# if epoch > self._save_n_checkpoints and epoch_to_delete != best_epoch:
# self._delete_checkpoint(
# trial_id, epoch_to_delete)
I have extended my hack further, to achieve the original goal in a non-standard way. I realized that best_step is only set at the end of the trial. So it is None during the trial. However, since we call save_model on epoch end, we can use the traditional approach of identifying best_epoch by comparing val_loss from logs and using this to find best_epoch and pass to the save model.
Here are my updates to kerastuner/engine/tuner.py file (follow the comments CHANGE HERE):
```
required to set the initial loss value to infinity (very large number)
import numpy as np
class Tuner(base_tuner.BaseTuner):
def __init__(self,
oracle,
hypermodel,
max_model_size=None,
optimizer=None,
loss=None,
metrics=None,
distribution_strategy=None,
directory=None,
project_name=None,
logger=None,
tuner_id=None,
overwrite=False):
# Subclasses of `KerasHyperModel` are not automatically wrapped.
# CHANGE HERE: initialized values for best_loss and best_epoch later set after each epoch
self.best_loss = np.inf
self.best_epoch = 0
if not isinstance(hypermodel, hm_module.KerasHyperModel):
hypermodel = hm_module.KerasHyperModel(
hypermodel,
max_model_size=max_model_size,
optimizer=optimizer,
loss=loss,
metrics=metrics,
distribution_strategy=distribution_strategy)
super(Tuner, self).__init__(oracle=oracle,
hypermodel=hypermodel,
directory=directory,
project_name=project_name,
logger=logger,
overwrite=overwrite)
self.distribution_strategy = distribution_strategy
# Support multi-worker distribution strategies w/ distributed tuning.
# Only the chief worker in each cluster should report results.
if self.distribution_strategy is not None:
self.oracle.multi_worker = (
self.distribution_strategy.extended._in_multi_worker_mode())
self.oracle.should_report = (
self.distribution_strategy.extended.should_checkpoint)
# Save only the last N checkpoints.
self._save_n_checkpoints = 10
self.tuner_id = tuner_id or self.tuner_id
CHANGE HERE: save_model function (Original is commented out)
def save_model(self, trial_id, model, step=0):
epoch = step
self._checkpoint_model(model, trial_id, epoch)
# TODO: save the top epoch checkpoints instead of last ones.
epoch_to_delete = epoch - self._save_n_checkpoints
best_epoch = self.oracle.get_trial(trial_id).best_step
if epoch > self._save_n_checkpoints and epoch_to_delete != best_epoch:
self._delete_checkpoint(
trial_id, epoch_to_delete)
def save_model(self, trial_id, model, step=0):
epoch = step
self._checkpoint_model(model, trial_id, epoch)
# TODO: save the top epoch checkpoints instead of last ones.
epoch_to_delete = epoch - self._save_n_checkpoints
best_epoch = self.oracle.get_trial(trial_id).best_step
print("BEST EPOCH, DELETE EPOCH, NCHKP: ", self.best_epoch, epoch_to_delete, self._save_n_checkpoints)
if epoch > self._save_n_checkpoints and epoch_to_delete != self.best_epoch:
self._delete_checkpoint(
trial_id, epoch_to_delete)
def on_epoch_end(self, trial, model, epoch, logs=None):
"""A hook called at the end of every epoch.
# Arguments:
trial: A `Trial` instance.
model: A Keras `Model`.
epoch: The current epoch number.
logs: Dict. Metrics for this epoch. This should include
the value of the objective for this epoch.
"""
# CHANGE HERE: check if current loss better than previous loss and set best_epoch accordingly
if logs['val_loss'] < self.best_loss:
self.best_epoch = epoch
self.best_loss = logs['val_loss']
print("best: ", self.best_epoch, self.best_loss, logs['val_loss'])
self.save_model(trial.trial_id, model, step=epoch)
# Report intermediate metrics to the `Oracle`.
status = self.oracle.update_trial(
trial.trial_id, metrics=logs, step=epoch)
trial.status = status
if trial.status == "STOPPED":
model.stop_training = True
Should we create a PR for kerastuner based on this?
SivamPillai
on 27 Jun 2020
Thanks a lot.
The piece of code is running now!
On Sat, Jun 27, 2020 at 5:59 AM Sivam Pillai notifications@github.com
wrote:
I have extended my hack further, to achieve the original goal in a
non-standard way. I realized that best_step is only set at the end of the
trial. So it is None during the trial. However, since we call save_model on
epoch end, we can use the traditional approach of identifying best_epoch by
comparing val_loss from logs and using this to find best_epoch and pass to
the save model.Here are my updates to kerastuner/engine/tuner.py file (follow the
comments CHANGE HERE):required to set the initial loss value to infinity (very large number)
import numpy as np
class Tuner(base_tuner.BaseTuner):
def __init__(self, oracle, hypermodel, max_model_size=None, optimizer=None, loss=None, metrics=None, distribution_strategy=None, directory=None, project_name=None, logger=None, tuner_id=None, overwrite=False): # Subclasses of `KerasHyperModel` are not automatically wrapped. # CHANGE HERE: initialized values for best_loss and best_epoch later set after each epoch self.best_loss = np.inf self.best_epoch = 0 if not isinstance(hypermodel, hm_module.KerasHyperModel): hypermodel = hm_module.KerasHyperModel( hypermodel, max_model_size=max_model_size, optimizer=optimizer, loss=loss, metrics=metrics, distribution_strategy=distribution_strategy) super(Tuner, self).__init__(oracle=oracle, hypermodel=hypermodel, directory=directory, project_name=project_name, logger=logger, overwrite=overwrite) self.distribution_strategy = distribution_strategy # Support multi-worker distribution strategies w/ distributed tuning. # Only the chief worker in each cluster should report results. if self.distribution_strategy is not None: self.oracle.multi_worker = ( self.distribution_strategy.extended._in_multi_worker_mode()) self.oracle.should_report = ( self.distribution_strategy.extended.should_checkpoint) # Save only the last N checkpoints. self._save_n_checkpoints = 10 self.tuner_id = tuner_id or self.tuner_idCHANGE HERE: save_model function (Original is commented out)
def save_model(self, trial_id, model, step=0):
epoch = step
self._checkpoint_model(model, trial_id, epoch)
# TODO: save the top epoch checkpoints instead of last ones.
epoch_to_delete = epoch - self._save_n_checkpoints
best_epoch = self.oracle.get_trial(trial_id).best_step
if epoch > self._save_n_checkpoints and epoch_to_delete != best_epoch:
self._delete_checkpoint(
trial_id, epoch_to_delete)
def save_model(self, trial_id, model, step=0): epoch = step self._checkpoint_model(model, trial_id, epoch) # TODO: save the top epoch checkpoints instead of last ones. epoch_to_delete = epoch - self._save_n_checkpointsbest_epoch = self.oracle.get_trial(trial_id).best_step
print("BEST EPOCH, DELETE EPOCH, NCHKP: ", self.best_epoch, epoch_to_delete, self._save_n_checkpoints) if epoch > self._save_n_checkpoints and epoch_to_delete != self.best_epoch: self._delete_checkpoint( trial_id, epoch_to_delete) def on_epoch_end(self, trial, model, epoch, logs=None): """A hook called at the end of every epoch. # Arguments: trial: A `Trial` instance. model: A Keras `Model`. epoch: The current epoch number. logs: Dict. Metrics for this epoch. This should include the value of the objective for this epoch. """ # CHANGE HERE: check if current loss better than previous loss and set best_epoch accordingly if logs['val_loss'] < self.best_loss: self.best_epoch = epoch self.best_loss = logs['val_loss'] print("best: ", self.best_epoch, self.best_loss, logs['val_loss']) self.save_model(trial.trial_id, model, step=epoch) # Report intermediate metrics to the `Oracle`. status = self.oracle.update_trial( trial.trial_id, metrics=logs, step=epoch) trial.status = status if trial.status == "STOPPED": model.stop_training = True—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/autokeras/issues/1210#issuecomment-650558229,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQBYMPDWKJTD7EXD3TH2GDTRYXUMTANCNFSM4OJA6YFA
.
chenmin1968
on 28 Jun 2020
This issue helps a lot. However, the solution doesn't work for me. I find that the best model is evaluated by the validation accuracy, not the validation loss. The best epoch should be the one with the highest accuracy. Simply replacing best loss with best accuracy works.
lolrudy
on 28 Jun 2020
Yes true, you need to pick the specific metric that needs to be compared to identify the best model. For me it was a regression problem so I chose val_loss (rmse).
SivamPillai
on 28 Jun 2020
I think that you do not need to fix this by hardcoding the metric that you need in a certan task.
The problem for me was that autokeras could not get correct model checkpoint for epoch because it was looking for a deleted one and that is why i got the error when the trials finished and the final "best model" loop started to loop the best trials.
The deleting algo was described earlier and is in save_model method. Debugging this I noticed that epoch value differs from step value in console and in trial.json step value equals the epoch_value. Then I noticed that the deleted epoch is the previous to the first that was saved in my checkpoints directory. For example if i have best step=9 for trial then my best epoch number is 10 in console log and checkpoint is saved in epoch_9 directory. And the save_model method just delets my directory epoch_9 cause it starts to delete from the wrong epoch number.
That is why I fixed the line in the method above:
epoch_to_delete = epoch - self._save_n_checkpoints
to this:
epoch_to_delete = epoch - self._save_n_checkpoints - 1
and now my best checkpoints are stored correctly. Hope this helps you too.
I-Kryachko
on 7 Jul 2020
Well, I had tried this first but for some reason, it did not work all the time. I still used to get an error some times. Need to figure out why.
SivamPillai
on 7 Jul 2020
I examined this bug and have the fix in the is PR #1229 .
haifeng-jin
on 12 Jul 2020
Hmm I tried pip installing using the .git link today but it still seems to have the same issue. Is the fix coming in a future version?
iwidodo
on 16 Jul 2020
Yes, this will be in a future version. However, you can use it in the master branch now.
haifeng-jin
on 20 Jul 2020
I can confirm that the issue is resolved in version 1.0.5 with TF 2.3.0. Thanks @haifeng-jin .
SivamPillai
on 4 Sep 2020
Related issues
arthursdays
·
5Comments
GagaLeung
·
4Comments
max1563
·
4Comments
ikscapes
·
3Comments
xuzhang5788
·
4Comments
Most helpful comment
I think that you do not need to fix this by hardcoding the metric that you need in a certan task.
The problem for me was that autokeras could not get correct model checkpoint for epoch because it was looking for a deleted one and that is why i got the error when the trials finished and the final "best model" loop started to loop the best trials.
The deleting algo was described earlier and is in
save_modelmethod. Debugging this I noticed thatepochvalue differs fromstepvalue in console and in trial.json step value equals the epoch_value. Then I noticed that the deleted epoch is the previous to the first that was saved in my checkpoints directory. For example if i have best step=9 for trial then my best epoch number is 10 in console log and checkpoint is saved in epoch_9 directory. And thesave_modelmethod just delets my directory epoch_9 cause it starts to delete from the wrong epoch number.That is why I fixed the line in the method above:
epoch_to_delete = epoch - self._save_n_checkpointsto this:
epoch_to_delete = epoch - self._save_n_checkpoints - 1and now my best checkpoints are stored correctly. Hope this helps you too.