Autokeras: Saving trained Model and trained model interpretation

[takeofuture]

Bug Description

This is more like the clarification on the tutorial description in very basic level. I tried
https://autokeras.com/tutorial/image_classification/
I try to understand the result. I got the following output by clf.fit(x_train, y_train,epochs=3)
Does this mean 3 models are compared and in the last step they try again with the bets score model? (Trial ID: 5ef9850ad12a412e6263423d2bccf89a, Score: 0.06611143700537893)
I think only best model (for this used dataset and specified epochs) can be saved by
model = clf.export_model()
model.save()
using regular Keras model class.
Is there any way to save other models used for this training (not only the best model)?

(60000, 28, 28)
(60000,)
[5 0 4]
Train for 1500 steps, validate for 375 steps
Epoch 1/3
2020-04-06 23:21:41.977044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-06 23:21:45.890081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
   1/1500 [..............................] - ETA: 7:15:32 - loss: 2.2885 - accuracy: 0.1250
   ~~
1500/1500 [==============================] - 31s 21ms/step - loss: 0.1746 - accuracy: 0.9469 - val_loss: 0.0689 - val_accuracy: 0.9793
Epoch 2/3
   1/1500 [..............................] - ETA: 5:37 - loss: 0.0783 - accuracy: 0.9688
   ~~
1500/1500 [==============================] - 13s 9ms/step - loss: 0.0774 - accuracy: 0.9762 - val_loss: 0.0488 - val_accuracy: 0.9863
Epoch 3/3
   1/1500 [..............................] - ETA: 5:28 - loss: 0.0239 - accuracy: 1.0000
   ~~
1500/1500 [==============================] - 13s 9ms/step - loss: 0.0625 - accuracy: 0.9806 - val_loss: 0.0477 - val_accuracy: 0.9860
[Trial complete]
[Trial summary]
 |-Trial ID: 6f63dad309051c3971521ba643fad0c7
 |-Score: 0.0476510140611014
 |-Best step: 0
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0.5
 |-classification_head_1/spatial_reduction_1/reduction_type: flatten
 |-dense_block_1/dropout_rate: 0
 |-dense_block_1/num_layers: 1
 |-dense_block_1/units_0: 128
 |-dense_block_1/use_batchnorm: False
 |-image_block_1/augment: False
 |-image_block_1/block_type: vanilla
 |-image_block_1/conv_block_1/dropout_rate: 0.25
 |-image_block_1/conv_block_1/filters_0_0: 32
 |-image_block_1/conv_block_1/filters_0_1: 64
 |-image_block_1/conv_block_1/kernel_size: 3
 |-image_block_1/conv_block_1/max_pooling: True
 |-image_block_1/conv_block_1/num_blocks: 1
 |-image_block_1/conv_block_1/num_layers: 2
 |-image_block_1/conv_block_1/separable: False
 |-image_block_1/normalize: True
 |-optimizer: adamTrain for 1500 steps, validate for 375 steps
Epoch 1/3
   1/1500 [..............................] - ETA: 3:21:48 - loss: 2.9383 - accuracy: 0.0938
   ~~
1500/1500 [==============================] - 157s 105ms/step - loss: 0.2569 - accuracy: 0.9306 - val_loss: 0.1597 - val_accuracy: 0.9577
Epoch 2/3
   1/1500 [..............................] - ETA: 7:40 - loss: 0.0488 - accuracy: 0.9688
   ~~
1500/1500 [==============================] - 151s 100ms/step - loss: 0.1119 - accuracy: 0.9716 - val_loss: 0.0661 - val_accuracy: 0.9804
Epoch 3/3
   1/1500 [..............................] - ETA: 8:19 - loss: 0.0624 - accuracy: 0.9688
   ~~
1500/1500 [==============================] - 148s 98ms/step - loss: 0.0708 - accuracy: 0.9797 - val_loss: 0.0751 - val_accuracy: 0.9791
[Trial complete]
[Trial summary]
 |-Trial ID: 5ef9850ad12a412e6263423d2bccf89a
 |-Score: 0.06611143700537893
 |-Best step: 0
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0
 |-dense_block_1/dropout_rate: 0
 |-dense_block_1/num_layers: 2
 |-dense_block_1/units_0: 32
 |-dense_block_1/units_1: 32
 |-dense_block_1/use_batchnorm: False
 |-image_block_1/augment: True
 |-image_block_1/block_type: resnet
 |-image_block_1/normalize: True
 |-image_block_1/res_net_block_1/conv3_depth: 4
 |-image_block_1/res_net_block_1/conv4_depth: 6
 |-image_block_1/res_net_block_1/pooling: avg
 |-image_block_1/res_net_block_1/version: v2
 |-optimizer: adam
Train for 1500 steps, validate for 375 stepsEpoch 1/3
   1/1500 [..............................] - ETA: 15:56 - loss: 2.4124 - accuracy: 0.0938
   ~~
1500/1500 [==============================] - 14s 9ms/step - loss: 0.1805 - accuracy: 0.9457 - val_loss: 0.0664 - val_accuracy: 0.9797
Epoch 2/3
   1/1500 [..............................] - ETA: 5:27 - loss: 0.0402 - accuracy: 1.0000
   ~~
1500/1500 [==============================] - 13s 9ms/step - loss: 0.0775 - accuracy: 0.9759 - val_loss: 0.0544 - val_accuracy: 0.9843
Epoch 3/3
   1/1500 [..............................] - ETA: 5:20 - loss: 0.0184 - accuracy: 1.0000
   ~~
1500/1500 [==============================] - 13s 9ms/step - loss: 0.0628 - accuracy: 0.9807 - val_loss: 0.0520 - val_accuracy: 0.9855
[Trial complete]
[Trial summary]
 |-Trial ID: 7af457cb193b4f2c9ed2b0a4051ea257
 |-Score: 0.05198277689473859
 |-Best step: 0
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0.5
 |-classification_head_1/spatial_reduction_1/reduction_type: flatten
 |-dense_block_1/dropout_rate: 0
 |-dense_block_1/num_layers: 1
 |-dense_block_1/units_0: 128
 |-dense_block_1/use_batchnorm: False
 |-image_block_1/augment: False
 |-image_block_1/block_type: vanilla
 |-image_block_1/conv_block_1/dropout_rate: 0.25
 |-image_block_1/conv_block_1/filters_0_0: 32
 |-image_block_1/conv_block_1/filters_0_1: 64
 |-image_block_1/conv_block_1/kernel_size: 3
 |-image_block_1/conv_block_1/max_pooling: True
 |-image_block_1/conv_block_1/num_blocks: 1
 |-image_block_1/conv_block_1/num_layers: 2
 |-image_block_1/conv_block_1/separable: False
 |-image_block_1/normalize: True
 |-optimizer: adamTrain for 1875 steps, validate for 375 steps
Epoch 1/3
   1/1875 [..............................] - ETA: 26:31 - loss: 2.2717 - accuracy: 0.0938
   ~~
1875/1875 [==============================] - 17s 9ms/step - loss: 0.1582 - accuracy: 0.9517 - val_loss: 0.0506 - val_accuracy: 0.9834
Epoch 2/3
   1/1875 [..............................] - ETA: 12:54 - loss: 0.0143 - accuracy: 1.0000
   ~~
1875/1875 [==============================] - 16s 9ms/step - loss: 0.0729 - accuracy: 0.9769 - val_loss: 0.0289 - val_accuracy: 0.9911
Epoch 3/3
   1/1875 [..............................] - ETA: 13:06 - loss: 0.0093 - accuracy: 1.0000
   ~~
1875/1875 [==============================] - 16s 9ms/step - loss: 0.0590 - accuracy: 0.9815 - val_loss: 0.0157 - val_accuracy: 0.9958

Setup Details

Include the details about the versions of:

• OS type and version: Ubuntu18.04
• Python: 3.6.9
• autokeras: 1.0
• keras-tuner:
• scikit-learn:
• numpy:
• pandas:
• tensorflow:2.1

[ghost]

Took me sometime to figure things out too, given the sparse documentation. But here's my take on your questions:

1. The number of models tested is determined by max_trials. In this tutorial example, max_trials = 10, meaning 10 models were tested.
2. For each model, the number of training rounds (epochs) are determined by epochs. In the tutorial, epochs=3, meaning only 3 rounds of training is attempted for the model. Normally, the more epochs, the higher chance that your model is better trained (depending on your dataset). Typically, it is normal to set epochs=200. The model will however stop training if no improvement in val_loss (lower score) is observed after 10 epochs, meaning it may stop training after 50 epochs even if epochs=200.
3. After max_trials is reached (in this tutorial, 10 trials/models), AutoKeras will pick the best model and attempt one final training using all your training data. This is the tricky bit. When you specify clf.fit(x_train, y_train, epochs=3), not all x_train are used for training because AutoKeras does a 80:20 train-validation split on your x_train data. For example, if number of x_train data/sample is 10 images, the split will results in 8 images for training, 2 images for validation. For the final training of best model, AK will use all the training data (10 images as per previous example ).
4. Is it possible to save other models? Probably, the AK team will need to provide a bit more description (which takes a long time) on the nature of the saved trials within the automodel folder. All your weights and trial details are actually stored in those folder. Just need to convert them into Keras model - which again we wait for instructions from AK team?

Hope this helps.

[takeofuture]

Thank you for your detail description.
I tested and I would like to share.
So basically you can save and load model by
from tensorflow.keras.models import load_model, save_model

You can save tf format and load tf format (h5 format does not work for me)

# Export as a Keras Model.
model = clf.export_model()

print(type(model))  # <class 'tensorflow.python.keras.engine.training.Model'>
model.summary()

# https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model
save_model(model, 'mymodel_tf.tf', save_format='tf')
load_tf_model = load_model('mymodel_tf.tf', custom_objects=ak.CUSTOM_OBJECTS)
load_tf_model.summary()

#save_model(model, 'mymodel_h5.h5', save_format='h5')　#Does not work
#load_h5_model = load_model('mymodel_h5.h5', custom_objects=ak.CUSTOM_OBJECTS) #Does not work

---
import autokeras as ak
from tensorflow.keras.models import load_model
load_tf_model = load_model('mymodel_tf.tf', custom_objects=ak.CUSTOM_OBJECTS)
load_tf_model.summary()

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.