Autokeras: Correct way of exporting keras model

[t-wtnb]

Bug Description

I have faced a bug around exporting and keras model to .h5 file, loading into another keras code, and using the model.
It is similar to issue #167 .

After searching, fitting, final fitting a model using Auto-Keras, I exported the model as a keras model file. Then, I loaded the model into another "pure keras" code. However, the performance of the loaded model was so much poorer than the one evaluated by an Auto-Keras code.

Reproducing Steps

I have run the sample code on the official website:

from keras.datasets import mnist
from autokeras.image_supervised import ImageClassifier

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(x_train.shape + (1,))
    x_test = x_test.reshape(x_test.shape + (1,))

    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    y = clf.evaluate(x_test, y_test)
    print(y)

fit and final_fit finished, and the print(y) outputted 0.9926, the accuracy score.
After the code written above, I added one-line code for exporting the keras model:

clf.load_searcher().load_best_model().produce_keras_model().save('my_model.h5')

Next, I loaded the model my_model.h5 into another script and evaluated again with keras evaluate function.

from keras.datasets import mnist
from keras.models import load_model
from keras.utils import to_categorical

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_test = x_test.reshape(x_test.shape + (1,))
    y_test = to_categorical(y_test)

    keras_model = load_model('my_model.h5')
    keras_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

    score = keras_model.evaluate(x_test, y_test)
    print(score)

In this code, I added compile function because the error message was shown:

/home/morningyrp/tensorflow/venv/local/lib/python3.5/site-packages/keras/engine/saving.py:269: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '

The print(score) outputted [4.088045687866211, 0.1313], which means that the accuracy score was 0.1313.
In spite of the same MNIST data, this result is so much poorer the result from clf.evaluate in Auto-Keras, 0.9926.

Finally, I tried to train the model again with keras codes like this:

history = keras_model.fit(x_train, y_train,\
                    batch_size=32,\
                    epochs=1000,\
                    verbose=1,\
                    validation_split=0.1)

However, the accuracy could not increase and the loss could not reduce through whole epochs.

I also tried to export the best model without final_fit of Auto-Keras, loaded it to another keras code, and trained using compile and fit function of pure keras.
However, the accuracy and the loss could not improve through whole epochs.

Expected Behavior

The keras model exported by clf.load_searcher().load_best_model().produce_keras_model().save('my_model.h5') can be used correctly on another "pure keras" code.
In other words, fit, evaluate, predict functions of the exported model can be used correctly in other keras codes.

Setup Details

Include the details about the versions of:

• OS type and version: Linux (Ubuntu 16.04)
• Python: 3.5.2
• autokeras: 0.2.13
• scikit-learn:
• numpy: 1.14.5
• keras: 2.2.2
• scipy: 1.1.0
• tensorflow: 1.10.1
• pytorch: 0.4.1

[AShyshkov]

I have the same issue.
Any ideas will be appreciated.

[haifeng-jin]

@t-wtnb @AShyshkov Thank you for the issue.
The Keras model itself is hard to have the same behavior as the Auto-Keras. Since it doesn't contain the steps of preprocessing the data and the labels.
Sometimes the data should be normalized. The labels should be one-hot encoded.
We are very open to any suggestions that would fix this.

Thanks.

[t-wtnb]

@jhfjhfj1 Thank you for your comment.
I'm sorry I missed the note on README.md:

Note: currently, Auto-Keras is only compatible with: Python 3.6.

Then, I tried the following:

• Using Python 3.6
• Input MNIST x (data): devided by 255.0 for normalization
• Input MNIST y (label): one-hot-vectored using keras.utils.to_categorical

However, this issue still existed, that is, the performance of the model is poor in a pure keras code.

Fortunately, the network structure of the model generated by Auto-Keras can be plotted into PNG file using keras.utils.plot_model.
I will try to construct the same network using pure keras code (functional API).

In addition, until this issue will be fixed, I try to conclude the whole process (searching model, final-fitting model, and predicting using the model) in an Auto-Keras code.

Thank you again for your kindness.

[AShyshkov]

Hey, I think we found the issue.
Seem like a defect in AutoKeras. The last layer in the best model for MNIST classifier is a Dense layer with linear activation function. Training such model ends up with exploding gradients issue. Change it to softmax (or top it off with layer Activation('softmax') ), retrain your model and you should be fine.

[t-wtnb]

@AShyshkov Thank you for your great advice! Your comment is very helpful for me.
After loading the model generated by Auto-Keras, I added Activation('softmax') layer at the final output Dense layer, and compiled it.

keras_model = load_model('my_model.h5')

x = keras_model.output
x = Activation('softmax', name='activation_add')(x)
keras_model = Model(keras_model.input, x)

keras_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Then, I retrained the model using fit function. Through the retraining, both loss and accuracy successfully improved.

Thank you again!

[AShyshkov]

You are we welcome!
(I have to say, the fix was found not by me, but by my manager)

The code is exactly what we did. Just FYI, you can do this instead:
keras_model.layers[:-1].activation = keras.activations.softmax

The result would be the same.

[ma3oun]

Same issue but on a custom dataset...

[liubo12]

Bug Description
when I predict the data after training model, the result :
array(['0'], dtype=' when I save and reload the model, the result:
array([[ 2.2954037, -1.2626443]], dtype=float32)

So I try to add softmax after whole model, but it also does not work.
Any ideas will be appreciated.

[nivniv850]

Hi,
I have the same issue, I tried @t-wtnb solution but it stills give me different values:

My Code:

x_test, y_test = load_image_dataset(csv_file_path="test/label.csv", images_path="test")

model = pickle_from_file('my_autokeras_model.h5ak')
results = model.evaluate(x_test, y_test)
print(results)

keras_model = load_model('model.h5')
x = keras_model.output
x = Activation('softmax', name='activation_add')(x)
new_model = Model(keras_model.input, x)
new_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
y_test = to_categorical(y_test)
score = new_model.evaluate(x_test, y_test)
print(score)

result:
0.7238095238095238
[8.135800362768627, 0.49523809523809526]

which is different .

I will appropriate any help.
thank you!

[nivniv850]

Hi,
Thanks to @t-wtnb , reading again its answer I think what I was missing is the re-train (fit) again.

[songqiqqq]

You are we welcome!
(I have to say, the fix was found not by me, but by my manager)

The code is exactly what we did. Just FYI, you can do this instead:
keras_model.layers[:-1].activation = keras.activations.softmax

The result would be the same.

the colon ":" is a typo ?. To revise the last layer's activation function, it should be keras_model.layers[-1].activation = keras.activations.softmax ? @AShyshkov

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.