Autokeras: Low eval accuracy for TextClassifier (Might because of cannot save text_vectorization)

[dzimmerman-nci]

I have run the Text Classification example here: https://autokeras.com/tutorial/text_classification/

My validation accuracy gets to the high 90s, but when I run inference on the test set, it is close to 50%. There's nothing inherently wrong with that if that's the expected output. However, it looks as though it is just using the last model trained and not the best model based on the printed results from the last model being trained.

Also, when I run clf.evaluate(x_train, y_train) I get the same results of 50% accuracy. I apologize if I'm missing something obvious here, but this does not look right to me.

[haifeng-jin]

@dzimmerman-nci It seems caused by the text vectorization. It does not support saving.
https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/layers/preprocessing/text_vectorization.py#L290
I believe this will be fixed by the next release of Tensorflow.
For the a walk around would be export the model and adapt the layer yourself.

keras_model = auto_model.export_model()
text_vectorization_layer = keras_model.layers[1]  # may be other layers, you can print keras_model.layers to see
text_vectorization_layer.adapt(dataset)  # dataset only contain x, no y.

Then the keras_model can be used as normal.
Let me know if it doesn't work.

[dzimmerman-nci]

Thanks for the help @haifeng-jin. To clarify, do I need to retrain the Keras model after running the Auto-Keras fit? Or do I only need to adapt the test data X values to the exported Keras model to run inference?

[haifeng-jin]

@dzimmerman-nci No retraining is needed. You only need to adapt the preprocessing layer, i.e., text_vectorization with the training data. The adapt function is the fit function for preprocessing layers. They currently cannot be fit with the model's fit function, but need to be adapt separate.

example here:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization

[dzimmerman-nci]

So AutoKeras also does this when searching for the best model architecture? That layer is just not saved properly after the AutoKeras fit method?

[dzimmerman-nci]

@haifeng-jin it seems the only way to get it to work is if I retrain the exported Keras model after adapting the TextVectorization layer like this:

auto_keras_model = ak.TextClassifier(max_trials=3)
auto_keras_model.fit(x_train, y_train)
keras_model = auto_keras_model.export_model()
text_vectorization_layer = keras_model.layers[1]
text_vectorization_layer.adapt(x_train)
keras_model.fit(x_train, y_train)
print(keras_model.evaluate(x_test, y_test))

If I exclude the keras_model.fit(x_train, y_train) line, I get the same results as I do when I do not adapt the TextVectorization layer.

[haifeng-jin]

@dzimmerman-nci Thank you for the work solution! Hope tf would release this "fit to adapt" feature soon.

[haifeng-jin]

If you use the latest tf-nightly, there won't be this problem anymore.