Autokeras: The best model obtained is too large and has poor performance. Are there specific dataset requirements for the tuner?

[SivamPillai]

Firstly, thanks for this library. Auto-Keras makes it really easy to find best model configuration through multiple trials with minimal code.

Bug Description

The Image_Regressor block fails to generate any useful model after trying 50+ trials. The best model achieved is too huge for the dataset provided and the prediction results (even for the training set) are completely off. Are there any minimum dataset requirement for the tuner to work more effectively? Are there settings to control the size or convergence?

Sample Code

reg = ak.ImageRegressor(
    output_dim=None,
    loss="mean_squared_error",
    metrics=None,
    project_name="img_regressor",
    max_trials=30,
    directory=model_dir,
    objective="val_loss",
    tuner=None,
    overwrite=False,
    seed=2806
)

reg.fit(xtr, ytr, epochs=200, verbose=2)
model = reg.explort_model()
ypred = model.predict(xtr)

Data used by the code:
I have a sensor-based data in the form of images with small sample size:

data shape: 5668, 32, 32, 2

Best Model Obtained

Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 32, 32, 2)]       0         
_________________________________________________________________
normalization (Normalization (None, 32, 32, 2)         5         
_________________________________________________________________
random_contrast (RandomContr (None, 32, 32, 2)         0         
_________________________________________________________________
resizing (Resizing)          (None, 71, 71, 2)         0         
_________________________________________________________________
conv2d (Conv2D)              (None, 71, 71, 3)         9         
_________________________________________________________________
xception (Functional)        (None, 3, 3, 2048)        20861480  
_________________________________________________________________
flatten (Flatten)            (None, 18432)             0         
_________________________________________________________________
regression_head_1 (Dense)    (None, 1)                 18433     
=================================================================
Total params: 20,879,927
Trainable params: 20,825,394
Non-trainable params: 54,533
_________________________________________________________________

Expected Behavior

If there is a data issue I would imagine the results will be off for the test set. However, despite tuning for several epochs the model performance is extremely worse even for the training set.

When I implement a simple Keras based 2 layer Convnet the results are still better.

Setup Details

Include the details about the versions of:

• OS type and version: Ubuntu 18.04
• Python: 3.6
• autokeras: 1.0.5
• keras-tuner: 1.0.2rc0
• scikit-learn: 0.23.2
• numpy: 1.18.4
• pandas: 1.1.1
• tensorflow: 2.3.0

The questions are:

• Are there customizations that can be done to improve the model performance obtained through tuning?
• Are there any data specifications?
• Is it fair to expect a model to at least overfit (or get decent fit) for the training data?

Thanks.

[haifeng-jin]

Yes. You can set an upper bound to the number of parameters which is supported by Keras Tuner, therefore supported by AutoKeras. We may add this soon to the FAQ.

I don't think there is any configurable way to improve the performance. Are you using any public dataset? We can take a look of why it is performing poorly.

Thanks.

[SivamPillai]

Thanks for the update. I am not able to share the dataset as it is restricted. The data is a time-series of images obtained from an industrial process. Main challenges are non-stationarity and noise. I am doing some feature extraction which I can test with AK once ready.

I will look into the Keras-Tuner feature to restrict the model size. I have used Keras-Tuner separately to tune models where I can provide the range of search for each parameter. I am not sure how this works when combining with AK. Is it something I should be able to pass as kwargs to automodel?

If there is a sample code or code doc that I can refer to, that will be great.

Thanks.

[haifeng-jin]

We will add how to constrain model size to the FAQ on our website soon. It is tracked by this issue #1341 .
However, it may not work very well. Another suggestion is to use some techniques to reduce the model size after you export it.
There should be some way to reduce the size of a tf model without sacrifice the performance.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.