Autokeras: F1 score support for objective

[alexcombessie]

Today objective = "val_f1" returns an error
Failed to train : : Could not infer optimization direction ("min" or "max") for unknown metric "val_f1". Please specify the objective asa kerastuner.Objective, for example kerastuner.Objective("val_f1", direction="min").

[haifeng-jin]

@alexcombessie Thank you for the issue! Would you please paste your code, too?

[alexcombessie]

Sure, here you go:

import pandas as pd
import numpy as np

from sklearn.base import BaseEstimator, RegressorMixin, ClassifierMixin
from autokeras import StructuredDataClassifier, StructuredDataRegressor

class AutoKerasClassifier(StructuredDataClassifier, BaseEstimator, ClassifierMixin):
    def __init__(self, 
        column_names = None, 
        column_types = None,
        num_classes = None,
        multi_label = False,
        loss = None,
        metrics = None, 
        name = 'structured_data_classifier',
        max_trials = 100,
        directory = None,
        objective = 'val_accuracy',
        overwrite = True,
        seed = None,
        classes_ = None,
        epochs = None,
        batch_size = 32):
        super().__init__(
            column_names = column_names,
            column_types = column_types,
            num_classes = num_classes,
            multi_label = multi_label,
            loss = loss,
            metrics = metrics,
            name = name,
            max_trials = max_trials,
            directory = directory,
            objective = objective,
            overwrite = overwrite,
            seed = seed)
        self.classes_ = classes_
        self.epochs = epochs
        self.batch_size = batch_size

    def fit(self, x = None, y = None, epochs = None, callbacks = None, validation_split = 0.2, **kwargs):
        self.classes_ = [str(i) for i in np.unique(y)]
        super().fit(x = x, y = y, epochs = self.epochs, callbacks = callbacks,
                       validation_split = validation_split, batch_size = self.batch_size, **kwargs)

    def predict(self, x, **kwargs):
        y = pd.Series(super().predict(x = x, batch_size = self.batch_size, **kwargs).flatten())
        return(y)

from autokeras_doctor import AutoKerasClassifier

clf = AutoKerasClassifier(
    max_trials = 5,
    epochs = 10,
    batch_size = 1024,
    objective = "val_f1"
)

I am using this wrapper class inside Dataiku DSS, which calls fit on clf.

Side-note: the only reason I wrote a wrapper class around StructuredDataClassifier is that I am missing the classes_ attribute, as defined in sklearn or xgboost classifiers.

[alexcombessie]

Addition: I did try with

clf = AutoKerasClassifier(
    max_trials = 5,
    epochs = 10,
    batch_size = 1024,
    objective = kerastuner.Objective("val_f1", direction="max")
)

But this time I get another error after Oracle exit:
Objective value missing in metrics reported to the Oracle, expected: ['val_f1'], found: dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

[abhinit]

Addition: I did try with

clf = AutoKerasClassifier(
    max_trials = 5,
    epochs = 10,
    batch_size = 1024,
    objective = kerastuner.Objective("val_f1", direction="max")
)

But this time I get another error after Oracle exit:
Objective value missing in metrics reported to the Oracle, expected: ['val_f1'], found: dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

Both the cases seem to be working in the current release

[haifeng-jin]

@alexcombessie We just released the new version of AutoKeras yesterday.
It should support the f1 for now.

[alexcombessie]

Thanks a lot, I will give it a try.

I am interested in contributing by the way, what would be a good way to start?

[haifeng-jin]

@alexcombessie Thank you so much!
Maybe we can start by writing a simple tutorial on how to resume the training with the overwrite argument in AutoModel?
Or you can just try f1 score works or not, if not you can work on this issue.
I will help you in the process and give more details after you tried.

[alexcombessie]

Heya @haifeng-jin,

Hope all is well in lockdown times...

I am answering quite late, sorry about that. I have a bit of a backlog going on at Dataiku. I can't commit, but if I find some time between now and summer I will try. But if anyone has time before that, definitely a better option.

Cheers,

Alex

[CharlesDove]

Good afternoon, all! I appear to be experiencing the same "Oracle, expected: ['val_f1']" error @alexcombessie detailed earlier, using release 1.0.2, so the problem doesn't appear to be fixed yet.
EDIT: In my case, this is for the "StructuredDataClassifier"

[alexcombessie]

Hi there,
Hope all is well. Any update on this topic?
Cheers,
Alex

[haifeng-jin]

I am working on it. It seems it is a bug that it always use val_loss as objective.

[haifeng-jin]

This snippet works for me.

from tensorflow.keras.datasets import mnist
import kerastuner

import autokeras as ak

from tensorflow.keras import backend as K


def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1_score(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

# Prepare the dataset.
(x_train, y_train), (x_test, y_test) = mnist.load_data()
print(x_train.shape)  # (60000, 28, 28)
print(y_train.shape)  # (60000,)
print(y_train[:3])  # array([7, 2, 1], dtype=uint8)

# Initialize the ImageClassifier.
clf = ak.ImageClassifier(
    max_trials=3,
    objective=kerastuner.Objective('val_f1_score', direction='min'),
    metrics=[f1_score],
)
# Search for the best model.
clf.fit(x_train, y_train, epochs=10)
# Evaluate on the testing data.
print('Accuracy: {accuracy}'.format(
    accuracy=clf.evaluate(x_test, y_test)))

[haifeng-jin]

https://autokeras.com/tutorial/faq/
It is now on the official website.

[alexcombessie]

Excellent! Thanks!

[ywtaccelerator]

This snippet works for me.

from tensorflow.keras.datasets import mnist
import kerastuner

import autokeras as ak

from tensorflow.keras import backend as K


def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1_score(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

# Prepare the dataset.
(x_train, y_train), (x_test, y_test) = mnist.load_data()
print(x_train.shape)  # (60000, 28, 28)
print(y_train.shape)  # (60000,)
print(y_train[:3])  # array([7, 2, 1], dtype=uint8)

# Initialize the ImageClassifier.
clf = ak.ImageClassifier(
    max_trials=3,
    objective=kerastuner.Objective('val_f1_score', direction='min'),
    metrics=[f1_score],
)
# Search for the best model.
clf.fit(x_train, y_train, epochs=10)
# Evaluate on the testing data.
print('Accuracy: {accuracy}'.format(
    accuracy=clf.evaluate(x_test, y_test)))

Hi, I guess we should let the "direction" be 'max', right?

[haifeng-jin]

@ywtaccelerator Yes, you are right. Would you help us change the faq in this file? Thank you!
https://github.com/keras-team/autokeras/blob/master/docs/templates/tutorial/faq.md

[ywtaccelerator]

@haifeng-jin Of course, I would love to help! But I found there are some other problems when using customized metrics (e.g. the way above we use f1_score) to select the best model, and I guess it would better to address them first before updating the FAQ:

(1) After the fitting process using the above code and exporting the model by:

model = clf.export_model()
model.save("model_autokeras.h5")

then there will be an error "ValueError: Unknown metric function: f1_score" if we would like to load the saved model by the following code directly (which is in https://autokeras.com/tutorial/export/):

from tensorflow.keras.models import load_model
loaded_model = load_model("model_autokeras", custom_objects=ak.CUSTOM_OBJECTS)

I guess that is because the .h5 file contains our customized function, so currently I found the following way can be a solution and the loaded_model works well for me:

my_custom_objects={'f1_score': f1_score}
my_custom_objects.update(ak.CUSTOM_OBJECTS)
loaded_model = load_model("model_autokeras", custom_objects=my_custom_objects)

(2) Another problem that I would really like to find a solution soon is that the above code snippet of using f1_score as customized metric cannot resume a previously killed run by running the same code again.

For example, after finishing running the above code snippet of using f1_score as customized metric, if I would like to run for two more trails, I simply let "max_trials=5" in the ImageClassifier and run the same snippet again, but there will be an error "ValueError: Unknown metric function: f1_score". I guess the error is caused by the same problem as in (1), i.e., in the process of loading model or checkpoint we should pass {'f1_score': f1_score} to the custom_objects, so I guess the error might not be that hard to fix.

(The following is the whole traceback of the error:

Traceback (most recent call last):
File "", line 1, in
File "/snap/pycharm-professional/211/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/snap/pycharm-professional/211/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/colotu01/csi4900tf2/20200728_try/try_to_continue_fitting.py", line 50, in
metrics=[f1_score],
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/tasks/image.py", line 85, in __init__
*kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/tasks/image.py", line 24, in __init__
*kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/auto_model.py", line 136, in __init__
*kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/tuners/task_specific.py", line 102, in __init__
*kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/tuners/greedy.py", line 237, in __init__
*kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/engine/tuner.py", line 40, in __init__
super().__init__(oracle, hypermodel, *kwargs)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/kerastuner/engine/tuner.py", line 104, in __init__
overwrite=overwrite)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/kerastuner/engine/base_tuner.py", line 71, in __init__
self.directory, self.project_name, overwrite=overwrite)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/kerastuner/engine/oracle.py", line 312, in _set_project_dir
self.reload()
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/kerastuner/engine/oracle.py", line 337, in reload
super(Oracle, self).reload(self._get_oracle_fname())
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/kerastuner/engine/stateful.py", line 64, in reload
self.set_state(state)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/tuners/greedy.py", line 84, in set_state
self.hypermodel = graph.Graph.from_config(state['hypermodel'])
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/graph.py", line 205, in from_config
blocks = [blocks_module.deserialize(block) for block in config['blocks']]
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/graph.py", line 205, in
blocks = [blocks_module.deserialize(block) for block in config['blocks']]
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/blocks/__init__.py", line 37, in deserialize
printable_module_name='hypermodels')
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 360, in deserialize_keras_object
return cls.from_config(cls_config)
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/engine/head.py", line 72, in from_config
config['metrics'] = deserialize_metrics(config['metrics'])
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/autokeras/engine/head.py", line 23, in deserialize_metrics
deserialized.append(tf.keras.metrics.deserialize(metric))
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/metrics.py", line 3443, in deserialize
printable_module_name='metric function')
File "/home/colotu01/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 378, in deserialize_keras_object
'Unknown ' + printable_module_name + ': ' + object_name)
ValueError: Unknown metric function: f1_score

)

Please let me know if you need any additional information, and I am happy to help!

[haifeng-jin]

For the first one, you got the right solution. For the second one, I think it is a big. Would you file a separate issue about it? Name it "resume fails with custom objective". Thank you!

[ywtaccelerator]

@haifeng-jin Thank you for your reply, I have filed a separate issue https://github.com/keras-team/autokeras/issues/1257 named "Resumption fails with custom objective", please feel free to let me know if you need any additional information, I hope we can find a solution soon, thanks!