Autokeras: Model saving doesn't work for StructuredDataRegressor

[jayavanth]

Bug Description

I can't save the best model as h5 or even a tf file

Bug Reproduction

Code for reproducing the bug:

model = regressor.export_model()

model.save('testmodel.h5')

gives me

NotImplementedError: Save or restore weights that is not an instance of `tf.Variable` is not supported in h5, use `save_format='tf'` instead. Got a model or layer CategoricalEncoding with weights ....

It suggests I use tf instead of h5, but when I do that I get this error

model.save('testmodel', save_format='tf')

2020-04-03 14:44:31.041453: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
*** RuntimeError: Attempting to capture an EagerTensor without building a function.

Data used by the code:
-

Expected Behavior

Save the best model in either h5 or tf format

Setup Details

• OS type and version: Ubuntu 18.04.01
• Python: 3.6.6
• autokeras: 1.0.2
• scikit-learn: 0.20.3
• numpy: 1.18.2
• pandas: 1.0.1
• tensorflow: 2.2.0-dev20200401 (nightly)

Additional context

I also can't load the model json after saving it but I think that's related to #1023

[Kraks]

I play with the regression example from the autokeras website and have the same issue when saving to tf format, but a different one to h5 format:

Traceback (most recent call last):
  File "auto_ex.py", line 47, in <module>
    model.save("m.h5")
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
    signatures, options)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 627, in save_weights_to_hdf5_group
    weight_values = K.batch_get_value(weights)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3270, in batch_get_value
    return [x.numpy() for x in tensors]
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3270, in <listcomp>
    return [x.numpy() for x in tensors]
AttributeError: 'TrackableWeightHandler' object has no attribute 'numpy'

Versions:
OS: Linux Mint 19.3 kernel 5.0.0-32-generic
Python 3.6.9
tensorflow: 2.1.0
pandas: 1.0.3
autokeras: 1.0.2
scikit-learn: 0.22.2.post1
bumpy: 1.18.2

[jayavanth]

@Kraks Try this and see if it works. It works for me and I can save it but I can't load it back.

https://stackoverflow.com/questions/52357542/attributeerror-tensor-object-has-no-attribute-numpy

[cordeirojoao]

Hello,
I'm having the same issue :(
Does anyone was able to export a model create with StructuredDataClassifier api??

model3.save("modelo3", save_format='tf')

Traceback (most recent call last)
in
1 print("tipo do modelo gravado", type(model3))
2 #tf.keras.models.save_model(model1, "modelo1") ##
----> 3 model3.save("modelo3", save_format='tf')

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\engine\network.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options)
1006 """
1007 save.save_model(self, filepath, overwrite, include_optimizer, save_format,
-> 1008 signatures, options)
1009
1010 def save_weights(self, filepath, overwrite=True, save_format=None):

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\saving\save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
113 else:
114 saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 115 signatures, options)
116
117

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\saving\saved_model\save.py in save(model, filepath, overwrite, include_optimizer, signatures, options)
76 # we use the default replica context here.
77 with distribution_strategy_context._get_default_replica_context(): # pylint: disable=protected-access
---> 78 save_lib.save(model, filepath, signatures, options)
79
80 if not include_optimizer:

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in save(obj, export_dir, signatures, options)
907 object_saver = util.TrackableSaver(checkpoint_graph_view)
908 asset_info, exported_graph = _fill_meta_graph_def(
--> 909 meta_graph_def, saveable_view, signatures, options.namespace_whitelist)
910 saved_model.saved_model_schema_version = (
911 constants.SAVED_MODEL_SCHEMA_VERSION)

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in _fill_meta_graph_def(meta_graph_def, saveable_view, signature_functions, namespace_whitelist)
551 resource_initializer_ops = []
552 with exported_graph.as_default():
--> 553 object_map, resource_map, asset_info = saveable_view.map_resources()
554 for resource_initializer_function in resource_initializer_functions:
555 asset_dependencies = []

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in map_resources(self)
249 # pylint: disable=protected-access
250 with ops.device(obj._resource_device):
--> 251 new_resource = obj._create_resource()
252 # pylint: enable=protected-access
253 resource_map[obj.resource_handle] = new_resource

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\ops\lookup_ops.py in _create_resource(self)
1930 value_shape=self._value_shape,
1931 initial_num_buckets=self._initial_num_buckets,
-> 1932 name=self._name)
1933 if context.executing_eagerly():
1934 self._table_name = None

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\ops\gen_lookup_ops.py in mutable_dense_hash_table_v2(empty_key, deleted_key, value_dtype, container, shared_name, use_node_name_sharing, value_shape, initial_num_buckets, max_load_factor, name)
1111 value_shape=value_shape,
1112 initial_num_buckets=initial_num_buckets,
-> 1113 max_load_factor=max_load_factor, name=name)
1114 _result = _outputs[:]
1115 if _execute.must_record_gradient():

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\framework\op_def_library.py in _apply_op_helper(op_type_name, name, **keywords)
466 dtype=dtype,
467 as_ref=input_arg.is_ref,
--> 468 preferred_dtype=default_dtype)
469 except TypeError as err:
470 if dtype is None:

~AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\framework\ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
1278 graph = get_default_graph()
1279 if not graph.building_function:
-> 1280 raise RuntimeError("Attempting to capture an EagerTensor without "
1281 "building a function.")
1282 return graph.capture(value, name=name)

RuntimeError: Attempting to capture an EagerTensor without building a function.

[cordeirojoao]

I tried to use
import tensorflow as tf
tf.enable_eager_execution()

But autokeras is running of TF 2.0 that doesn't work that way, I think.
@jayavanth Can you please let me know how you did it?

@Kraks Try this and see if it works. It works for me and I can save it but I can't load it back.

https://stackoverflow.com/questions/52357542/attributeerror-tensor-object-has-no-attribute-numpy

[pacohernandezg]

Same problem here, when I try to save the exported model from the StructuredDataClassifier example, I get the following error:

RuntimeError: Attempting to capture an EagerTensor without building a function.

Thank you.

[KartikChugh]

I can save but loading gives me ValueError: Unknown layer: CategoricalEncoding

[ogabrielluiz]

I'm having the same problem.

[mattou78400]

I'm also having the issue with StructuredDataInput

[alexfdo]

I want to offer a solution that I use myself
It is based on Pickle + using the directories in the class structure at https://autokeras.com/structured_data_regressor/

 Class StructuredDataRegressor
 autokeras.StructuredDataRegressor (
   ...............................
      metrics = no
      project_name = "structd_data_regressor",
      max_trials = 100,
      directory = no,
     **kwargs
 )

If you do not specify your names for directories, then the system uses the default values, each time overwriting the information in the same directory ./structd_data_regressor
I use this variant to set my own names for directories.

model = ak.StructuredDataRegressor(max_trials=max_trials,
                                                ....................................................
                                               project_name=model_name,
                                               directory='data/models_saved_data/'
                                               )

model_name - this is the name under which the model will be saved. And this same model_name I indicate to pickle which model to load. I save the model completely, without any operation such as export_model.

[utkarshgupta137]

@alexfdo I'm facing the same issue. Could you elaborate on what you've done?
I tried to provide a directory myself, but still getting the same error.

[alexfdo]

@utkarshgupta137

save and load model

def load_pickle_model_from_file(name_file):
    with open(name_file, 'rb') as file:
        loaded_model = pickle.load(file)
    return loaded_model

def save_pickle_model_to_file(model, name_model):
    with open(Path(str(name_model)+".pickle"), 'wb') as file:
        pickle.dump(model, file)
    print(str(Path(str(name_model)+".pickle"))+' - model saved!')



md5-ee3c80a18a5a6fecb029fd10b9f2da06



model = ak.StructuredDataRegressor(max_trials=max_trials,
                                               column_names=data_cols,
                                               column_types=data_type,
                                               project_name=model_name
                                               directory='data/models_saved_data/'
                                               )

save after creating: model_name - your selected name on previouse stage

save_pickle_model_to_file(model, model_name)

[belkov0912]

I have the same error.
I'm adding DenseHashTable in DNNModel, intend to save embeddings in it, this is code:

`class DNNModel(tf.keras.Model):
def __init__(self, xxx):
self.table = self._create_embedding_table(vocabulary_list, embeddings)

def _create_embedding_table(self, vocab_list, embeddings):
    dimension = embeddings.shape[1]
    table = tf.lookup.experimental.DenseHashTable(tf.string, tf.float32, [2.0]*dimension, "empty_key", "deleted_key")
    # 对比，测试到底注释前后有没有效果
    keys = tf.constant([i for i in vocab_list], tf.string)
    values = tf.convert_to_tensor(embeddings, tf.float64)
    values = tf.cast(values, tf.float32)
    table.insert(keys, values)
    return table

def call(self, inputs, training=None, mask=None):
    fc_embeddings = self._input_layer(inputs)
    bert_embeddings = self._look_up(inputs)
    net = tf.concat([fc_embeddings, bert_embeddings], axis=1)
   ....

`
all is ok when runn it, but when i export model with the code tf.keras.models.save_model(model, FLAGS.servable_model_dir), it will raise the same error:RuntimeError: Attempting to capture an EagerTensor without building a function.

I debug the code, it likes serialize the DenseHashTable in dnnModel will raise Exceptions:

File "/Users/jiananliu/work/neirongrecom/model/ctr_model/wide_n_deep/wide_n_deep_keras_main.py", line 1021, in run
tf.keras.models.save_model(model, FLAGS.servable_model_dir, include_optimizer=False)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 138, in save_model
signatures, options)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save.py", line 78, in save
save_lib.save(model, filepath, signatures, options)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 951, in save
obj, export_dir, signatures, options, meta_graph_def)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 1027, in _build_meta_graph
options.namespace_whitelist)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 595, in _fill_meta_graph_def
object_map, resource_map, asset_info = saveable_view.map_resources()
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 270, in map_resources
new_resource = new_obj._create_resource()
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/ops/lookup_ops.py", line 1945, in _create_resource
name=self._name)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/ops/gen_lookup_ops.py", line 1113, in mutable_dense_hash_table_v2
max_load_factor=max_load_factor, name=name)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 470, in _apply_op_helper
preferred_dtype=default_dtype)
File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1307, in convert_to_tensor
raise RuntimeError("Attempting to capture an EagerTensor without "
RuntimeError: Attempting to capture an EagerTensor without building a function.

[haifeng-jin]

As I tested the latest version of AutoKeras can save the structureddataregressor fine.

[utkarshgupta137]

Issue solved in the latest version, thanks.

[Samar-080301]

Issue solved in the latest version, thanks.

In autokeras ==1.0.8?

[gbmax]

Saving model with StructruedDataRegressor example(California house ex.) produces same error..

I tried to save tf-format, keras-format, convert to sequential model and save, all results the same.

seq_model = tf.keras.Sequential(model.layers)
seq_model.save("model_test/h5_california.h5", save_format="h5")

haifeng-jin said saving model is OK, so I guess it is because of packages version difference.
Below is my test spec., different? or another package dependency?

< test spec. >
autokeras.__version = 1.0.8
tensorflow.__version = 2.3.0
python = 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)

< source - almost from autokeras example code except model.save >

from sklearn.datasets import fetch_california_housing
import numpy as np
import pandas as pd
import tensorflow as tf
import autokeras as ak

house_dataset = fetch_california_housing()
df = pd.DataFrame(
    np.concatenate((
        house_dataset.data, 
        house_dataset.target.reshape(-1,1)),
        axis=1),
    columns=house_dataset.feature_names + ['Price'])
train_size = int(df.shape[0] * 0.9)
df[:train_size].to_csv('train.csv', index=False)
df[train_size:].to_csv('eval.csv', index=False)
train_file_path = 'train.csv'
test_file_path = 'eval.csv'

import pandas as pd
import numpy as np
x_train = pd.read_csv('train.csv')
y_train = x_train.pop('Price')

x_test = pd.read_csv('eval.csv')
y_test = x_test.pop('Price')

reg = ak.StructuredDataRegressor(max_trials=3, overwrite=True)
reg.fit(x_train, y_train, epochs=10)

model = reg.export_model()
model.save("model_test/h5_california.h5", save_format="h5")

< Error Message >

---

NotImplementedError Traceback (most recent call last)
in
----> 1 seq_model.save("model_test/h5_california.h5", save_format="h5")
2 # tf.keras.models.save_model(model, "model_test/pb_california", save_format="tf")

~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options)
1977 """
1978 save.save_model(self, filepath, overwrite, include_optimizer, save_format,
-> 1979 signatures, options)
1980
1981 def save_weights(self,

~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
129 'or using save_weights.')
130 hdf5_format.save_model_to_hdf5(
--> 131 model, filepath, overwrite, include_optimizer)
132 else:
133 saved_model_save.save(model, filepath, overwrite, include_optimizer,

~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in save_model_to_hdf5(model, filepath, overwrite, include_optimizer)
117 model_weights_group = f.create_group('model_weights')
118 model_layers = model.layers
--> 119 save_weights_to_hdf5_group(model_weights_group, model_layers)
120
121 # TODO(b/128683857): Add integration tests between tf.keras and external

~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in save_weights_to_hdf5_group(f, layers)
633 for layer in sorted(layers, key=lambda x: x.name):
634 g = f.create_group(layer.name)
--> 635 weights = _legacy_weights(layer)
636 weight_values = K.batch_get_value(weights)
637 weight_names = [w.name.encode('utf8') for w in weights]

~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in _legacy_weights(layer)
882 'Save or restore weights that is not an instance of tf.Variable is '
883 'not supported in h5, use save_format=\'tf\' instead. Got a model '
--> 884 'or layer {} with weights {}'.format(layer.__class__.__name__, weights))
885 return weights

NotImplementedError: Save or restore weights that is not an instance of tf.Variable is not supported in h5, use save_format='tf' instead. Got a model or layer MultiCategoryEncoding with weights [, , ]

[haifeng-jin]

Does the tf format work?
As we tested, it should work.
The h5 may not at the moment.

[gbmax]

Thanks for good work.
Yes, tf format works.
let me ask another question.
I am using autokeras for generating prediction model efficiently(StructuredData).
For portable device deployment, I have to use TensorflowLite or Tensorflowjs.
I tested various combination of format and convert tool.
Unfortunately, even keras-generated model from keras homepage example fails to convert.
(https://keras.io/examples/structured_data/structured_data_classification_from_scratch/)
conversion script:

import tensorflowjs as tfjs
tfjs.converters.save_keras_model(model, 'model_test/tfjs_model')

BTW what I want to know clear is, autokeras-generated model would be just keras model,
so that if keras model is fine for converion to tflite or tfjs format, autokeras model would be OK also.
Am I right?

update : comment posted below was deleted because the issue is different, sorry for that : ).

[haifeng-jin]

I am not very familiar with tfjs or tf lite. I am not sure why it doesn't convert.

For the exported Keras model from AutoKeras, it is just a Keras Model.
It should convert if other Keras Models do.