Keras: Lambda Layer doesn't serialize to JSON or YAML

[RaffEdwardBAH]

It appears networks with Lambda Layers can't be saved though the to_json or to_yaml methods

To implement graves style skip connections for a classification problem I'm using a lambda layer to get the results from the last time step

Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape)

where

def last_step_shape(input_shape):
    shape = list(input_shape)
    assert len(shape) == 3
    return tuple((shape[0], shape[2]))

This model does successfully start training, but I can't save the architecture specification. The model is defined using the Graph API. Looks something like

model = Graph()
model.add_input(name='input', batch_input_shape=(batchsize, maxlen), dtype='int')
model.add_node(Embedding(257, embed_size, input_length=maxlen, name='embedding', input='input')

prev_name = 'embedding'

#add intermediate hidden LSTM layers, they need to return their sequences. 
for l in range(num_layers-1):
    ls = str(l)
    if l > 0:
        model.add_node(LSTM(lstm_layer_size, return_sequences=True), 
                       name='_f'+ls, inputs=[prev_name, 'embedding'])
        model.add_node(Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape), name='f'+ls, input='_f'+ls)
    else:
        model.add_node(LSTM(lstm_layer_size, return_sequences=True), 
                       name='_f'+ls, input=prev_name)
        model.add_node(Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape), name='f'+ls, input='_f'+ls)
    prev_name = '_f'+ls

#add last LSTM layer
ls = str(num_layers-1)
if num_layers > 1:
    model.add_node(LSTM(lstm_layer_size), name='f'+ls, inputs=[prev_name, 'embedding'])
    model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', inputs=['f'+str(x) for x in range(num_layers)], merge_mode='concat')
else:
    model.add_node(LSTM(lstm_layer_size, dropout_W=0.5, dropout_U=0.5, W_regularizer=l2(1e-5)), name='f'+ls, input=prev_name)
    model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', input='f'+ls, merge_mode='concat')

model.add_output(name='output', input='sigmoid')

# try using different optimizers and different optimizer configs
optimizer = Adam(lr=0.001, clipnorm=grad_clip)

model.compile(optimizer, {'output': 'binary_crossentropy'})

I get the error

UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-25-e4d9f49c02c8> in <module>()
----> 5 json_string = model.to_json()
      6 # open(name+'.json', 'w').write(json_string)
      7 

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in to_json(self, **kwargs)
   2375             'config': config,
   2376         }
-> 2377         return json.dumps(model_config, default=get_json_type, **kwargs)
   2378 
   2379     def to_yaml(self, **kwargs):

/usr/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)
    248         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    249         separators=separators, encoding=encoding, default=default,
--> 250         sort_keys=sort_keys, **kw).encode(obj)
    251 
    252 

/usr/lib/python2.7/json/encoder.pyc in encode(self, o)
    205         # exceptions aren't as detailed.  The list call should be roughly
    206         # equivalent to the PySequence_Fast that ''.join() would do.
--> 207         chunks = self.iterencode(o, _one_shot=True)
    208         if not isinstance(chunks, (list, tuple)):
    209             chunks = list(chunks)

/usr/lib/python2.7/json/encoder.pyc in iterencode(self, o, _one_shot)
    268                 self.key_separator, self.item_separator, self.sort_keys,
    269                 self.skipkeys, _one_shot)
--> 270         return _iterencode(o, 0)
    271 
    272 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

UnicodeDecodeError: 'utf8' codec can't decode byte 0x83 in position 28: invalid start byte

[fchollet]

You're probably using an outdated version of Keras. Lambdas can be
serialized as of 1.0. Also I recommend not using "Graph" (which is
deprecated) and use the functional API instead.

On 2 May 2016 at 11:24, RaffEdwardBAH [email protected] wrote:

It appears networks with Lambda Layers can't be saved though the to_json
or to_yaml methods

To implement graves style skip connections for a classification problem
I'm using a lambda layer to get the results from the last time step

Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape)

where

def last_step_shape(input_shape):
shape = list(input_shape)
assert len(shape) == 3
return tuple((shape[0], shape[2]))

This model does successfully start training, but I can't save the
architecture specification. The model is defined using the Graph API. Looks
something like

model = Graph()
model.add_input(name='input', batch_input_shape=(batchsize, maxlen), dtype='int')
model.add_node(Embedding(257, embed_size, input_length=maxlen, name='embedding', input='input')
prev_name = 'embedding'

add intermediate hidden LSTM layers, they need to return their sequences. for l in range(num_layers-1):

ls = str(l)
if l > 0:
    model.add_node(LSTM(lstm_layer_size, return_sequences=True),
                   name='_f'+ls, inputs=[prev_name, 'embedding'])
    model.add_node(Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape), name='f'+ls, input='_f'+ls)
else:
    model.add_node(LSTM(lstm_layer_size, return_sequences=True),
                   name='_f'+ls, input=prev_name)
    model.add_node(Lambda(lambda x: x[:,-1,:], output_shape=last_step_shape), name='f'+ls, input='_f'+ls)
prev_name = '_f'+ls

add last LSTM layerls = str(num_layers-1)if num_layers > 1:

model.add_node(LSTM(lstm_layer_size), name='f'+ls, inputs=[prev_name, 'embedding'])
model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', inputs=['f'+str(x) for x in range(num_layers)], merge_mode='concat')else:
model.add_node(LSTM(lstm_layer_size, dropout_W=0.5, dropout_U=0.5, W_regularizer=l2(1e-5)), name='f'+ls, input=prev_name)
model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', input='f'+ls, merge_mode='concat')

model.add_output(name='output', input='sigmoid')

try using different optimizers and different optimizer configsoptimizer = Adam(lr=0.001, clipnorm=grad_clip)

model.compile(optimizer, {'output': 'binary_crossentropy'})

I get the error

---

UnicodeDecodeError Traceback (most recent call last)
in ()
----> 5 json_string = model.to_json()
6 # open(name+'.json', 'w').write(json_string)
7

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in to_json(self, _kwargs)
2375 'config': config,
2376 }
-> 2377 return json.dumps(model_config, default=get_json_type, *_kwargs)
2378
2379 def to_yaml(self, *kwargs):

/usr/lib/python2.7/json/init.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, *_kw)
248 check_circular=check_circular, allow_nan=allow_nan, indent=indent,
249 separators=separators, encoding=encoding, default=default,
--> 250 sort_keys=sort_keys, *_kw).encode(obj)
251
252

/usr/lib/python2.7/json/encoder.pyc in encode(self, o)
205 # exceptions aren't as detailed. The list call should be roughly
206 # equivalent to the PySequence_Fast that ''.join() would do.
--> 207 chunks = self.iterencode(o, _one_shot=True)
208 if not isinstance(chunks, (list, tuple)):
209 chunks = list(chunks)

/usr/lib/python2.7/json/encoder.pyc in iterencode(self, o, _one_shot)
268 self.key_separator, self.item_separator, self.sort_keys,
269 self.skipkeys, _one_shot)
--> 270 return _iterencode(o, 0)
271
272 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

UnicodeDecodeError: 'utf8' codec can't decode byte 0x83 in position 28: invalid start byte

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2582

[RaffEdwardBAH]

I'm using the head version of Keras.

I also re-implemented it with the functional API (literally just now) and get the same error.

[dimazhiyanov]

I am seeing the same error

pip show keras

Metadata-Version: 2.0
Name: Keras
Version: 1.0.2

[dimazhiyanov]

core.py
Potential problem?

    if isinstance(self._output_shape, python_types.LambdaType):
        if py3:
            output_shape = marshal.dumps(self._output_shape.__code__)     #missing.decode('raw_unicode_escape')
        else:
            output_shape = marshal.dumps(self._output_shape.func_code)
        output_shape_type = 'lambda'

[roryhr]

@dimazhiyanov Yes. marshal is generating bytes which json complains about later.

[robertremus]

I've got the same problem -- I cannot successfully marshall/unmarshall a model that uses a lambda within a vector merge, neither for JSON nor for YAML.

[yongthelaoma]

Didn't work for me as well. I'm building lstms with attention where I used Lambda layer to get some bilinear dot product done.

[yongthelaoma]

Update: I used pickle to dump the model config dictionary object (including functions) and it worked for me. Not pretty, but I guess that's one of Python 2.x's problems.

[drhyrum]

explicitly, this workaround appears to work

pickle.dump( model.get_config(), open( modelname, 'w') ) )

and

newmodel = model.from_config( pickle.load( modelname,'r') )

[fchollet]

And you guys have a suggestion? Note that if you use pickle you can't serialize that to JSON, either.

[yongthelaoma]

The perfect solution would be a modification to json.dumps() method...
IMHO pratically, if we have to work within current python versions, two solutions exist:
1) Provide a model.to_pickle() method -- quick solution
2) Seperate the serialization of the model itself and the marshaled functions, and combine them together while loading the model.

[drhyrum]

Alternatively, since this appears to be an issue in json.dumps in marshalling some unicode strings, one could change this line in the Lambda get_config() method
https://github.com/fchollet/keras/blob/master/keras/layers/core.py#L455

from

 function = marshal.dumps(self.function.func_code).decode('raw_unicode_escape')

to

 function = base64.b64encode( marshal.dumps(self.function.func_code).decode('raw_unicode_escape') )

to make json happy. Of course, you'd similarly add a base64.b64decode in the from_config class method, and mirror the operations for the py3 case also.

[rmcgibbo]

Base64 encoding in the Lambda.get_config() would seem like a great solution.

[fchollet]

If this turns out to work, then we could do that indeed.

On 2 June 2016 at 19:44, Robert T. McGibbon [email protected]
wrote:

Base64 encoding in the Lambda.get_config() would seem like a great
solution.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2582#issuecomment-223477646,
or mute the thread
https://github.com/notifications/unsubscribe/AArWb_u3jjMxURFzsVkkLXwAEnVLvM-0ks5qH5UGgaJpZM4IVlXh
.

[fchollet]

We would need a way to ensure backwards compatibility though.

BTW JSON serialization of Lambda is absolutely supposed to work and in fact
we have tests for it. I am successfully able to save various kinds of
Lambda and Merge layers. If you have concrete examples of cases that don't
work, that would be interesting to check out.

On 2 June 2016 at 19:52, François Chollet francois.[email protected]
wrote:

If this turns out to work, then we could do that indeed.

On 2 June 2016 at 19:44, Robert T. McGibbon [email protected]
wrote:

Base64 encoding in the Lambda.get_config() would seem like a great
solution.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2582#issuecomment-223477646,
or mute the thread
https://github.com/notifications/unsubscribe/AArWb_u3jjMxURFzsVkkLXwAEnVLvM-0ks5qH5UGgaJpZM4IVlXh
.

[DingKe]

I encountered this problem too. For my case, it's due to that negative index. I used something like that:

output_shape=lambda shape: (shape[0], shape[-1]). 

When I changed to

output_shape=lambda shape: (shape[0], shape[2]). 

(assuming shape has 3 dims), then everything just worked well.

[robertremus]

These are the 2 lambdas I use for defining the shape and the merge mode, respectively (and which I seem to be unable to de-serialize):

    def merge_output_shape(self):
        return 1,

    def merge_mode(self, sigmoid_tensors):
        if len(sigmoid_tensors) % 2 != 1:
            raise Exception("len(sigmoid_tensors) must be uneven!")

        nb_zeroes = 0
        nb_ones = 0
        for sigmoid_tensor in sigmoid_tensors:
            if T.ge(sigmoid_tensor, 0.5):
                nb_zeroes += 1
            else:
                nb_ones += 1

        if nb_zeroes > nb_ones:
            return T.as_tensor_variable(0)
        else:
            return T.as_tensor_variable(1)

[robertremus]

Because this still didn't work for me in Keras 1.0.4, I implemented the lambda's functionality as a custom Keras layer (for which serialization/de-serialization) then works out-of-the-box.

[davidawad]

Can confirm that this is still an issue when using a Lambda layer on keras 1.2.1 and windows 10.

Using the Lambda layer makes the to_json() call fail but switching to batch normalization works just fine.

model = Sequential()

# model.add(Lambda(lambda x: x / 255. - .5, input_shape=INPUT_DIMENSIONS))
model.add(BatchNormalization(input_shape=INPUT_DIMENSIONS, axis=1))
. . . 
# POST PROCESSING, SAVE MODEL TO DISK
with open('model.json', 'w') as json_file:
    json_file.write(model.to_json())

Hope this helps someone struggling with this in the future.

[cWren0110]

I am running keras 2.0.3, with a Tensorflow-gpu 1.1.0 backend on Windows 10 with Python 3.5.3 and am still seeing this issue. I can save the weights without issue, but if I use model.to_json() or model.save() I get:

File "yad2k.py", line 255, in _main
json_string = model.to_json()
File "D:Users...Anaconda3libsite-packageskerasenginetopology.py", line 2546, in to_json
model_config = self._updated_config()
File "D:Users...Anaconda3libsite-packageskerasenginetopology.py", line 2513, in _updated_config
config = self.get_config()
File "D:Users...Anaconda3libsite-packageskerasenginetopology.py", line 2250, in get_config
layer_config = layer.get_config()
File "D:Users...Anaconda3libsite-packageskeraslayerscore.py", line 668, in get_config
function = func_dump(self.function)
File "D:Users...Anaconda3libsite-packageskerasutilsgeneric_utils.py", line 173, in func_dump
code = marshal.dumps(func.__code__).decode('raw_unicode_escape')
UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 195-196: truncated uXXXX

Please let me know if I can include any more to help with the issue.

[Joshuaalbert]

I can confirm this is still an issue, with Keras head. Shall I post an example?

[yadonskov]

I have a problem with keras 2.0.8 intalled via pip, python3.5 during ModelCheckpoint:
TypeError('Not JSON Serializable:', slice(0, 10, 1))

with following code:
````
def make_soft_mixture_of_experts(number_of_features=20, number_of_hidden_layers=2):

input_layer_x = keras.layers.Input(shape=(number_of_features, ))

experts_x_masks = [
    slice(0, 10, 1),
    slice(16, 20, 1),
    slice(10, 20, 1)
]

nn_gating = make_gating_slekeleton(
    input_layer=input_layer_x,
    number_of_experts=len(experts_x_masks),
    total_number_of_features=number_of_features,
    number_of_hidden_layers=number_of_hidden_layers
)

expert_nns = []
for n_expert, x_mask in enumerate(experts_x_masks):
    #nn_gating_output = keras.layers.Lambda(lambda x: x[slice(n_expert, n_expert + 1, 1)], output_shape=(1, ))(nn_gating)
    nn_expert = make_expert_skeleton(
        input_layer=input_layer_x,
        input_filter_slice=x_mask,
        number_of_hidden_layers=number_of_hidden_layers
    )
    expert_nns.append(nn_expert)


#experts_layer = keras.layers.concatenate(expert_nns)
mixture_layer = keras.layers.concatenate(expert_nns + [nn_gating])
output_layer_y = keras.layers.Lambda(lambda x: K.sum(x[:, :3] * x[:, 3:6]), output_shape=(1, ))(mixture_layer)
#output_layer_y = keras.layers.multiply([experts_layer, nn_gating])


mixture_of_experts = keras.models.Model(inputs=[input_layer_x], outputs=output_layer_y)
mixture_of_experts.summary()

mixture_of_experts.compile(
                    loss=keras.losses.binary_crossentropy,
                    optimizer="adam",
                    metrics=[
                        keras.metrics.MSE,
                        keras.metrics.MAE,
                        keras.metrics.binary_accuracy,
                        mean_pred
                    ]
)

return mixture_of_experts

def get_slice_len(slice_val):
return (slice_val.stop - slice_val.start) // slice_val.step

def make_expert_skeleton(input_layer, input_filter_slice, number_of_hidden_layers):
expert_number_of_features = get_slice_len(input_filter_slice)
nn_expert = keras.layers.Lambda(lambda x: x[:, input_filter_slice], output_shape=(expert_number_of_features, ))(input_layer)
for _ in range(0, number_of_hidden_layers):
nn_expert = keras.layers.Dense(units=(3*expert_number_of_features//2), activation="relu")(nn_expert)
nn_expert = keras.layers.Dropout(0.3)(nn_expert)
#gating_layer = keras.layers.concatenate([nn_expert, gating_par
```t])
#nn_expert = GatedDense(units=1, activation="sigmoid")(gating_layer)
nn_expert = keras.layers.Dense(units=1, activation="sigmoid")(nn_expert)

#debug_nn = keras.models.Model(inputs=input_layer, outputs=nn_expert)
#debug_nn.summary()
return nn_expert

def make_gating_slekeleton(input_layer, number_of_experts, total_number_of_features, number_of_hidden_layers):
nn_gating = keras.layers.Dense(units=(3 * total_number_of_features // 2), activation="relu")(input_layer)
nn_gating = keras.layers.Dropout(0.3)(nn_gating)
for _ in range(0, number_of_hidden_layers - 1):
nn_gating = keras.layers.Dense(units=(3 * total_number_of_features // 2), activation="relu")(nn_gating)
nn_gating = keras.layers.Dropout(0.3)(nn_gating)
nn_gating = keras.layers.Dense(units=number_of_experts, activation="softmax")(nn_gating)

# debug_nn = keras.models.Model(inputs=input_layer, outputs=nn_gating)
# debug_nn.summary()
return nn_gating

`

Do I need to delete all lambdas?

[AziziShekoofeh]

I also have the same problem with Lambda layer. I am trying to add temperature to my softmax by:

model.add(Lambda(lambda x: x / temp))

but getting the error that Lambda object is not JSON serializable. Any suggestion to solve this or any suggestion to add the temperature to softmax?

Keras: 2.1.3
Tensorflow: 1.5
No GPU.

[kapsl]

Same problem here:
Windows 7
Tensorflow 1.5. GPU
Keras which comes with tensorflow

But I don't have lambda layers. Its nothing special:

Layer (type) Output Shape Param #

input_1 (InputLayer) (None, 1650, 7) 0

---

conv1d_1 (Conv1D) (None, 1650, 64) 1408

---

max_pooling1d_1 (MaxPooling1 (None, 825, 64) 0

---

conv1d_2 (Conv1D) (None, 825, 40) 7720

---

max_pooling1d_2 (MaxPooling1 (None, 413, 40) 0

---

conv1d_3 (Conv1D) (None, 413, 20) 2420

---

conv1d_4 (Conv1D) (None, 413, 18) 1098

---

max_pooling1d_3 (MaxPooling1 (None, 207, 18) 0

---

conv1d_5 (Conv1D) (None, 207, 18) 990

---

conv1d_6 (Conv1D) (None, 207, 20) 1100

---

up_sampling1d_1 (UpSampling1 (None, 414, 20) 0

---

conv1d_7 (Conv1D) (None, 414, 40) 2440

---

up_sampling1d_2 (UpSampling1 (None, 828, 40) 0

---

conv1d_8 (Conv1D) (None, 828, 64) 7744

---

up_sampling1d_3 (UpSampling1 (None, 1656, 64) 0

---

conv1d_9 (Conv1D) (None, 1650, 7) 3143

---

conv1d_10 (Conv1D) (None, 1650, 7) 154

Total params: 28,217
Trainable params: 28,217
Non-trainable params: 0

[DariusAlexander]

Also getting issues with JSON when trying to save or fit model- suspect it's down to the use of Lambda.

Keras 2.1.3
Python 2.7.10
OSX 10.12.6
TF GPU backend

    model.fit(X_train, Y_train, batch_size=batch_num, nb_epoch=num_epoch, validation_split=val_split, callbacks=[reduce_lr, check_point, early_stop, tb]) #
  File "/Library/Python/2.7/site-packages/keras/engine/training.py", line 1669, in fit
    validation_steps=validation_steps)
  File "/Library/Python/2.7/site-packages/keras/engine/training.py", line 1226, in _fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/Library/Python/2.7/site-packages/keras/callbacks.py", line 76, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/Library/Python/2.7/site-packages/keras/callbacks.py", line 418, in on_epoch_end
    self.model.save(filepath, overwrite=True)
  File "/Library/Python/2.7/site-packages/keras/engine/topology.py", line 2573, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/Library/Python/2.7/site-packages/keras/models.py", line 112, in save_model
    }, default=get_json_type).encode('utf8')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/Library/Python/2.7/site-packages/keras/models.py", line 96, in get_json_type
    raise TypeError('Not JSON Serializable:', obj)
TypeError: ('Not JSON Serializable:', Dimension(8))

Dimension(8) is probably related to how many lambdas I'm doing.

[kapsl]

Hi,
i found a way how to solve my issue: I was reading the dimension of a layer to parametrize the next layer:

curr_seq_length = x._shape[1] kernel_size = curr_seq_length - self._sequence_length + 1 x = Conv1D(self._nr_features, kernel_size=[kernel_size], activation='relu', padding='valid')(x)

the problem was that curr_seq_length was from type Dimension - and it seems that there something changed in the lastest tensorflow-integrated keras version. And the json serializer doesn't know how to deal with the Dimension object.

Changing the line to
curr_seq_length = x._shape[1].value
solves the problem...

[philtomson]

Are Lambda layers serialized to Python bytecode? We have a GSoC student trying to read in the Keras models that have been serialized to JSON and running into issues with Lambda layers - I would guess the function in the JSON is serialized to Python bytecode and that crossing the language barrier would not be possible (without doing some sophisticated compiler-ish type things)