Coremltools: bi-lstm does not pass state

[jellevanwezel]

I have a problem with passing the initial state for a sequence with my bi-lstm network.

I am trying to do image sequence predictions with coreml from a trained keras network.
I split my network into 2 networks but trained it on one with Keras.
As suggested in this post: #146

I have set the weights from the trained network to both parts of the split network and converted the networks to coreml, which seems to work just fine.

Then I proceed as follows:
1: I feed the network my images, they get flattened to a (256) array.
2: I feed these arrays one by one to the bi-lstm network.

For clarity this is the bi-lstm network as outputted by keras:
bidirectional_1 (Bidirection (None, 44, 512)
bidirectional_2 (Bidirection (None, 44, 1024)
time_distributed_1 (TimeDist (None, 44, 37)

Here time_distributed_1 is an output dense layer.
It does classify most of my images correctly but it does not pass the c and h output states nor the reverse output states. instead, it passes an array full with zeros.

This leads the network to think it is in its first state of the sequence all the time resulting in a worse classification rate when compared with the keras networks.

I looked at this post: #147
and executed that code but this seems to have the same problem: c and h are always zero arrays in the output map.

How do I get the initial states from my network?

[esdu]

👍 I was about the create a new issue and noticed this. I'm having the same problem. Any ideas?

(I also have code to reproduce the fact that it passes an array full of zeros, let me know if anyone wants to see it.)

[aseemw]

c and h are inputs to the mlmodel, so you should be able to set them to whatever values you want.
Does the generated mlmodel not have additional inputs and outputs corresponding to h and c ?

[lauriebyrum]

Hi @aseemw
There is definitely something going wrong either in the tools or in coreml itself. Here's some code:
import keras
from keras.models import Sequential
import coremltools
import numpy as np

model = Sequential()
model.add(keras.layers.InputLayer(input_shape=(1, 2)))
model.add(keras.layers.Bidirectional(keras.layers.LSTM(3, batch_size=2)))
coreml_model = coremltools.converters.keras.convert(model, input_names='input')
coreml_model.save('myfile.mlmodel')

model = coremltools.models.MLModel('myfile.mlmodel')
inputarray = np.ones((2, 1,2), dtype=np.float32)
predictions = model.predict({'input': inputarray})
print(predictions)

Note that the c and h output params (printed when predictions are printed) are always 0. That's wrong.

(The tfcoreml code doesn't work with lstms -- see https://github.com/tf-coreml/tf-coreml/issues/124 -- so i'm using the keras converter as a model to help me manually convert. That's why i don't really care that the code above uses random weights)

[aseemw]

Yes this does seem to be a bug, getting all zeros for the hidden states output from a bi-dir LSTM. We will look into it.

[aseemw]

While it is true that the h and c outputs are not populated properly, why do you need to know the values for them?

Since output of the bi-lstm is exactly same as the h state (concatenated from the two forward and reverse LSTMs), so for a stack of bi-lstms, there is no problem, as the state will be available in the output vector.

And since the states are passed within the two LSTMs automatically, there is no need to access c directly.

Fo instance, the following code snippet converts correctly,

from keras.layers import *
from keras.layers.wrappers import Bidirectional, TimeDistributed
from keras.models import Model
import coremltools
import numpy as np

input_data = Input(name='the_input', shape=(None, 10))
y1 = Bidirectional(LSTM(64,return_sequences=True, activation='relu'),
                  merge_mode='concat')(input_data)
y2 = Bidirectional(LSTM(32,return_sequences=True, activation='relu'),
                  merge_mode='concat')(y1)
y = TimeDistributed(Dense(30, name="y_pred", activation="softmax"))(y2)

model = Model(input_data, y)
model.set_weights([np.random.rand(*w.shape) * 0.2 - 0.1 for w in model.get_weights()])
print(model.summary())

x = np.random.rand(1,3,10)
kout = model.predict(x)
mlmodel = coremltools.converters.keras.convert(model)
mlmodel.save('/tmp/bb.mlmodel')
cx = np.transpose(x,[1,0,2])
cout = mlmodel.predict({"input1": cx}, useCPUOnly=True)['output1']
print('keras: ',kout.shape, np.mean(kout[:]))
print('coreml: ',cout.shape, np.mean(cout[:]))
print(np.sum(np.abs(kout[:] - cout[:])))

Do you have a case where for the first bi-lstm in the stack, you want to provide a non zero state vector (non zero h_forward/c_forward and h_forward/c_forward)?

@jellevanwezel the model summary of the code snippet above is same as what you mentioned and that converts correctly. Do you have another code snippet that gives an error?

[lauriebyrum]

Honestly, I'm trying to figure out what is going on with https://github.com/apple/coremltools/issues/303 (which I just filed). I thought this was a clue, but maybe not.

[jellevanwezel]

While it is true that the h and c outputs are not populated properly, why do you need to know the values for them?

Since output of the bi-lstm is exactly same as the h state (concatenated from the two forward and reverse LSTMs), so for a stack of bi-lstms, there is no problem, as the state will be available in the output vector.

And since the states are passed within the two LSTMs automatically, there is no need to access c directly.

Fo instance, the following code snippet converts correctly,

from keras.layers import *
from keras.layers.wrappers import Bidirectional, TimeDistributed
from keras.models import Model
import coremltools
import numpy as np

input_data = Input(name='the_input', shape=(None, 10))
y1 = Bidirectional(LSTM(64,return_sequences=True, activation='relu'),
                  merge_mode='concat')(input_data)
y2 = Bidirectional(LSTM(32,return_sequences=True, activation='relu'),
                  merge_mode='concat')(y1)
y = TimeDistributed(Dense(30, name="y_pred", activation="softmax"))(y2)

model = Model(input_data, y)
model.set_weights([np.random.rand(*w.shape) * 0.2 - 0.1 for w in model.get_weights()])
print(model.summary())

x = np.random.rand(1,3,10)
kout = model.predict(x)
mlmodel = coremltools.converters.keras.convert(model)
mlmodel.save('/tmp/bb.mlmodel')
cx = np.transpose(x,[1,0,2])
cout = mlmodel.predict({"input1": cx}, useCPUOnly=True)['output1']
print('keras: ',kout.shape, np.mean(kout[:]))
print('coreml: ',cout.shape, np.mean(cout[:]))
print(np.sum(np.abs(kout[:] - cout[:])))

Do you have a case where for the first bi-lstm in the stack, you want to provide a non zero state vector (non zero h_forward/c_forward and h_forward/c_forward)?

@jellevanwezel the model summary of the code snippet above is same as what you mentioned and that converts correctly. Do you have another code snippet that gives an error?

If I remember correctly the model does convert. The problem was that when I fed the model my image sequence of 44 * 1 * 1 * 32 * 32 (Seq, B, C, H, W) images I would get a texture2DDescriptor error from coreML. I don't remember the exact error but I think it had something to do with the sequence being too long and taking too much memory. So I thought let's try and feed the model the images one by one (1, 1, 1, 32, 32) and passing the hidden states manually. But when I tried this I found that the hidden states were always zero. Hence my post here.

[aseemw]

the long texture array comes when the model tries to run on the GPU. If you use the "CPUOnly" flag, it should work fine.

The hidden states are still available as the "output" of the model, so that can be used (the "output" of an LSTM is essentially the hidden states themselves). And internal memory state "c" are never shared outside the LSTM layer.