Keras: LSTM with different length of sequences

[OssamaAlshabrawy]

I have sequences of different lengths (e.g. 560000x128, 437000x128, 897000x128, etc)
I'm actually new to keras and to python as well. So I don't know exactly where is the problem in my first trial. I've loaded dummy data just to understand what's going on but I have an error. I'm using batch size = 1. So could you please help me to figure it out?
Here is my code:

import numpy as np
from keras.models import Sequential
from keras.layers import LSTM,Dense,Dropout
from keras.utils import to_categorical

## Loading dummy data
s1 = np.array([[1,2,3],[1,2,3],[1,2,3]])
s2 = [[4,5,6],[4,5,6]]
y1 = np.array(to_categorical([0,0,0],num_classes=2))
y2 = to_categorical([1,1],num_classes=2)
x_test = np.array([[1,3,2],[4,6,6],[1,2,3]])
y_test = np.array([0,1,0])

n_epochs=40

model = Sequential()
model.add(LSTM(100, input_shape=(None,3), return_sequences=True, activation='tanh', recurrent_activation='hard_sigmoid')) # 100 num of LSTM units
model.add(LSTM(100, activation='tanh', recurrent_activation='hard_sigmoid'))
model.add(Dense(2, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print(model.summary())

for _ in range(n_epochs):
    model.train_on_batch(s1,y1)
    model.train_on_batch(s2,y2)

scores = model.evaluate(x_test, y_test) # evaluate batch at a time..
print scores

And this is the error I've got:

Traceback (most recent call last):
  File "LSTM1.py", line 32, in <module>
    model.train_on_batch(s1,y1)
  File "/usr/lib/python2.7/site-packages/keras/models.py", line 933, in train_on_batch
    class_weight=class_weight)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1614, in train_on_batch
    check_batch_axis=True)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1307, in _stant's going on but I have an error. So could you please help me to figure it out?dardize_user_data
    _check_array_lengths(x, y, sample_weights)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 229, in _check_array_lengths
    'and ' + str(list(set_y)[0]) + ' target samples.')
ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 3 target samples.

[fchollet]

Axis 0 of the shapes of the input data arrays and target data arrays must be equal. I.e. for each input sample, there must be a corresponding target entry.

[OssamaAlshabrawy]

@fchollet Thanks so much for reply. However, when I edited as above to have equal input data arrays and data target arrays. I've got another error:

Traceback (most recent call last):
  File "LSTM1.py", line 32, in <module>
    model.train_on_batch(s1,y1)
  File "/usr/lib/python2.7/site-packages/keras/models.py", line 933, in train_on_batch
    class_weight=class_weight)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1614, in train_on_batch
    check_batch_axis=True)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1295, in _standardize_user_data
    exception_prefix='model input')
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 121, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (3, 3)

When I made the input to have 3 dimensions I've got the error:

Traceback (most recent call last):
  File "LSTM1.py", line 36, in <module>
    model.train_on_batch(s1,y1)
  File "/usr/lib/python2.7/site-packages/keras/models.py", line 933, in train_on_batch
    class_weight=class_weight)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1614, in train_on_batch
    check_batch_axis=True)
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 1299, in _standardize_user_data
    exception_prefix='model target')
  File "/usr/lib/python2.7/site-packages/keras/engine/training.py", line 121, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (1, 3, 1)

[bstriner]

For this kind of error, print the shapes of your arrays (print "S1: {}".format(s1.shape)) and the summary, and the mismatch should be straightforward. From what you have so far, your input should be 3 dims (3,3,1) and your output should be 2 dims (3,2). If you still have issues, post the summary, the shapes, and the current code.

Cheers

[fchollet]

Generally, check out the LSTM examples scripts in examples/, and look at the shape of the input data and target data.

[OssamaAlshabrawy]

@fchollet Thanks very much for interest. I really appreciated your help. I've solved the problem after having a deep look on the examples.

I just have one more question:
When I used train_on_batch, I could not print loss and acc for the training like in fit.
Actually, I could not find something like that in examples.
So, Could you please help me to figure out how to print loss and acc for each epoch like in fit?

[bstriner]

The loss and acc are just the return value of train_on_batch. If you're not using fit, it is up to you to print the values, although you could use classes from keras or other libraries to make progress bars.

You can take all the losses for an epoch and use np.mean to get an average loss for the batch and deal with it yourself. You can also create an instance of ProgbarLogger and replicate keras progress bars.

However, you are probably better off using fit and the functionality it provides. You only need to use train_on_batch in special circumstances.

https://keras.io/models/model/

train_on_batch returns:

Scalar training loss (if the model has a single output and no metrics) or list of scalars (if the model has multiple outputs and/or metrics). The attribute model.metrics_names will give you the display labels for the scalar outputs.

[OssamaAlshabrawy]

@bstriner Really thanks for your clarification. I really appreciate your explanation and it was definitely helpful.
I've used train_on_batch because the sequences lengths are different. I've found I could use pad_sequences but I prefer to feed the sequences as they are without padding.
I know guys I'm overwhelming you but actually I'm a PhD and I'm trying to enhance the performance and thus to understand what's behind each concept.
I'm know I'm picky but I still need to see the performance at each epoch to observe what is going on and find out more about the network performance after each epoch. So,
Is there any mean to evaluate the performance on each epoch? Noting that I'm not using fit and hence cannot use Callbacks

[bstriner]

The return value from train_on_batch is whatever loss and metrics you compiled into the model. You can just use those values in your own code however you see fit.

fit_generator is your friend. You can't have different sequence lengths in each batch with fit but you can with fit_generator. If you rewrite your code as a generator, you can let keras handle all the logging and other stuff.

Just write a function like this:

def generator():
  while True:
    yield x, y
model.fit_generator(generator(), ...)

Cheers