Keras: Convolutional LSTM

[jasper95]

I'm new to Keras and no experience on implementing ConvNets and LSTM. I need to implement the model described in this paper http://arxiv.org/pdf/1506.04214v1.pdf. This question is already asked here.

Can i achieve this with Keras?

[KeironO]

Yes, there is. @anayebi has added some additional layers to Keras here:

https://github.com/anayebi/keras-extra

[tboquet]

I think you will also be interested by #443. Currently, keras-extra does not implement a convolutional LSTM. It provides tools to process sequences of 2D images.

[jasper95]

@tboquet @KeironO thanks for your replies man. Can you give me some steps/code (just a short one 10-15 lines would do) using the API to implement this model?

[KeironO]

Can you give me a short use case as to what you want to achieve?
On 21 Feb 2016 2:26 am, "jasper95" [email protected] wrote:

@tboquet https://github.com/tboquet @KeironO
https://github.com/KeironO thanks man for your replies man. Can you
give me some steps/code (just a short one 10-15 lines would do) using the
API to implement this model?

—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/1773#issuecomment-186721771.

[jasper95]

My Input is a 3D grid cells.
I'm doing a sequence modeling using Convolutional LSTM encoder-decoder framework.

Classic LSTM

formula:

Convolutional LSTM

formula:

**Note: As stated on the paper, ' * ' denotes convolutional operator and 'o' denotes element-wise multiplication.

Like FC-LSTM, ConvLSTM can also be adopted as a building block for more complex structures.
For our spatiotemporal sequence forecasting problem, we use the structure shown in Fig. 3 which
consists of two networks, an encoding network and a forecasting network. Like in [21], the initial
states and cell outputs of the forecasting network are copied from the last state of the encoding
network. Both networks are formed by stacking several ConvLSTM layers. As our prediction target
has the same dimensionality as the input, we concatenate all the states in the forecasting network
and feed them into a 1x1 convolutional layer to generate the final prediction.
We can interpret this structure using a similar viewpoint as [23]. The encoding LSTM compresses
the whole input sequence into a hidden state tensor and the forecasting LSTM unfolds this hidden
state to give the final prediction. This structure is also similar to the LSTM future predictor model in [21] except that our input and output elements are all 3D tensors which preserve all the spatial information. Since the network has multiple stacked ConvLSTM layers, it has strong representational power which makes it suitable for giving predictions in complex dynamical systems like the precipitation nowcasting problem we study here.

That's it man :).

[EderSantana]

@jasper95 consider giving Seya's ConvGRU a try:
https://github.com/EderSantana/seya/blob/master/seya/layers/conv_rnn.py#L16-L20

It assumes that the image sequences comes as batch x time x dim and does the reshaping of dim to image size inside the for loop, and reshapes everything back again to 3D in the output. This way it can play together with any other RNN or Keras layer. If you modify to create a ConvLSTM and make a PR, I'd review your code carefully.

[KeironO]

+1 on the ConvGRU.

[jasper95]

cool @EderSantana

i tried to install seya and got this error

Processing seya
Writing /home/jasper/Documents/seya/setup.cfg
Running setup.py -q bdist_egg --dist-dir /home/jasper/Documents/seya/egg-dist-tmp-K8RaAB
warning: no previously-included files matching 'pycache' found under directory '_'
warning: no previously-included files matching '_.py[co]' found under directory '*'
Sorry: IndentationError: unindent does not match any outer indentation level (containers.py, line 179)
File "build/bdist.linux-x86_64/egg/seya/layers/draw.py", line 111
def _get_attention.trainable_weights(self, h, L, b, N):
^
SyntaxError: invalid syntax
creating /usr/local/lib/python2.7.9/lib/python2.7/site-packages/seya-0.1.0-py2.7.egg
Extracting seya-0.1.0-py2.7.egg to /usr/local/lib/python2.7.9/lib/python2.7/site-packages
Sorry: IndentationError: unindent does not match any outer indentation level (containers.py, line 179)
File "/usr/local/lib/python2.7.9/lib/python2.7/site-packages/seya-0.1.0-py2.7.egg/seya/layers/draw.py", line 111
def _get_attention.trainable_weights(self, h, L, b, N):
^
SyntaxError: invalid syntax
Adding seya 0.1.0 to easy-install.pth file
Installed /usr/local/lib/python2.7.9/lib/python2.7/site-packages/seya-0.1.0-py2.7.egg
Processing dependencies for seya==0.1.0
Finished processing dependencies for seya==0.1.0

Can you provide a sample usage of your ConvGRU on your docs? :))

[EderSantana]

hi @jasper95 , better docs are coming. For now check how to use it with this test:
https://github.com/EderSantana/seya/blob/master/tests/test_conv_rnn.py#L34-L48

[jamesmf]

I have an example of using the TimeDistributed wrapper with Convolution2D layers and GRU layers here:

https://github.com/jamesmf/mnistCRNN

It takes a sequence of images from MNIST and learns to predict the sum of the digits.

[KeironO]

Way ahead of you
On 25 May 2016 15:23, "jamesmf" [email protected] wrote:

I have an example of using the TimeDistributed wrapper with Convolution2D
layers and GRU layers here:

https://github.com/jamesmf/mnistCRNN

It takes a sequence of images from MNIST and learns to predict the sum of
the digits.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/1773#issuecomment-221591742

[ranjanZ]

Is PR for Conv-LSTM and Conv-GRU is still up ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, but feel free to re-open it if needed.

[ericwu09]

@jasper95, were you able to get a reliable implementation of the ConvLSTM?
Thanks

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[geniusbabak]

@jasper95 I'm looking at the same paper, did you get anything back?
I'm thinking of creating a Conv1DLSTM.

[IotaDraconis]

@geniusbabak, it appears that issue #1558 may also be helpful.