Keras: Using Lambda Layer to implement bi-directional rnn

[liweitj47]

I am currently trying to implement various rnn structures, I followed the code in the examples,"imdb_bidirectional_lstm.py", however, I want to do sequence labeling, which not only needs the last step, but also needs the complete sequence.
So here's the issue, I want the forward lstm and the backward lstm to be strictly aligned in the same order of the input sequence. But if I only set the go_backwards=True, it will return a backward sequence, which is not the same order as the input. Since the code in seya and other repos haven't updated for a long time, and seem difficult to apply, I added a Lambda Layer upon the backward lstm. Will it work as I expected? Thanks!
106 model.add_node(LSTM(200,return_sequences=True),name='lstm',input='dropou t')
107 model.add_node(LSTM(200,return_sequences=True,go_backwards=True),name='l stm2',input='dropout')
108 model.add_node(Lambda(lambda x:x[:,::-1,:]),name='reverse',input='lstm2' )
109 model.add_node(Dropout(0.5),name='dropout2',inputs=['lstm','reverse'],me rge_mode='concat')
110 model.add_node(TimeDistributedDense(nb_label),name='tdd',input='dropout2 ')

[EderSantana]

Sorry for the delay updating Seya.

This Lambda layer will indeed reverse you sequences again, which is what you need to align forward and backward passes.

Test it and let me know if it worked out ok.

[ymcui]

probably right.
but be careful when you use masked input.
AFAIK, current version of Keras has some issues in using backward RNN with masked input.

[EderSantana]

@ymcui is right! To cope with that, since you are in a graph, you should use two regular forward LSTMs and revert the input and mask as a second input. In that case your new Lambda layer will be useful as well.

[ymcui]

@EderSantana When it comes to bidirectional RNN, do you have any ideas on how to realize a shared Embedding layer with masked input?
I thought it would be easy by just using the add_shared_node(), and put Embedding layer in it.
But I have noticed that the add_shared_node() will not produce an masked output.
So I have to set two Embedding layers, and set the second one to have weights=graph.nodes['first_embedding'].get_weights() and trainable=False, am I right?

[EderSantana]

The idea would be to have two separate embedding layers and make sure they point to the same symbolic Tensor. Or, much better, you should create your own embedding layer that outputs two looked-up series: one with left padding and the second with right padding and reversed.

@farizrahman4u wrote the Siamese layers, maybe he has an opinion on that as well. Also, if that doesn't exist yet open up an issue called "Embedding layers for Bidirectional RNN" and post your solution there. This should be a useful future reference for other users.

[ymcui]

@EderSantana OK, i'll post it right away.