Keras: How to add reshape layer following embedding(for the purpose of 2d convolution)?

[rantsandruse]

Hello,

I am trying to do a 2d convolution after an embedding layer for some nlp/sentiment analysis work, and I found the following thread to be very helpful:
https://github.com/fchollet/keras/issues/233

However, presumably due to keras update, the "reshape" function no longer works:

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten, Reshape, Merge
from keras.layers.embeddings import Embedding

sequential = Sequential()
embedding_size = 64 
nb_feature_maps = 32
max_features = 300
maxlen = 10 

sequential.add(Embedding(max_features, embedding_size))
sequential.add(Reshape(1,maxlen,embedding_size)) 

This results in error as follows:

TypeError                                 Traceback (most recent call last)
<ipython-input-8-9d892949f64b> in <module>()
     10 
     11 sequential.add(Embedding(max_features, embedding_size))
---> 12 sequential.add(Reshape(1,maxlen,embedding_size))

TypeError: __init__() takes exactly 2 arguments (4 given)

When I checked the documentation of reshape it is suggested that double parenthesis be used around the reshape function:

sequential.add(Reshape((1,maxlen,embedding_size))) 

And this leads to yet another error:

TypeError                                 Traceback (most recent call last)
<ipython-input-9-b57f40d4dce3> in <module>()
     10 
     11 sequential.add(Embedding(max_features, embedding_size))
---> 12 sequential.add(Reshape((1,maxlen,embedding_size)))

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/models.pyc in add(self, layer)
    143                  output_shapes=[self.outputs[0]._keras_shape])
    144         else:
--> 145             output_tensor = layer(self.outputs[0])
    146             if type(output_tensor) is list:
    147                 raise Exception('All layers in a Sequential model '

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/engine/topology.pyc in __call__(self, x, mask)
    483         if inbound_layers:
    484             # this will call layer.build() if necessary
--> 485             self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
    486             input_added = True
    487 

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/engine/topology.pyc in add_inbound_node(self, inbound_layers, node_indices, tensor_indices)
    541         # creating the node automatically updates self.inbound_nodes
    542         # as well as outbound_nodes on inbound layers.
--> 543         Node.create_node(self, inbound_layers, node_indices, tensor_indices)
    544 
    545     def get_output_shape_for(self, input_shape):

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/engine/topology.pyc in create_node(cls, outbound_layer, inbound_layers, node_indices, tensor_indices)
    149             output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))
    150             # TODO: try to auto-infer shape if exception is raised by get_output_shape_for
--> 151             output_shapes = to_list(outbound_layer.get_output_shape_for(input_shapes[0]))
    152         else:
    153             output_tensors = to_list(outbound_layer.call(input_tensors, mask=input_masks))

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_output_shape_for(self, input_shape)
    204 
    205     def get_output_shape_for(self, input_shape):
--> 206         return (input_shape[0],) + self._fix_unknown_dimension(input_shape[1:], self.target_shape)
    207 
    208     def call(self, x, mask=None):

/Users/xxx/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in _fix_unknown_dimension(self, input_shape, output_shape)
    193                 known *= dim
    194 
--> 195         original = np.prod(input_shape, dtype=int)
    196         if unknown is not None:
    197             if known == 0 or original % known != 0:

/Users/xxx/anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in prod(a, axis, dtype, out, keepdims)
   2490         except AttributeError:
   2491             return _methods._prod(a, axis=axis, dtype=dtype,
-> 2492                                   out=out, keepdims=keepdims)
   2493         return prod(axis=axis, dtype=dtype, out=out)
   2494     else:

/Users/xxx/anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in _prod(a, axis, dtype, out, keepdims)
     33 
     34 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
---> 35     return umr_prod(a, axis, dtype, out, keepdims)
     36 
     37 def _any(a, axis=None, dtype=None, out=None, keepdims=False):

TypeError: long() argument must be a string or a number, not 'NoneType'

More broadly, I understand that imdb_cnn.py provides a great example for 1d convolution following embedding, but I have yet to find an example that works with 2d convolution following embedding. (There is an example from the kaggle submission of "bags of words vs bags of popcorn", but it no longer works due to the same mishap of the "reshape" function). I would really appreciate it if such an example is provided.

Thanks a lot!

[joelthchao]

Specify input_length.

sequential.add(Embedding(max_features, embedding_size, input_length=maxlen))
sequential.add(Reshape((1, maxlen, embedding_size)))

[zhhongzhi]

@joelthchao
There is two sequences with different length in my model depend on the char-level embedding, thus I can't set the input_lengthparament, do you know any other solution?
Thanks for your attention.

[joelthchao]

@zhhongzhi Three ways to achieve it

1. If you are using Recurrent layer after embedding, then you can pad input sequence with 0s to make all input sequence in the length of max_length, then using mask to ignore those 0s.
2. Use a dynamic network framework, such as TF-fold to build the model
3. Not specify input_length, but doing pooling operation (e.g. avg pooling) on this dynamic length axis.

I would say it depends on your task to choose the best way. I am looking forward to some other good ways to achieve it.