Keras: How come input_shape accepts (None, n)?

[carlthome]

This model can be trained and used for predictions, but what does it actually predict? Is it one-to-one, many-to-many, or what? Batch size is set in model.fit(...) but what is the RNN sequence length when input_shape = (None, n)? Is it one? Is it automatically determined by the input shape when calling model.fit(...)? Shouldn't this raise an error?

n = 100
model = Graph()
model.add_input(name='input', input_shape=(None, n))
model.add_node(LSTM(n, return_sequences=True), name='lstm', input='input')
model.add_node(TimeDistributedDense(n, activation='sigmoid'), name='tdd', input='lstm')
model.add_output(name='output', input='tdd')
model.compile(loss={'output': 'mse'}, optimizer='rmsprop')

[fchollet]

A None dimension in a shape tuple means that the network will be able to
accept inputs of any dimension. For instance, input_shape=(None, 32)
indicates variable-length sequences of 32-dimensional vectors. Of course,
it's not always possible to have such free dimensions (for instance it's
not possible to have variable-length sequences with TensorFlow, but it is
with Theano).

On 23 March 2016 at 13:26, Carl Thomé [email protected] wrote:

This model can be trained and used for predictions, but what does it
actually predict? Batch size is set in model.fit() but what is the LSTM
sequence length in this case? Is it one?

n = 100
model = Graph()
model.add_input(name='input', input_shape=(None, n))
model.add_node(LSTM(n, return_sequences=True), name='lstm', input='input')
model.add_node(TimeDistributedDense(n, activation='sigmoid'), name='tdd', input='lstm')
model.add_output(name='output', input='tdd')
model.compile(loss={'output': 'mse'}, optimizer='rmsprop')

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2054

[carlthome]

Wow, that's incredibly powerful. So the RNN is unrolled as needed, dictated by the data? This feature could be documented.

add_input() currently reads:

input_shape: a tuple of integers, the expected shape of the input samples.

but could probably say:

input_shape: a tuple of integers, the expected shape of the input samples. Set a dimension to None to accept inputs of variable-length (Theano backend only).

[NasenSpray]

@carlthome: It only really concerns RNNs

input_length: Length of input sequences, to be specified when it is constant. This argument is required if you are going to connect...

[carlthome]

@NasenSpray only ;)

But yes, perhaps it would muddy up the API documentation.

input_shape: a tuple of integers, the expected shape of the input samples. For RNNs, set a dimension to None to accept inputs of variable-length (Theano backend only).

[fisayo-omilana]

Can a dynamic length input_dim=None be applied to a simple neural network? Specifically, Keras Sequential model. I've been running into errors when trying to employ the same concept. I've already seen the documentation that seems to support this functionality:

https://keras.io/getting-started/functional-api-guide/

But when I do the following...

model = Sequential()
model.add(Dense(num_feat, input_dim = None, kernel_initializer = 'normal', activation='relu'))
model.add(Dense(num_feat, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dropout(.2))
model.add(Dense(num_feat, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dropout(.2))
model.add(Dense(num_feat, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dropout(.2))
model.add(Dense(ouput.shape[1], kernel_initializer = 'normal', activation = 'linear'))

...I get this error:
ValueError: ('Only Theano variables and integers are allowed in a size-tuple.', (None, 63), None)

Any help or ideas would be greatly appreciated!!

[carlthome]

Only some layers support variably shaped inputs and Dense is not really one of them. Convolution layers and recurrent layers such as Conv2D and LSTM would function but Dense has to allocate a trainable weight matrix of shape N x M, where N is the number of input features and M is the number of output units.

Each input example will be multiplied by this matrix to form correctly shaped outputs, and it's the weights in the matrix that are tweaked to make the outputs look more like your target labels when you call model.fit with training data.

With that said, this works fine for me on TensorFlow 1.9 as the weight matrix isn't allocated until it has to be:

import numpy as np
from tensorflow.python.keras.models import *
from tensorflow.python.keras.layers import *

N = np.random.randint(1, 10)
M = np.random.randint(32, 128)

model = Sequential()
model.add(Dense(M, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile('sgd', 'binary_crossentropy')
x = np.zeros((100, N))
y = np.zeros((100, 1))
model.fit(x, y)