Keras: Merging multiple TimeDistributed(Dense()) not working? Missing attribute 'get_output_shape_at'

[carlthome]

I'm trying to rewrite a Graph model in the new functional API in Keras 1.0 but get errors when trying to merge TimeDistributed objects.

The old model was declared as:

m = Graph()
m.add_input(name='mix', input_shape=(None, features))
m.add_node(LSTM(features, return_sequences=True), name='lstm1', input='mix')
m.add_node(LSTM(features, return_sequences=True), name='lstm2', input='lstm1')
m.add_node(TimeDistributedDense(features, activation='sigmoid'), name='tdd1', input='lstm2')
m.add_node(TimeDistributedDense(features, activation='sigmoid'), name='tdd2', input='lstm2')
m.add_node(Lambda(function=lambda d: mask(d['tdd1'], d['tdd2'], d['mix']), output_shape=(None, features)), name='tfm1', inputs=['tdd1', 'tdd2', 'mix'], merge_mode='join')
m.add_node(Lambda(function=lambda d: mask(d['tdd2'], d['tdd1'], d['mix']), output_shape=(None, features)), name='tfm2', inputs=['tdd2', 'tdd1', 'mix'], merge_mode='join')
m.add_output(name='background', input='tfm1')
m.add_output(name='voice', input='tfm2')
m.compile(loss={'background': 'mse', 'voice': 'mse'}, optimizer='rmsprop')

Since the join mode has been removed I'm trying to merge the TimeDistributed objects with 'concat' instead, such as:

mix = Input(batch_shape=(sequences, timesteps, features))
lstm = LSTM(features, return_sequences=True)(LSTM(features, return_sequences=True)(mix))
tdd1 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
tdd2 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
voice = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(Merge([tdd1, tdd2, mix], mode='concat'))
background = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(Merge([tdd2, tdd1, mix], mode='concat'))
model = Model(input=[mix], output=[voice, background])
model.compile(loss='mse', optimizer='rmsprop')

Resulting in
AttributeError: 'TensorVariable' object has no attribute 'get_output_shape_at'

For reference here's what I'm trying to do (audio source separation):

[fchollet]

"Merge" is a layer, used to merge other layers. To merge _tensors_, use the
function "merge" (same arguments), which takes a list of tensors and
returns a tensor.

On 13 April 2016 at 06:18, Carl Thomé [email protected] wrote:

I'm trying to rewrite a Graph model in the new functional API in Keras 1.0
but get errors when trying to merge TimeDistributed objects.

The old model was declared as:

m = Graph()
m.add_input(name='mix', input_shape=(None, features))
m.add_node(LSTM(features, return_sequences=True), name='lstm1', input='mix')
m.add_node(LSTM(features, return_sequences=True), name='lstm2', input='lstm1')
m.add_node(TimeDistributedDense(features, activation='sigmoid'), name='tdd1', input='lstm2')
m.add_node(TimeDistributedDense(features, activation='sigmoid'), name='tdd2', input='lstm2')
m.add_node(Lambda(function=lambda d: mask(d['tdd1'], d['tdd2'], d['mix']), output_shape=(None, features)), name='tfm1', inputs=['tdd1', 'tdd2', 'mix'
], merge_mode='join')
m.add_node(Lambda(function=lambda d: mask(d['tdd2'], d['tdd1'], d['mix']), output_shape=(None, features)), name='tfm2', inputs=['tdd2', 'tdd1', 'mix'
], merge_mode='join')
m.add_output(name='background', input='tfm1')
m.add_output(name='voice', input='tfm2')
m.compile(loss={'background': 'mse', 'voice': 'mse'}, optimizer='rmsprop')

Since the join mode has been removed I'm trying to merge the
TimeDistributed objects with 'concat' instead, such as:

mix = Input(batch_shape=(sequences, timesteps, features))
lstm = LSTM(features, return_sequences=True)(LSTM(features, return_sequences=True)(mix))
tdd1 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
tdd2 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
voice = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(Merge([tdd1, tdd2, mix], mode='concat'))
background = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(Merge([tdd2, tdd1, mix], mode='concat'))
model = Model(input=[mix], output=[voice, background])
model.compile(loss='mse', optimizer='rmsprop')

Resulting in
_AttributeError: 'TensorVariable' object has no attribute
'get_output_shape_at'_

For reference here's what I'm trying to do:
[image: model]
https://cloud.githubusercontent.com/assets/1595907/14494082/5609aa4c-0189-11e6-92c5-0fe8824f2838.png

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2299

[bapla]

unrelated: but is there a way to make this nice graphs?

[fchollet]

Yes, see: http://keras.io/visualization/

On 13 April 2016 at 10:17, bapla [email protected] wrote:

unrelated: but is there a way to make this nice graphs?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2299#issuecomment-209551174

[carlthome]

@bapla

from keras.utils.visualize_util import plot
# Define model
plot(model, to_file='model.png')

[carlthome]

@fchollet Thanks, forgive my confusion regarding Merge and merge.

Working solution:

mix = Input(batch_shape=(sequences, timesteps, features))
lstm = LSTM(features, return_sequences=True)(LSTM(features, return_sequences=True)(mix))
tdd1 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
tdd2 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
voice = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(merge([tdd1, tdd2, mix], mode='concat'))
background = Lambda(function=lambda x: mask(x[0], x[1], x[2]))(merge([tdd2, tdd1, mix], mode='concat'))
model = Model(input=[mix], output=[voice, background])
model.compile(loss='mse', optimizer='rmsprop')

[bapla]

The plotting is very cool! Only that I can't get it working yet on python3.5

module 'pydot' has no attribute 'find_graphviz'

EDIT: I installed also a python2.7 environment on conda, on that pydot and the visualization works.

[xjcvip007]

@carlthome ,can you tell me what the 'mask' doing in your voice part?

[carlthome]

@xjcvip007, here's a good write-up: http://web.cse.ohio-state.edu/~dwang/papers/Wang.tia08.pdf

[xjcvip007]

@fchollet ，Thx~

[faroit]

@carlthome we are currently implementing some LSTM separation baselines and I wonder if these kind of network structure as mentioned from you above lead to some good results.

• Do you have some references except maybe the last Uhlich paper?
• Did you use any standardization (mean 0 var 1) of the STFT before feeding into the LSTM or just min-max scaling?
• did you experiment with different sequence length?