Nmt: Tensorflow: Troubles with BeamSearchDecoder - dynamic_decode

[thomasschmied]

Hi,
I am implementing a seq2seq model with bidirectional multilstm layers with attention and beamsearch. (Only posted the necessary code to keep it simple)

# helper to create the layers
def make_lstm(rnn_size, keep_prob):
lstm = tf.nn.rnn_cell.LSTMCell(rnn_size, initializer = tf.random_uniform_initializer(-0.1, 0.1, seed=2))
lstm_dropout = tf.nn.rnn_cell.DropoutWrapper(lstm, input_keep_prob = keep_prob)
return lstm_dropout

# helper to create the attention cell with
def decoder_cell(dec_cell, rnn_size, enc_output,  lengths):
    attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(
            num_units              = rnn_size,
            memory                 = enc_output,
            memory_sequence_length = lengths,
            normalize                  = True,
            name  = 'BahdanauAttention')

    return tf.contrib.seq2seq.AttentionWrapper(
            cell                 = dec_cell,
            attention_mechanism  = attention_mechanism,
            attention_layer_size = rnn_size) 

Encoder:

# foward 
cell_fw = tf.nn.rnn_cell.MultiRNNCell([make_lstm(rnn_size, keep_prob) for _ in range(n_layers)])




# backward    
cell_bw = tf.nn.rnn_cell.MultiRNNCell([make_lstm(rnn_size, keep_prob) for _ in range(n_layers)])





enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw,
                                                        cell_bw,
                                                        rnn_inputs,
                                                        sequence_length=sequence_length,
                                                        dtype=tf.float32,
                                                        )

enc_output = tf.concat(enc_output,-1)

Decoder:

beam_width = 10
dec_cell = tf.nn.rnn_cell.MultiRNNCell([make_lstm(rnn_size, keep_prob) for _ in range(num_layers)])
output_layer = Dense(vocab_size, kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))
dec_cell = decoder_cell(dec_cell, rnn_size, enc_output, text_length)


with tf.variable_scope("decode"):
    # (dec_embed_input comes from another function but should not be 
    #   relevant in this context. )
    helper = tf.contrib.seq2seq.TrainingHelper(inputs = dec_embed_input, 
                                               sequence_length = summary_length,
                                               time_major = False)

    decoder = tf.contrib.seq2seq.BasicDecoder(cell = dec_cell,
                                              helper = helper,
                                              initial_state = dec_cell.zero_state(batch_size, tf.float32),
                                              output_layer = output_layer)

    logits = tf.contrib.seq2seq.dynamic_decode(decoder=decoder, 
                                           output_time_major=False, 
                                           impute_finished=True, 
                                           maximum_iterations=max_summary_length)




enc_output = tf.contrib.seq2seq.tile_batch(enc_output, multiplier=beam_width)
enc_state = tf.contrib.seq2seq.tile_batch(enc_state, multiplier=beam_width)
text_length = tf.contrib.seq2seq.tile_batch(text_length, multiplier=beam_width)

dec_cell = tf.nn.rnn_cell.MultiRNNCell([make_lstm(rnn_size, keep_prob) for _ in range(num_layers)])
dec_cell = decoder_cell(dec_cell, rnn_size, enc_output, text_length)

start_tokens = tf.tile(tf.constant([word2ind['<GO>']], dtype = tf.int32), [batch_size], name = 'start_tokens')

with tf.variable_scope("decode", reuse = True):


    decoder = tf.contrib.seq2seq.BeamSearchDecoder( cell=dec_cell,
                                                    embedding=embeddings,
                                                    start_tokens=start_tokens,
                                                    end_token=end_token,
                                                    initial_state=dec_cell.zero_state(batch_size = batch_size*beam_width , dtype = tf.float32),
                                                    beam_width=beam_width,
                                                    output_layer=output_layer,
                                                    length_penalty_weight=0.0)



    logits = tf.contrib.seq2seq.dynamic_decode(decoder=decoder, 
                                           output_time_major=False, 
                                           impute_finished=True, 
                                           maximum_iterations=max_summary_length)`

In this line:

logits = tf.contrib.seq2seq.dynamic_decode(decoder=decoder, 
                                               output_time_major=False, 
                                               impute_finished=True, 
                                               maximum_iterations=max_summary_length)

I get following error:

ValueError: Shapes must be equal rank, but are 3 and 2 for 'decode_1/decoder/while/Select_4' (op: 'Select') with input shapes: [64,10], [64,10,256], [64,10,256].

Tensorflow: 1.6.0,
batch_size = 64,
rnn_size = 256

I cannot figure out why the fist one from different shape.
Has anyone got some advice on how to fix this?

Thanks a lot in advance!

[brandenchan]

Hi, I had exactly the same problem. Would you mind sharing how you fixed it?

[zackkhan]

^ also running into this issue as well. Any help would be appreciated!

[kev1876]

Set the impute_finished=False, I don't know why, but it work for me.

[MeteorsHub]

It seems that tf.where doesn't support dimension extension. If impute_finished is True, the helper will check if the decoding is finished and perform different inputs, that is tf.where(is_finished, x, y). But is_finished is [batch_size, beam_size], and x y are [batch_size, beam_size, num_input_units].

So I think this is a bug

[nlp4whp]

Set the impute_finished=False, I don't know why, but it work for me.

It seems that tf.where doesn't support dimension extension. If impute_finished is True, the helper will check if the decoding is finished and perform different inputs, that is tf.where(is_finished, x, y). But is_finished is [batch_size, beam_size], and x y are [batch_size, beam_size, num_input_units].

So I think this is a bug

Thank you so much, you save my weekend !