Nmt: Attention with beam Search

[piyank22]

I have written my own code with reference to this wonderful tutorial and I am not able to get results when using attention with beam search as per my understanding in the class AttentionModel the _build_decoder_cell function creates separate decoder cell and attention wrapper for inference mode , assuming this ,

`with tf.name_scope("Decoder"):

mem_units = 2*dim
dec_cell = tf.contrib.rnn.BasicLSTMCell( 2*dim )
beam_width = 3
out_layer = Dense( output_vocab_size )

with tf.name_scope("Training"):
    attn_mech = tf.contrib.seq2seq.BahdanauAttention( num_units = mem_units,  memory = enc_rnn_out, normalize=True)
    attn_cell = tf.contrib.seq2seq.AttentionWrapper( cell = dec_cell,attention_mechanism = attn_mech ) 

    batch_size = tf.shape(enc_rnn_out)[0]
    initial_state = attn_cell.zero_state( batch_size = batch_size , dtype=tf.float32 )
    initial_state = initial_state.clone(cell_state = enc_rnn_state)

    helper = tf.contrib.seq2seq.TrainingHelper( inputs = emb_x_y , sequence_length = seq_len )
    decoder = tf.contrib.seq2seq.BasicDecoder( cell = attn_cell, helper = helper, initial_state = initial_state ,output_layer=out_layer ) 
    outputs, final_state, final_sequence_lengths= tf.contrib.seq2seq.dynamic_decode(decoder=decoder,impute_finished=True)

    training_logits = tf.identity(outputs.rnn_output )
    training_pred = tf.identity(outputs.sample_id )

with tf.name_scope("Inference"):

    enc_rnn_out_beam   = tf.contrib.seq2seq.tile_batch( enc_rnn_out   , beam_width )
    seq_len_beam       = tf.contrib.seq2seq.tile_batch( seq_len       , beam_width )
    enc_rnn_state_beam = tf.contrib.seq2seq.tile_batch( enc_rnn_state , beam_width )

    batch_size_beam      = tf.shape(enc_rnn_out_beam)[0]   # now batch size is beam_width times

    # start tokens mean be the original batch size so divide
    start_tokens = tf.tile(tf.constant([27], dtype=tf.int32), [ batch_size_beam//beam_width ] )
    end_token = 0

    attn_mech_beam = tf.contrib.seq2seq.BahdanauAttention( num_units = mem_units,  memory = enc_rnn_out_beam, normalize=True)
    cell_beam = tf.contrib.seq2seq.AttentionWrapper(cell=dec_cell,attention_mechanism=attn_mech_beam,attention_layer_size=mem_units)  

    initial_state_beam = cell_beam.zero_state(batch_size=batch_size_beam,dtype=tf.float32).clone(cell_state=enc_rnn_state_beam)

    my_decoder = tf.contrib.seq2seq.BeamSearchDecoder( cell = cell_beam,
                                                       embedding = emb_out,
                                                       start_tokens = start_tokens,
                                                       end_token = end_token,
                                                       initial_state = initial_state_beam,
                                                       beam_width = beam_width
                                                       ,output_layer=out_layer)

    beam_output, t1 , t2 = tf.contrib.seq2seq.dynamic_decode(  my_decoder,
                                                                maximum_iterations=maxlen )

    beam_logits = tf.no_op()
    beam_sample_id = beam_output.predicted_ids

with tf.name_scope("Training_op"):

loss = tf.losses.softmax_cross_entropy(  tf.one_hot( Y,output_vocab_size ) , training_logits )
train_op = tf.train.AdamOptimizer().minimize(loss)

correct = tf.reduce_sum( tf.cast( tf.equal( training_pred , Y ) , dtype=tf.float32 ) ) / maxlen

`

when i call beam _sample_id after training i am not getting correct result.

my guess is that we are supposed to using the same attention wrapper but that is not possible since we have to tile_sequence for beam search to be used.

Any insights / suggestion would be much appreciated.

[piyank22]

i have asked this same question on stackoverflow

[piyank22]

@lmthang @songmeixu @cclauss @ebrevdo @martinwicke , please help my confusion.

[ghost]

Hi @piyank22, I'm in the same boat as you.
If i'm not wrong, both AttentionMechanism and AttentionWrapper contain trainable layers.
So, we must reuse the trained instances in beam-search inference too.

I tried tf.variable_scope(tf.get_variable_scope(), reuse=True) but it is not enough.

I'm guessing , save-restore using separate graphs for train & infer models is the only way of reusing here.
Please let me know if you have success with any approaches.

[piyank22]

Yes thanks that worked for me , initially i had made my own assumptions and convinced myself that making multiple graphs was incorrect and didnt try that approach , on seeing your comment giving it a second thought i tried to list all the trainable parameters for both the graphs and they were same , so its intuitively correct for me that we can now restore the variables to a different graph with same trainable parameters. Thanks a lot

[ebrevdo]

It can be done but you have to be very careful about how scopes and layers
are reused. We moved to the separate graph approach because the code ended
up being cleaner and we didn't have to fudge with variable scopes at all.
Also it's necessary for any real application to have separate graphs.

On Sep 5, 2017 9:25 AM, "Piyank Sarawagi" [e-mail] wrote:

Closed #93 https://github.com/tensorflow/nmt/issues/93.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/nmt/issues/93#event-1235239837, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABtim_XwZHXwe1EVRKgws9OTwg6U1qXrks5sfXYBgaJpZM4PLFhD
.

[piyank22]

Yes in my attempt to do it with scopes was repeatedly failing miserably, for which i had to learn and understand how separate graphs made sense and in the end its all worth the effort put in as i can now really see how the previous method lacked clarity and this one is intuitively so much better.

[justDL]

@piyank22
sorry to bother you?i am a new learner. can you give me a example about how you fix it?thanks in advance

[piyank22]

can you give me exactly what u are facing difficulties with and what type of error u r getting

[justDL]

I have send it to your mailbox.Maybe it is in your junk.Could you spend some time to read it? @piyank22

[o-github-o]

@piyank22 sorry to bother you,I use seq2seq in the same way as you did ,using attention and basicDecoder in the training phase, and use BeamSearchDecoder and attention in the infer phase ,can you tell me how to solve it and make separate graph, thanks a lot.

[piyank22]

https://github.com/piyank22/Blog/blob/master/TheJourneyOfNLP.ipynb
you can skip to the part with the title, "clever way to implement two seperate graphs with the same trainable variables". ask me if u seek more help.

[o-github-o]

@piyank22 Thank yo a lot ,my problem has been solved with your method(the seperate graph) .I have got in trouble a lot of time... all is over,thanks again.

[lizaigaoge550]

@o-github-o can you send your code to me? Thank you very much

[ghost]

@piyank22 how do you create separate graph for training and inference?

[colin0000007]

@lizaigaoge550 https://github.com/piyank22/Blog/blob/master/TheJourneyOfNLP.ipynb
skip to the "clever way to implement two seperate graphs with the same trainable variables".

[colin0000007]

@hastirad https://github.com/piyank22/Blog/blob/master/TheJourneyOfNLP.ipynb
skip to the "clever way to implement two seperate graphs with the same trainable variables"

[allenyllee]

I give another solution in stackoverflow: https://stackoverflow.com/a/61820477/1851492

You can use tf.cond() to create different pathes between training and inference stage:

def get_tile_batch(enc_output, source_sequence_length, enc_state, useBeamSearch):
    enc_output = tf.contrib.seq2seq.tile_batch(enc_output, multiplier=useBeamSearch)
    source_sequence_length = tf.contrib.seq2seq.tile_batch(source_sequence_length, multiplier=useBeamSearch)
    enc_state = tf.contrib.seq2seq.tile_batch(enc_state, multiplier=useBeamSearch)
    return enc_output, source_sequence_length, enc_state

## for beam search: at training stage, use tile_batch multiplier = 1,
## at infer stage, use tile_batch multiplier = useBeamSearch
## tile_batch is just duplicate every sample in a batch,
## so it'll change batch_size to batch_size * useBeamSearch at runtime once batch_size was determined
enc_output, source_sequence_length, enc_state = tf.cond(
    self.on_infer, # is inference stage?
    lambda: get_tile_batch(enc_output, source_sequence_length, enc_state, useBeamSearch=useBeamSearch),
    lambda: get_tile_batch(enc_output, source_sequence_length, enc_state, useBeamSearch=1)
)

# attention mechanism
attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(num_units=rnn_size, memory=enc_output, memory_sequence_length=source_sequence_length)
dec_cell = tf.contrib.seq2seq.AttentionWrapper(dec_cell, attention_mechanism)
## for beam search: change batch_size to batch_size * useBeamSearch at infer stage
decoder_initial_state = tf.cond(
    self.on_infer, # is inference stage?
    lambda: dec_cell.zero_state(batch_size=batch_size * useBeamSearch, dtype=tf.float32),
    lambda: dec_cell.zero_state(batch_size=batch_size * 1, dtype=tf.float32)
)
enc_state = decoder_initial_state.clone(cell_state=enc_state)