Nmt: Attention Heatmap?

[ngcabang]

How do I create the attention heat map based from Attention model like the image below?

PS: I can access the graph through tfdebug but I don't know what I am searching for

[oahziur]

Hi @ngcabang

You can start with the code for generate the image summary here. That will generate a heat map without sentences on the left and top margin. The image will be displayed in the image section on Tensorboard.

[evanweissburg]

@oahziur I run into two issues with the template code for Attention summary generation:

When beam search is enabled, the above throws an error. I'm assuming beam search and alignment history are not compatible right now.

When I disable beam search, I get the following error:

Traceback (most recent call last):
  File "/home/nave01314/IdeaProjects/tf-nmt/main.py", line 19, in <module>
    infer_model = model_builder.create_infer_model(hparams)
  File "/home/nave01314/IdeaProjects/tf-nmt/model_builder.py", line 125, in create_infer_model
    tgt_vocab_table=tgt_vocab_table)
  File "/home/nave01314/IdeaProjects/tf-nmt/models.py", line 41, in __init__
    self.attn_summary = self.make_attention_summary(final_context_state)
  File "/home/nave01314/IdeaProjects/tf-nmt/models.py", line 192, in make_attention_summary
    attention_images = (final_context_state.alignment_history.stack())
AttributeError: 'tuple' object has no attribute 'stack'

[Hazepusik]

Hi, @ngcabang !
I faced the same problem now. Have you solved your issue?
Thanks

[ttrouill]

Same here, I would like to get each alignment map for each of the top-k translations when beam search is activated, is this planned any time soon? If not, what should be changed for this to work @oahziur ?

[oahziur]

@ttrouill @nave01314 Actually, I am not sure if the attention image for BeamSearch will be correct out of the box since we traversed the decoding tree in BeamSearchDecoder to get the final results, so I guess we will need to do the similar gather_tree operation in the end. We can revisit this after beam_search and alignment_history are compatible.

@nave01314 Can you print the final_context_state.alignment_history see what's inside the tuple?

[ttrouill]

@oahziur Ok thanks for the precisions, is there an already existing branch working on beam_search and alignment_history compatibility that I can follow/contribute?

[oahziur]

@ttrouill I don't know if there is any existing branch or pull request on this, but I think you can follow this issue to get update if there is any.

[ttrouill]

@oahziur Looks like beam_search and alignment_history compatibility issue will soon be resolved and merged

[DaoD]

@nave01314 the same problem, how do you solve it?
I do not use beam search, but I still can't get the alignment.

[JulianZhang]

@oahziur Dear, I can see the attention image by tensorboard now , But there this is only one image for one step, does that mean the image is for all training data?

How can I get image for one given sentences?