Tensor2tensor: Attention visualization

[stefan-it]

Hi,

in the appendix of Attention Is All You Need some very nice visualizations of the attention mechanism can be seen.

How can I create those visualizations for my own dataset/language pair?

Would be great if the authors can give some hints :)

[lukaszkaiser]

LLion, who wrote the scripts to do the visualizations, is on vacation this week. I'll leave this open and ping when he's back.

[stefan-it]

@lukaszkaiser That would be great :)

[vishalnus]

@lukaszkaiser I too will wait for this. Thanks very much for the guidance

[rsepassi]

It seems that dot_product_attention (which is used in multihead_attention which is used in the Transformer) calls attention_image_summary which adds an image summary op, which should be written out to TensorBoard during training. Have you checked the image summaries tab in TensorBoard during training?

[rsepassi]

https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py#L285

[stefan-it]

I meant these kind of visualizations:

:)

[lukaszkaiser]

An iPython notebook with our viz code is included in the new release 1.2.0, give it a try!