Opennmt-py: Confusion about the `LayerNorm`s in `TransformerEncoder`

[anoidgit]

According to line 69, the input to the TransformerEncoderLayer will be normalized, but when I see line 120 and line 135, I found that the embedding to the first TransformerEncoderLayer will also be normalized. I am not sure whether this is right, since the original paper says nothing about this in section 5.4.

[guillaumekln]

This is an improvement coming from the reference implementation and used as default in their Transformer configuration. See:

https://github.com/tensorflow/tensor2tensor/blob/v1.6.5/tensor2tensor/layers/common_hparams.py#L110-L112

https://github.com/tensorflow/tensor2tensor/blob/v1.6.5/tensor2tensor/models/transformer.py#L1133-L1134

[anoidgit]

Thank you. It really helps. @guillaumekln

[Tristan-J]

This is an improvement coming from the reference implementation and used as default in their Transformer configuration. See:

https://github.com/tensorflow/tensor2tensor/blob/v1.6.5/tensor2tensor/layers/common_hparams.py#L110-L112

https://github.com/tensorflow/tensor2tensor/blob/v1.6.5/tensor2tensor/models/transformer.py#L1133-L1134

Hi Guillaumekln, still want to know why it's better... is there any mathematical explanation? Or it's according to the experimental results? Thanks!

[guillaumekln]

I believe it is according to experimental results from the original authors of the paper.