Opennmt-py: How is coverage loss implemented?

[nikhilweee]

101 Talks about coverage loss as per Abigail See's paper, which was implemented in the following manner.

https://github.com/OpenNMT/OpenNMT-py/blob/ecbce3330acbe97c2319e3cb13ce38a5a399b876/onmt/Loss.py#L176-L178
I don't seem to find the relevant piece of code in master. Can someone help me out?

[pltrdy]

The idea is quite the same as in abisee, for each sequence, there is:

• a generation probability over the (decoder) vocabulary
• a soft copy switch (probability to copy)
• a copy probability over source tokens (here, attention scores are used, eq 2 of See 2017)

Final scores takes all of that into account, therefore there is no "coverage loss", it's just part of the loss calculation.

---

More precisely:

During training, with -copy_attn option set, the generator (onmt.modules.CopyGenerator) will compute soft-switch probabilities (line 94).

We concatenate scores from "generation" and "copy" and return it (line 102). Since the model may copy tokens that are in the vocabulary (so it also has a "generation" probability) we "collapse" those scores afterward (line 189).

---

Hope it's clear, tell me if it's not.

[nikhilweee]

@pltrdy Thanks for the elaborate explanation! I realise that I wasn't very clear while describing the issue, but I was actually talking about the coverage mechanism described in section 2.3 of @abisee's paper which takes coverage attention into account while calculating the _coverage_ loss. (eq. 13 in the paper). I wanted to know whether this has been implemented. If yes, I'd be happy if you could point out the relevant code.

[vince62s]

On your original point, look here:
https://github.com/OpenNMT/OpenNMT-py/pull/172/files
@srush may explain better, but I guess at that point in time, it was decided to switch completely to "copy training" which differs from the original paper.

closing, but reopen if needed.