Simpletransformers: Is it possible to return the attention heads learned when using longformer classificaitons.

[codeninja]

I am attempting to use the new longformer model and I am missing how I can configure the system to return the attentions when making a prediction.

Please point me in the right direction. Thanks.

[DebanjanaKar]

I want to know the same thing with regard to NER Models.

[rhamnett]

This would be extremely useful @ThilinaRajapakse

[DebanjanaKar]

I did a workaround and I am posting it here hoping it might help others as well :

First step, in your model config, add 'output_attention' : True

Then, in the predict function of the source code, find the line where the output is getting returned. Usually it is stored in a variable outputs . After that line add :

if self.config.output_attentions: attn = outputs[-1]

After that skip to the line containing the return statement and right before the return statement add :
if self.config.output_attentions: return preds, model_outputs, attn

This would return the attention weights of your model.

[DebanjanaKar]

@ThilinaRajapakse : if you want I could raise a PR for this so that all the modules can return the attention weights as well. Let me know your thoughts on it. :)

[codeninja]

@ThilinaRajapakse : if you want I could raise a PR for this so that all the modules can return the attention weights as well. Let me know your thoughts on it. :)

This would be extremely helpful! BERT et. al. would benefit from this.

[codeninja]

UPDATE: It appears as though output_hidden_states also returns attentions.

From o[3] we have access to all attention heads!

@DebanjanaKar I don't feel like a PR is necessary at this point. Thank you very much for your feedback as it pointed me in the right direction!

This issue may be closed, however I think the documentation should be updated to reflect that attentions are also returned. I'm leaving the Issue open should the developers want to take action. But it's appropriate to close this issue as the initial problem is resolved.

[ThilinaRajapakse]

output_hidden_states is currently only implemented for Classification tasks. I believe @DebanjanaKar was suggesting a PR for the other tasks as well. I think it may be useful to have it for the other tasks too.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.