Transformers: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'

[bigzhouj]

❓ Questions & Help

fine tuning bert use wikitext2

[bigzhouj]

in probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

[TheEdoardo93]

Please, describe your environment, post the source code for reproducibility and the error.

in probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

[LysandreJik]

Please upgrade your Pytorch version to 1.2.0+.

[bkkaggle]

You're probably passing in a boolean tensor (true or false) instead of a byte tensor (0 or 1) for your attention mask.

Try changing

probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

to

probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)