Allennlp: upgrade pytorch-pretrained-bert => pytorch-transformers

[joelgrus]

they renamed the library, I'm not sure if they changed any APIs

[joelgrus]

or if we can take advance of a unified API

[rulai-huajunzeng]

Also XLNet is added to pytorch-transformers, are we going to add XLNet support in allennlp?

[BrambleXu]

The API of tokenizer is also changed in the latest version of pytorch-transformers

from pytorch_pretrained_bert.tokenization import BertTokenizer

->

from pytorch_transformers import BertTokenizer

[joelgrus]

I am working on this right now, my goal is to present a unified TransformerTokenIndexer + TransformerTokenEmbedder that wraps the huggingface abstraction and works with all of their models, but certainly the devil will be in the details

[amolk]

@joelgrus Any updates? I can use this for a project if ready, otherwise will have to hack something together myself.

[matt-gardner]

This was added a while ago for cases where you have matched tokenization and embedding (see the pretrained_transformer options for tokenizers, indexers, and embedders). It's only the mismatched case that still needs work (and there might be features missing in the new stuff, too). I'm going to close this issue; I think we have an issue for handling mismatched tokenization already, but if now, we should open a separate one.

[ruijianw]

@matt-gardner Could you please elaborate more about the mismatched tokenization?

[matt-gardner]

Mismatched tokenization is when you tokenize by words but do modeling on subwords (or any other similar mismatch). You might do this, e.g., for using BERT for tagging, where you have labels and want to make predictions on words, not on subword units. There are a few ways to handle this, but one way is to have this mismatched tokenization. This is how the pretrained BERT indexer / embedder work (the ones before I added the new pretrained_transformer classes). The new functionality doesn't handle this case, but we should add it.