Farm: Is there a pretrained German Albert / Distilbert model using the deepset.ai word pieces?

[johann-petrak]

The deepset.ai German pretrained BERT model with the German word pieces works well but has the common problem of all of the original transformer models in that it uses a lot of memory and is not too fast for long sequence lengths.

There are other model architectures now in the transformer library which promise to alleviate those problems at least to some extent like ALBERT or DistilBERT. Are there equivalent pretrained models for these architectures for German based on the deepset.ai word pieces or any hope that there will be such models at some point?

How do people generally deal with the issue of short sequence lengths (or tiny minibatches, or both)?

[tholor]

_TL;DR: Not yet, but hopefully ALBERT soon. Not sure about distilBERT yet. Gradient accumulation and AMP could help._

We are currently training multiple German ALBERT models (more data + new vocab), but face problems with loss convergence and downstream performance (so far not better than BERT).
We have a few more things that we wanna try here and hopefully get some better results then.

We also have plans for a German Roberta, but this would be mostly to boost performance and less about speed / memory.

Regarding DistilBERT: Have you seen/tried the German distilBERT by @stefan-it?
Is there a particular reason why you would like to have the same word-pieces as in our BERT (e.g. for comparisons)? If yes, we could rather easily kick-off training here.

How do people generally deal with the issue of short sequence lengths (or tiny minibatches, or both)?

For training there's two popular options, but I guess you already tested them:

• gradient accumulation (allows larger seq lengths + large effective batch sizes as they become independent of your RAM; doesn't help with speed)
• Automatic mixed precision (AMP) via nvidia's apex (reduces RAM footprint; increases speed by ~35%; no or small drop in performance)
  Both are quite easy to use in FARM (let me know if I should give you some pointers).

If you need even larger sequence lengths (> 512) (e.g. long document for classification or QA), the most common workaround is currently to divide them into shorter sequences, get logits from the model and aggregate everything back to a single prediction. This is clearly not optimal. We are therefore pretty excited about current research on new architectures (e.g. reformer and sparse attention). Will hopefully give us some alternatives within 2020.

[johann-petrak]

Thank you for this helpful answer!

Is there a particular reason why you would like to have the same word-pieces as in our BERT (e.g. for comparisons)? If yes, we could rather easily kick-off training here.

When I say "same word-pieces" I meant, using the same, German-specific method which seems (I am not sure about the details) respect German compounds and morphology more than the method used for the original multilingual BERT models. I compare the deepset.ai pretrained model with the origingal Germal model (both cased) and got better results with the deepset model. Of course the reason could also be that the deepset.ai was trained on a different corpus as well, but word-pieces specifically created for German just sounds very reasonable! :)

BTW is there a paper or some other place to get more detailled info about how the deepset.ai German word pieces have been created?

Do you have some info about the effort to train from scratch you could share. We have a very interesting corpus here that could be very interesting to train on (the corpus cannot be shared but the model could perhaps/likely be shared), so I would be curious if it is worth considering to try that.

[stefan-it]

I think the training command for SentencePiece can be found in this issue:

https://github.com/huggingface/transformers/pull/688#issuecomment-514560713

So I would say it respects German compounds/morphology better, because of the (almost) mono-lingual German training corpus.

However, there's room for more performance boost when it comes to BPE and for German there's a nice paper about linguistically-motivated BPE (topic is machine translation, but I'm sure it is also working for sequence labeling or classification):

In this paper, we investigate word segmentation strategies that incorporate more linguistic knowledge.

In: "Target-side Word Segmentation Strategies for Neural Machine Translation".

Suffix splitting + BPE lead to better results (compared to standalone BPE). So if you're working on more linguistic-motivated word segmentation you should really try it out :)

[tholor]

Nice paper, thanks for sharing @stefan-it! Might be worth testing in a new BERT/roBERTa training run :)

@johann-petrak, I would suggest you give the Stefan's German DistilBERT a try. You can easily load it in FARM by referring to its public name in huggingface's model hub:

```
lang_model = "distilbert-base-german-cased"

tokenizer = Tokenizer.load(
    pretrained_model_name_or_path=lang_model,
    do_lower_case=False)

language_model = LanguageModel.load(lang_model)

```

I compare the deepset.ai pretrained model with the original Germal model (both cased) and got better results with the deepset model.

Which "original German model" are you referring to?

Do you have some info about the effort to train from scratch you could share. We have a very interesting corpus here that could be very interesting to train on (the corpus cannot be shared but the model could perhaps/likely be shared), so I would be curious if it is worth considering to try that.

We are thinking of writing up some learnings & results in a light-weight paper, but not sure when we will find time for it. There are basically two options:
a) Training on TPUs in the google cloud (using Tensorflow & google's BERT repo)
b) Training with GPUs locally or in any cloud (using Tensorflow or pytorch)

Workflows differ a bit and TPUs are quite cheap compared to cloud GPUs. On the other hand, GPUs are simpler to handle for monitoring/eval/debugging. If b) is interesting for you: We are currently implementing training from scratch with FARM. We are integrating it closely with AWS SageMaker (to use ~70% cheaper spot instances), but you could also train on local GPUs.
In both cases, we are happy to share our experience and pipelines. If you want we can discuss this further in a quick call. We will contact you via email.

[johann-petrak]

@johann-petrak, I would suggest you give the Stefan's German DistilBERT a try. You can easily load it in FARM by referring to its public name in huggingface's model hub

Thanks, I gave this a try, very easy to use. In my case distilbert used onlt 63% of the time to train and xval-evaluate 9 datasets, however 4 of the 9 datasets got so much worse that it hurts while none got better. Might be an unlucky random seed of course, I only did one run each so far.