Farm: Getting different predictions on different runs with same ELECTRA model.

[PhilipMay]

I trained an Electra Model on text classification. 3 Classes.
Saved it with

    save_dir = Path("saved_models/bert-german-doc-tutorial")
    model.save(save_dir)
    processor.save(save_dir)

Now I load Data I want to predict on and getting different results:

1st run:
labal_n          5055
label_i           855
label_e          8

2nd run
labal_n          4609
label_e          990
label_i           319

3rd run
label_i        3355
labal_n       2510
label_e       53

How on earth can that be? Even if the model would be a total crap the prediction should be deterministic - right?

[Timoeller]

How do you load it? Its Friday evening, this might also affect the preds or at least the code ;)

[PhilipMay]

Well - I did use torch 1.6.0. Might that be the reason?
And transformers 3.1.0

[PhilipMay]

Is it possible that there is a bug in transformers that does not let me load saved delectra models from local disk?

[PhilipMay]

How do you load it?

    model = Inferencer.load(model_path,
                        num_processes=4,
                        batch_size=batch_size,
                        gpu=True,
                        )
    result = model.inference_from_dicts(dicts=unlabeled_text_dict, multiprocessing_chunksize=4000)

[PhilipMay]

I tested it with a normal BERT model.
There I get 100% stable results.
Only with electra I get unstable things...
Loading / saving electra models with FARM / transformers has a bug is my current hypothesis.

Maybe some layers are not saved / loaded and so are random initialized...

Do you have any idea how and where I could start a debugging?
This might also effect your "future release plans" if you know what I mean... @Timoeller @stefan-it

[Timoeller]

Mhh, I believe the culprit might be hidden somewhere else : )

save_dir = Path("saved_models/bert-german-doc-tutorial")

in FARM we still have string based model loading, so if you name your model "bert" it will be treated as such. Could that be the problem?

[PhilipMay]

in FARM we still have string based model loading, so if you name your model "bert" it will be treated as such. Could that be the problem?

It is "./models/german-nlp-group-electra-base-german-uncased"

[Timoeller]

How about posting the code you actually used so we can find potential bugs? :p

As of debugging, I would step into the debugger when loading the model and see if the weights are assigned correctly and are the same across multiple restarts.

[PhilipMay]

How about posting the code you actually used so we can find potential bugs? :p

Yes - I will try to build an example with an open dataset.

@Timoeller I am doing EarlyStopping which saves the model and loads the best after stopping to evaluate against test.
That seems to work. Maybe there is a difference in loading it while training and loading it for Inference?

I will check that tonight...

[stefan-it]

Very strange, because the Inferencer constructor calls set_all_seeds 🤔

[PhilipMay]

I always thought that the Trainer (callback) saving and loading for test evaluation is working correctly and that there is only a bug in the Interferer. But that is not the case. See here #522.

When training, the model is not loaded from disk when doing test set evaluation but the old is used.

[PhilipMay]

Very strange, because the Inferencer constructor calls set_all_seeds thinking

@stefan-it do you think this might reset important parts of the neutwork?

[PhilipMay]

I doule checked this. With "normal" BERT models there is no problem. Seems to be electra specific.

[Timoeller]

So early stopping is unrelated to Inferencer loading, lets not try to mix bugs : )

Can you create a minimal script reproducing the ELECTRA Inferencer error?

[PhilipMay]

Example code to reproduce: https://gist.github.com/PhilipMay/bd250cba591b3252b8da2f3d31ee5b64

I am using the master branch from FARM, transformers 3.1.0 and torch 1.6.0.

First run it with lang_model = "bert-base-german-dbmdz-uncased". After that run with lang_model = "german-nlp-group/electra-base-german-uncased". Compare the results as I did below:

Output for the bert model

lang_model = "bert-base-german-dbmdz-uncased":

result from early stopping (on dev set) 0.7936361061278443
result from test set (with best loaded trial) 0.7892217833682655
Please compare result from early stopping (on dev set) and result from test set (with best loaded trial)...

Best early stopping result on dev set is 0.79 while the test set result is 0.78. This is the expected result.
... note that the trainer stored the model after each evaluation (when it was better) and then loads and evaluates it vs. test.
Test set evaluation is a little bit lower but in the same range. Everything is ok...

Output for the bert model

lang_model = "german-nlp-group/electra-base-german-uncased":

result from early stopping (on dev set) 0.7837095866567335
result from test set (with best loaded trial) 0.47395779141955396

Best early stopping result on dev set is 0.78 while the test set result is 0.47.
The test evaluation is much lower which is not normal.
F1 Macro of 0.47 should be close to the naive baseline (random prediction) on this dataset.
My guess is that there is a bug in saving or loading electra models.

[PhilipMay]

Are you sure of this code:

https://github.com/deepset-ai/FARM/blob/master/farm/modeling/language_model.py#L1255-L1265

@stefan-it you wrote it 4 months ago. Could you check that?

[Timoeller]

@philipmay I thought we were talking about different inferencer results.

How about we break this down. Load the inferencer once, make preds1, load it another time, make preds2. If preds1 != preds2 we should fix something.

Including crossval and early stopping adds too much complexity on this issue. Is that correct or am I missing something?

[PhilipMay]

Ok @Timoeller - sorry for the bad example. You are right.
Here is a more streamlined example without crossvalidaton: https://gist.github.com/PhilipMay/2e42eeb7174cf0a122036a26ab38ceba#file-electra_load_test-py

I load a model, train it on germeval18 wait for automatic test set evaluation and write down the f1 macro.
Then the model is saved and I press return to continue.
Then model is loaded to inferencer and evaluated on test set again. Then f1 macro is calculated and printed.

For lang_model = "bert-base-german-dbmdz-uncased"
Test F1 macro on training: 0.7699
Test F1 macro on inference: 0.7686296419156032

For lang_model = "german-nlp-group/electra-base-german-uncased"
Test F1 macro on training: 0.7628
Test F1 macro on inference: 0.4487975932065501

The difference in case of electra shows the bug. The numbers should be that same!

@stefan-it could you maybe have a look on this? You were the nice guy who added electra support to FARM.

Please make sure to install the current transformers version 3.1.0 to run this. Otherwise the tokenizer will drop umlauts.

[Timoeller]

The issue comes from the class weights, those are not properly loaded and not specifying task_type="classification" in the inferencer.
And I also observe the label mapping is inverted, but only sometimes. Unsure why this happens...

[PhilipMay]

@Timoeller do you know more details?

• Are you sure that the problem is coming from loading and not from saving?
• Is it the language model that is not correctly loaded or is it the head?
• Is the bug located in HF or Farm?

[PhilipMay]

Very strange, because the Inferencer constructor calls set_all_seeds thinking

I removed that line. Bug still persists.

[Timoeller]

Ok, I did some further testing as well because there are apparently weird things happening.

There seems to be a difference when running the script

1. with model training + inferencing in one go vs
2. model training, saving, stopping the script. Running a new script with inference on the saved model only

Could you please verify this @PhilipMay

[PhilipMay]

@Timoeller I can approve this. Running training, final test set evaluation and saving in one script and then loading and inference in other script the bug does not happen and we have the same F1 macro on test set on both runs.

[stefan-it]

All GPUs are currently occupied for EACL, but I'll try to run your script soon :)

[PhilipMay]

All GPUs are currently occupied for EACL, but I'll try to run your script soon :)

@stefan-it that would be awsome!

[Timoeller]

Running training, final test set evaluation and saving in one script and then loading and inference in other script the bug does not happen and we have the same F1 macro on test set on both runs.

Ok, so now we have a hook on the mystery. It obviously must be that variables do not get GCed and that affects loading or predictions somehow... I rather believe this is a pytorch GC issue on the computation graph and some variables being not deleted.

[PhilipMay]

Running training, final test set evaluation and saving in one script and then loading and inference in other script the bug does not happen and we have the same F1 macro on test set on both runs.

Ok, so now we have a hook on the mystery. It obviously must be that variables do not get GCed and that affects loading or predictions somehow... I rather believe this is a pytorch GC issue on the computation graph and some variables being not deleted.

The strange thing is: Why does it happen with Electra but not with BERT? The most important difference I see is here:
https://github.com/deepset-ai/FARM/blob/master/farm/modeling/language_model.py#L1255-L1265

[PhilipMay]

Running training, final test set evaluation and saving in one script and then loading and inference in other script the bug does not happen and we have the same F1 macro on test set on both runs.

Ok, so now we have a hook on the mystery. It obviously must be that variables do not get GCed and that affects loading or predictions somehow... I rather believe this is a pytorch GC issue on the computation graph and some variables being not deleted.

Yes and no...
For this example this is right. But I have a case where I use early stopping. The electra model that is saved by early stopping is even not working correctly when loaded and evaluated in an other script. But maybe that is a 2nd bug...

[Timoeller]

I would also separate these cases for simplicity.

How do you want to proceed with the pytorch GC issue?

[PhilipMay]

How do you want to proceed with the pytorch GC issue?

Well - to be honest: I hope @stefan-it checks and fixes it.

I made some experiments with things I thought would be the reason but did not come to a solution. I maybe check the recent torch issues in github. But since it works with BERT but not with Electra I still think it might be a bug in FARM.

[stefan-it]

On it...

[stefan-it]

It's very strange, because the root cause is the language_model = LanguageModel.load(lang_model) call. And it's working for DistilBERT (that also uses the SequenceSummary pooling stuff)...

[PhilipMay]

Yes. I tested it with the german Distilbert. And it was all ok. No Bug.

[PhilipMay]

Are there any new insights in this topic?

@stefan-it would it help to have a look at transformers and how they added the head on top of electra?

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_electra.py#L424

[PhilipMay]

Hi, just wanted to bunp this. Any idea how to continue with this?
Thanks
Philip

[Timoeller]

Since we (stefan-it and deepset) will be releasing our joint German Electra model soon we will then also look into this issue.

We will most likely be able to include checking this issue in our next sprint.

[Timoeller]

We are boiling down this strange model re loading behaviour. Our next steps will be to determine when the error happens. Then we will deep dive into why:

• Producing bug
    ◦ Single script training and loading again
    ◦ Single script model loading twice without training
    ◦ CPU vs CUDA
    ◦ Inferencer vs Evaluator (Evaluator also needs to be loaded from disk as in our Trainner.train fct during early stopping)
• Not producing bug (sanity check)
    ◦ Separate scripts training + saving + loading
• What is going wrong in weight initialization
    ◦ Is LM initialized wrong
    ◦ Or just PH
    ◦ Or both?

[PhilipMay]

Great @Timoeller ! Many thanks.