Haystack: ALBERT XXL doesn't work on FARMReader

[antoniolanza1996]

Describe the bug
I was testing your suggestions in your documentation _(btw, thank you, they are really useful and well-explained)_. However, I encountered this bug when I tried ALBERT XXL.

Error message
Code:
reader = FARMReader("ahotrod/albert_xxlargev1_squad2_512")

Error:

09/28/2020 19:18:11 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None
09/28/2020 19:18:11 - INFO - farm.infer -   Could not find `ahotrod/albert_xxlargev1_squad2_512` locally. Try to download from model hub ...
09/28/2020 19:18:20 - WARNING - farm.modeling.language_model -   Could not automatically detect from language model name what language it is. 
     We guess it's an *ENGLISH* model ... 
     If not: Init the language model by supplying the 'language' param.
09/28/2020 19:18:32 - WARNING - farm.modeling.prediction_head -   Some unused parameters are passed to the QuestionAnsweringHead. Might not be a problem. Params: {"loss_ignore_index": -1}
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-20-e4db3dcdf3a9> in <module>()
----> 1 reader = FARMReader("ahotrod/albert_xxlargev1_squad2_512")

7 frames
/usr/local/lib/python3.6/dist-packages/sentencepiece.py in LoadFromFile(self, arg)
    175 
    176     def LoadFromFile(self, arg):
--> 177         return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    178 
    179     def Init(self,

TypeError: not a string

To Reproduce
Just execute:
reader = FARMReader("ahotrod/albert_xxlargev1_squad2_512")

[ahotrod]

It's been several months since I last ran successfully reader = FARMReader("ahotrod/albert_xxlargev1_squad2_512") on Haystack. I just tried running it now and get the same "not a string" error as you. This particular ALBERT xxlarge model was fine-tuned on Transformers 2.3.0, so it may need a fine-tune refresh on the latest Transformers 3.X master.

BTW, I've moved-on to this large model for my QA use-cases:

reader = FARMReader(model_name_or_path = "ahotrod/electra_large_discriminator_squad2_512",
                    context_window_size = 512, batch_size = 448, use_gpu = True,
                    no_ans_boost = 3, top_k_per_candidate = 6, top_k_per_sample = 2,
                    num_processes = 12, max_seq_len = 512, doc_stride = 128)

which gives these Haystack reader_eval_results:

No_Ans_Boost: 3
  Reader Top-N-Accuracy: 0.991
     Reader Exact Match: 0.847
        Reader F1-Score: 0.89

[antoniolanza1996]

Hi @ahotrod ,

it may need a fine-tune refresh on the latest Transformers 3.X master.

Yes, indeed the problem is not only with FARM, but also with Transformers (i.e. reader = TransformersReader("ahotrod/albert_xxlargev1_squad2_512")).

... which gives these Haystack reader_eval_results:
No_Ans_Boost: 3
Reader Top-N-Accuracy: 0.991
Reader Exact Match: 0.847
Reader F1-Score: 0.89

I will also try your ELECTRA model, thank you so much. Are these results achieved on SQuAD 2.0 dev set?

[ahotrod]

@antoniolanza1996, Yes these reader.eval_on_file() results are achieved on the SQuAD 2.0 dev set.

@tholor @Timoeller @tanaysoni @brandenchan The new Haystack benchmarks, website, documentation, logo, etc. are looking good, congrats!

[ahotrod]

@antoniolanza1996,

Just posted ahotrod/albert_xxlargev1_squad2_512 fine-tuned on Transformers 3.1.0. Works for me now.

Thought I'd take the opportunity to compare ELECTRA vs updated ALBERT in Haystack for accuracy (most important for me), hosting hardware requirements, and speed.

[antoniolanza1996]

Hi @ahotrod , thank you very much. I'll run some tests tomorrow and I'll let you know if all works from my side.

Thought I'd take the opportunity to compare ELECTRA vs updated ALBERT in Haystack for accuracy (most important for me), hosting hardware requirements, and speed.

Really interesting. If you can, please share your tips/findings about this comparison. This could be really useful for the whole community :)

[ahotrod]

@antoniolanza1996, I'm happy to share my preliminary comparison results:

My use case is single-document (so no retriever), roughly 10-20K words per document, generally 100-200 questions per analysis.
The software & system used for this comparison:

   FARM-Haystack: 0.4.0
   Transformers: 3.1.0
   PyTorch: 1.6.0
   TensorFlow: 2.3.1
   Python: 3.8.1
   OS: Linux-5.4.0-48-generic-x86_64-with-glibc2.10
   CPU/GPU: Intel i9-9900K / NVIDIA Titan RTX 24GB

I iterated over lists of values for no_ans_boost, top_k_per_candidate, top_k_per_sample for ELECTRA, and the values below gave the best accuracy results. I haven't had the time to do the same for ALBERT, so I used the ELECTRA values for this initial comparison.

reader = FARMReader(model_name_or_path = "ahotrod/electra_large_discriminator_squad2_512",
                    context_window_size = 512, batch_size = 448, use_gpu = True,
                    no_ans_boost = 2.5, top_k_per_candidate = 16, top_k_per_sample = 16,
                    num_processes = 12, max_seq_len = 512, doc_stride = 128)

#  24104MiB GPU memory used for inferencing, approx. 5 seconds per inference on my questions
#  approx. 7 minutes for reader.eval_on_file("dev-v2.0.json")
#  Reader Top-N-Accuracy: 0.9987
#     Reader Exact Match: 0.8468
#        Reader F1-Score: 0.8897

reader = FARMReader(model_name_or_path="ahotrod/albert_xxlargev1_squad2_512",
                    context_window_size = 512, batch_size = 112, use_gpu = True,
                    no_ans_boost = 2.5, top_k_per_candidate = 16, top_k_per_sample = 16,
                    num_processes = 12, max_seq_len = 512, doc_stride = 128)

#  24102MiB GPU memory used for inferencing, approx. 34 seconds per inference on my questions
#  approx. 53.5 minutes for reader.eval_on_file("dev-v2.0.json")
#  Reader Top-N-Accuracy: 0.9989
#     Reader Exact Match: 0.8073
#        Reader F1-Score: 0.8772

Initial conclusion, ELECTRA is much more "efficient" (size & speed) than ALBERT. Even though speed is my third-ranked criteria, it is an issue. Inferencing with ELECTRA is considerably faster, approximately 7x. Another indicator, fine-tuning ELECTRA_large on SQuAD2.0 takes 4-1/2 hours, ALBERT_xxl takes 15 hours with similar parameters.

Just digging into the supplied answers on MY questions, but after initial inspection, it seems ELECTRA produces better answers from a qualitative perspective. More later. YMMV

[tholor]

Interesting that your optimal setup has top_k_per_sample = 16. I suppose this mainly increased Top-N-Accuracy but not so much the other metrics?

[ahotrod]

Interesting that your optimal setup has top_k_per_sample = 16. I suppose this mainly increased Top-N-Accuracy but not so much the other metrics?

@tholor You are correct. Increases in top_k_per_sample and top_k_per_candidate increased Top-N-Accuracy without appreciable, if any effect, on Reader Exact Match and Reader F1-Score. I didn't notice any significant speed penalty using 16 for both parameters with my system & use case. There are diminished increases in Top-N-Accuracy for parameter values above 16, having reached a plateau at 0.9989, virtually 1. Notice in my first post above that top_k_per_candidate = 6 & top_k_per_sample = 2, produced a Top-N-Accuracy of 0.991. Might be "splitting hairs" at this point, can't exactly say, would need to examine the code.

[antoniolanza1996]

Hi @ahotrod ,
I finally managed to test your ALBERT XXL model.
I can confirm that it works well, both in Haystack's FARMReader and TransformersReader. Hence, I'm gonna close this issue. Thank you very much for your support.