Haystack: Support Camembert Model in FARMReader

[Ierezell]

I'm trying to plug Camembert (french bert) from HuggingFace into Haystack.
https://huggingface.co/transformers/model_doc/camembert.html

Notebook :

from haystack import Finder
from haystack.indexing.cleaning import clean_wiki_text
from haystack.indexing.io import write_documents_to_db, fetch_archive_from_http
from haystack.reader.farm import FARMReader
from haystack.reader.transformers import TransformersReader
from haystack.utils import print_answers
from transformers import (CamembertConfig, CamembertModel, CamembertTokenizer) 

from haystack.database.elasticsearch import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")

doc_dir = "./test_q_a/fr"
# doc_dir = "./test_q_a/en"
write_documents_to_db(document_store=document_store, document_dir=doc_dir, clean_func=clean_wiki_text, only_empty_db=True, split_paragraphs=True)

from haystack.retriever.elasticsearch import ElasticsearchRetriever
retriever = ElasticsearchRetriever(document_store=document_store)

This is where I changed to plug my model.

bert = CamembertModel.from_pretrained("camembert-base")
bert_tok = CamembertTokenizer.from_pretrained("camembert-base")
reader = TransformersReader(model=bert, tokenizer=bert_tok, use_gpu=-1)

finder = Finder(reader, retriever)

prediction = finder.get_answers(question="Quels sont les symptômes ? ", top_k_retriever=10, top_k_reader=5)
print_answers(prediction, details="all")



md5-da2070dc8d603d6320d4a2b7f71086b9



---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-d63ddbd66c4b> in <module>
----> 1 prediction = finder.get_answers(question="Quels sont les symptômes ? ", top_k_retriever=10, top_k_reader=5)
      2 print_answers(prediction, details="minimal")

~/git_clones/haystack/haystack/finder.py in get_answers(self, question, top_k_reader, top_k_retriever, filters)
     41         len_chars = sum([len(d.text) for d in documents])
     42         logger.info(f"Reader is looking for detailed answer in {len_chars} chars ...")
---> 43         results = self.reader.predict(question=question,
     44                                       documents=documents,
     45                                       top_k=top_k_reader)

~/git_clones/haystack/haystack/reader/transformers.py in predict(self, question, documents, top_k)
     75         for doc in documents:
     76             query = {"context": doc.text, "question": question}
---> 77             predictions = self.model(query, topk=self.n_best_per_passage)
     78             # assemble and format all answers
     79             for pred in predictions:

~/.local/lib/python3.8/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
   1019                 # Mask padding and question
   1020                 start_, end_ = (
-> 1021                     start_ * np.abs(np.array(feature.p_mask) - 1),
   1022                     end_ * np.abs(np.array(feature.p_mask) - 1),
   1023                 )

ValueError: operands could not be broadcast together with shapes (384,768) (384,)

[Timoeller]

Hey @Ierezell thanks for using haystack.

It seems you just inserted a "plain" language model into the reader. The reader should have Question Answering (extraction of answer span) capabilities.

Could you try this model: https://huggingface.co/fmikaelian/camembert-base-squad so
"fmikaelian/camembert-base-squad" instead of only "camembert-base"
and report back if it works?

If that works we should create better error messages in our reader to tell people they havent loaded the right model...

[Ierezell]

It works perfectly with a Q&A model, thanks !
However, I cannot plug it in the FarmReader with just the name, I need to load it and pass it to the TransformerReader.

My bad, sorry I didn't thought that it needed an Q&A model but it's obvious...

Adding an error message is a good idea and will avoid stupid mistakes like that.

Thanks a lot for the really fast support !

P.s do you now if I can get a bigger context for the answer and if it's possible to flush the database from haystack (changing languages).

[Timoeller]

Hey thanks for the info about the FarmReader.

Just to clarify

reader = TransformersReader(model="fmikaelian/camembert-base-squad",
                            tokenizer="fmikaelian/camembert-base-squad",
                            use_gpu=-1)

works as well on my end (without passing the loaded model and tok to the TransformersReader.

You are right: The FarmReader currently doesnt support camembert (should be an easy fix on our side, since it is the RoBERTa architecture).

---

Regarding longer context: both readers have an argument context_window_size which is the size of the context in characters around the midpoint of the answer. The context size always stays constant (and can be shorter than the answer itself when the answer is longer than the context_window_size value).

---

Regarding flushing of document database: this is a great feature we will implement very soon. Could you maybe create an individual issue for this?
A possible workaround for now would be to use another "index" in the ElasticsearchDocumentStore and set only_empty_db=False in the write_documents_to_db() function. That way you have both new and old data in Elasticsearch but will query only the docs in the newly created index.

[Timoeller]

Issue seems fixed. Closing this now. Feel free to reopen or add your insights @Ierezell