Haystack: RequestError(400, 'search_phase_execution_exception', 'runtime error') when running tutorial 4 with Elasticsearch (using Docker)

[guillim]

When running tutorial 4, FAQ style, I am having an issue at the end, when trying to retrieve an answer using finder.get_answers_via_similar_questions function.

Error message

09/21/2020 12:52:50 - WARNING - elasticsearch -   POST http://newHaystack_elasticsearch_1:9200/document/_search [status:400 request:0.089s]
---------------------------------------------------------------------------
RequestError                              Traceback (most recent call last)
<ipython-input-5-b475e6abf8ee> in <module>
      1 finder = Finder(reader=None, retriever=retriever)
----> 2 prediction = finder.get_answers_via_similar_questions(question="How is the virus spreading?", top_k_retriever=10)
      3 print_answers(prediction, details="all")

/home/user/haystack/finder.py in get_answers_via_similar_questions(self, question, top_k_retriever, filters, index)
     93
     94         # 1) Apply retriever to match similar questions via cosine similarity of embeddings
---> 95         documents = self.retriever.retrieve(question, top_k=top_k_retriever, filters=filters, index=index)
     96
     97         # 2) Format response

/home/user/haystack/retriever/dense.py in retrieve(self, query, filters, top_k, index)
    316         query_emb = self.embed(texts=[query])
    317         documents = self.document_store.query_by_embedding(query_emb=query_emb[0], filters=filters,
--> 318                                                            top_k=top_k, index=index)
    319         return documents
    320

/home/user/haystack/document_store/elasticsearch.py in query_by_embedding(self, query_emb, filters, top_k, index)
    420
    421             logger.debug(f"Retriever query: {body}")
--> 422             result = self.client.search(index=index, body=body, request_timeout=300)["hits"]["hits"]
    423
    424             documents = [self._convert_es_hit_to_document(hit, adapt_score_for_embedding=True) for hit in result]

/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py in _wrapped(*args, **kwargs)
    150                 if p in kwargs:
    151                     params[p] = kwargs.pop(p)
--> 152             return func(*args, params=params, headers=headers, **kwargs)
    153
    154         return _wrapped

/usr/local/lib/python3.7/site-packages/elasticsearch/client/__init__.py in search(self, body, index, doc_type, params, headers)
   1615             params=params,
   1616             headers=headers,
-> 1617             body=body,
   1618         )
   1619

/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py in perform_request(self, method, url, headers, params, body)
    390                         raise e
    391                 else:
--> 392                     raise e
    393
    394             else:

/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py in perform_request(self, method, url, headers, params, body)
    363                     headers=headers,
    364                     ignore=ignore,
--> 365                     timeout=timeout,
    366                 )
    367

/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py in perform_request(self, method, url, params, body, timeout, ignore, headers)
    267                 method, full_url, url, orig_body, duration, response.status, raw_data
    268             )
--> 269             self._raise_error(response.status, raw_data)
    270
    271         self.log_request_success(

/usr/local/lib/python3.7/site-packages/elasticsearch/connection/base.py in _raise_error(self, status_code, raw_data)
    299
    300         raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
--> 301             status_code, error_message, additional_info
    302         )
    303

RequestError: RequestError(400, 'search_phase_execution_exception', 'runtime error')

After using logging.getLogger().setLevel(logging.DEBUG) to print more details about ElasticSearch, here is the message :

DEBUG - elasticsearch -   < 
{
   "error":{
      "root_cause":[
         {
            "type":"script_exception",
            "reason":"runtime error",
            "script_stack":[
               "org.elasticsearch.xpack.vectors.query.ScoreScriptUtils$DenseVectorFunction.<init>(ScoreScriptUtils.java:85)",
               "org.elasticsearch.xpack.vectors.query.ScoreScriptUtils$CosineSimilarity.<init>(ScoreScriptUtils.java:175)",
               "cosineSimilarity(params.query_vector,doc['question_emb']) + 1.0",
               "                                        ^---- HERE"
            ],
            "script":"cosineSimilarity(params.query_vector,doc['question_emb']) + 1.0",
            "lang":"painless"
         }
      ],
      "type":"search_phase_execution_exception",
      "reason":"all shards failed",
      "phase":"query",
      "grouped":true,
      "failed_shards":[
         {
            "shard":0,
            "index":"document",
            "node":"9nvMTfJ-RFqkWTe7GzpHcQ",
            "reason":{
               "type":"script_exception",
               "reason":"runtime error",
               "script_stack":[
                  "org.elasticsearch.xpack.vectors.query.ScoreScriptUtils$DenseVectorFunction.<init>(ScoreScriptUtils.java:85)",
                  "org.elasticsearch.xpack.vectors.query.ScoreScriptUtils$CosineSimilarity.<init>(ScoreScriptUtils.java:175)",
                  "cosineSimilarity(params.query_vector,doc['question_emb']) + 1.0",
                  "                                        ^---- HERE"
               ],
               "script":"cosineSimilarity(params.query_vector,doc['question_emb']) + 1.0",
               "lang":"painless",
               "caused_by":{
                  "type":"illegal_argument_exception",
                  "reason":"For vector functions, the 'field' argument must be of type String or VectorScriptDocValues"
               }
            }
         }
      ]
   },
   "status":400
}

Expected behavior
I expect the prediction to work as described.

To Reproduce

• do a fresh git clone of the project
• launch the docker-compose up command (I am using Docker under MacOS)
• get into the API container using docker container exec -it haystack-api bash
• inside the container, install ipython
• follow the tutoral 4 until the end : at line prediction = finder.get_answers_via_similar_questions(question="How is the virus spreading?", top_k_retriever=10) you should come up with a bug.

TroubleShooting
I noticed that deleting the defaulty created index document and recreating one with specific mapping types would do the fix.

• How to delete the index

curl -X DELETE localhost:9200/document

• How to create a new one

curl -X PUT "localhost:9200/document?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "question": { "type": "keyword" }, "text" : { "type" : "text"}, "question_emb" : {"type": "dense_vector", "dims": 768}
    }
  }
}
'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "document"
}

Here is the default mapping for the document index localhost:9200/document/_mapping:

{"document":{"mappings":{"dynamic_templates":[{"strings":{"path_match":"*","match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"name":{"type":"keyword"},"text":{"type":"text"}}}}}

System:

• OS: according the Docker images
• GPU/CPU: CPU
• Haystack version (commit or version number): 0.4.0
• DocumentStore: ElasticsearchDocumentStore
• Reader: NA
• Retriever: EmbeddingRetriever with deepset/sentence_bert model

[guillim]

Hey @tholor , I did a PR #415 to fix this problem.

I am using elasticsearch API function put_mapping to do that

Hardest part of this troubleshooting was that on this view the mapping didn't show up so it made me think the indexes were the same. I think there were more than one issue related to this topic.

Tks @psorianom for your help.

Note : I didn't check other document_store to make sure it works with them.