Haystack: Support for nested fields in ElasticsearchDocumentStore

[moise-g]

Question
Does the ElasticsearchDocumentStore support nested data types . If so how could I select a nested field as search field or text field

Additional context

• I am able to retrieve documents using document_store.get_all_documents_in_index('document_index')

In [6]: print(document_store.get_document_count())
   ...: for doc in document_store.get_all_documents_in_index('document_index'):
   ...:     pp.pprint(doc)
   ...:     break
   ...:
   ...:
11196
{   '_id': 'qWwo9nIB7CZChbLK4FjC',
    '_index': 'document_index',
    '_score': None,
    '_source': {   'actor_type': 'content',
                   'media': [   {   'actor_id': [],
                                    'actor_type': 'content',
                                    'body': 'FTS International Has Fragility '
                                            'In The Short-Term; Balance Sheet '
                                            'Remains Vulnerable (NYSE:FTSI) '
                                            'Its high leverage ratio is a '
                                            'significant risk factor in the '
                                            'current environment.\n'
                                            '\n'
                                            ...
                                    'linked_concept_search_id': [342],
                                    'locations': None,
                                    'media_type': 'rss',
                                    'meta': [   {   'key': 'link',
                                                    'value': 'https://seekingalpha.com/article/4352427-fts-international-fragility-in-short-term-balance-sheet-remains-vulnerable'}],
                                    'similar_dictionaries': [],
                                    'sql_handle_id': None,
                                    'sql_media_id': 'gXVpzzMzkQ',
                                    'tags': [   231027,
                                                233849,
                                                233408,
                                                231786,
                                                231102,
                                                231124,
                                                231857,
                                                233795
                                            ],
                                    'title': 'FTS International Has Fragility '
                                             'In The Short-Term; Balance Sheet '
                                             'Remains Vulnerable '
                                             '(NYSE:FTSI)'}]},
    '_type': 'actor',
    'sort': [2]}

• document_store.get_all_documents() gives a KeyError:

In [7]: document_store.get_all_documents()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-0571d29188b7> in <module>
----> 1 document_store.get_all_documents()

~/miniconda3/lib/python3.7/site-packages/haystack/database/elasticsearch.py in get_all_documents(self)
    153     def get_all_documents(self) -> List[Document]:
    154         result = scan(self.client, query={"query": {"match_all": {}}}, index=self.index)
--> 155         documents = [self._convert_es_hit_to_document(hit) for hit in result]
    156         return documents
    157

~/miniconda3/lib/python3.7/site-packages/haystack/database/elasticsearch.py in <listcomp>(.0)
    153     def get_all_documents(self) -> List[Document]:
    154         result = scan(self.client, query={"query": {"match_all": {}}}, index=self.index)
--> 155         documents = [self._convert_es_hit_to_document(hit) for hit in result]
    156         return documents
    157

~/miniconda3/lib/python3.7/site-packages/haystack/database/elasticsearch.py in _convert_es_hit_to_document(self, hit, score_adjustment)
    285         document = Document(
    286             id=hit["_id"],
--> 287             text=hit["_source"][self.text_field],
    288             external_source_id=hit["_source"].get(self.external_source_id_field),
    289             meta=meta_data,

KeyError: 'media.full_body'

• I have tried using a flattened notation and no results:

In [1]: from haystack.retriever.sparse import ElasticsearchRetriever
   ...: from haystack.database.elasticsearch import ElasticsearchDocumentStore

In [2]: document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", text_field="media.full_body", index="vopak-monitoring",name_field="media.title", sea
   ...: rch_fields=["media.full_body", "media.title"], create_index=False)
   ...: retriever = ElasticsearchRetriever(document_store=document_store)
   ...: res = retriever.retrieve("Energy")
   ...: print(res)
[]

• I also tried using a custom query without succes.

┆Issue is synchronized with this Jira Task by Unito

[tanaysoni]

Hi @moise-g, the document stores currently do not support nested fields. A possible solution, for now, is to create a new index with a flat structure.

Would it be something you'd be interested in working on? I think it might be a useful feature for other users as well, so long as we keep the ElasticsearchDocumentStore interface compliant with other document stores.

[guillim]

Hi @tanaysoni ,

I could see some interesting use case about this topic. Flattening the object with a simple _ during the document_store.write_documents(dicts) process could be a first and easy step. What do you say ?

[tanaysoni]

Hi @guillim, could you provide more context for your use case? Do you intend to flatten nested dicts before writing to document_store?

I suppose there's an existing Elasticsearch index created outside Haystack that is being queried in @moise-g's case. The issue is how to configure the ElasticsearchDocumentStore to read a nested field as the text. Would a solution for it help in your case?

[guillim]

Tks for your answer @tanaysoni . In my situation, I have a property in the _meta_ data that is nested. so my document index looks like that :

[...]
'meta' : {
  'name' : 'file1',
  'prop' : {
    'nested_prop1' : 'something'
  },
}
[...]

Since for some reason I can't get the Haystack API work with this using filter on prop.nested_prop1 = 'something', my quick fix at the moment was to transform it before injection into the ES into

[...]
'meta' : {
  'name' : 'file1',
  'prop_nested_prop1' : 'something'
  },
}
[...]

This way, filtering on prop_nested_prop1 = 'something' works. But I would love being able to filter directly into the nested object if that was possible

[tanaysoni]

Hi @guillim, I think it should be possible to query for nested fields using the dot notation. Here's an example:

from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from haystack import Document

document_store = ElasticsearchDocumentStore()

documents = [
    Document(text="Doc1", meta={"prop": {"nested_prop": "test"}}),
    Document(text="Doc2", meta={"nested_prop": "test"}),
]
document_store.write_documents(documents)

documents = document_store.get_all_documents(filters={"prop.nested_prop": ["test"]})  # query nested field
assert documents[0].text == "Doc1"

Would this work for your case?

[guillim]

Well if this gave me some results yes ! But for some reason it doesn't.

So here is what I do :

from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from haystack import Document
document_store = ElasticsearchDocumentStore(host="haystack_elasticsearch_1", username="", password="",
                                                index="document",
                                                embedding_field="question_emb",
                                                embedding_dim=768,
                                                excluded_meta_data=["question_emb"])
document_store.get_all_documents(filters={"arborescence.theme" : ["Justice"] })

Gives me []

While if I dig to make sure my database is properly filled, here is what I have

documents = document_store.get_all_documents()
print(documents[0].meta['arborescence']['theme'])

will give me "Justice"

Si for some reason it sounds it is not functioning properly.

Note that if I type :

documents = document_store.get_all_documents(filters={"link" : ["someLink"] })
print(documents[0].meta['arborescence']['theme'])

will give me "Justice"

So only the nested prop sounds disfunctioning

[tanaysoni]

Hi @guillim, could you try doing a query directly to Elasticsearch to see if you get the document?

{
    "query": {
        "bool": {
            "filter": [
                {
                    "terms": {
                        "arborescence. theme": [
                            "Justice"
                        ]
                    }
                }
            ]
        }
    }
}

The URL for the query is: http://{ES_HOST}:{ES_PORT}/{INDEX_NAME}/_search

[guillim]

still empty :

{
   "took":4,
   "timed_out":false,
   "_shards":{
      "total":1,
      "successful":1,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":{
         "value":0,
         "relation":"eq"
      },
      "max_score":null,
      "hits":[]
   }
}

By the way, this is what I see if I go to _search?pretty=true (I only showed 1 hit not to pollute the thread)

{
   "took":60,
   "timed_out":false,
   "_shards":{
      "total":1,
      "successful":1,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":{
         "value":10000,
         "relation":"gte"
      },
      "max_score":1.0,
      "hits":[
         {
            "_index":"document",
            "_type":"_doc",
            "_id":"36cd6576-0284-413d-a528-a04f1b830282",
            "_score":1.0,
            "_source":{
               "text":"\n Un mineur étranger résidant en France, n'est pas soumis à l'obligation de détenir un titre de séjour.\n\t\t\tToutefois, pour  faciliter ses déplacements hors de France, il peut obtenir un document de circulation pour étranger mineur (DCEM).",
               "question":"Document de circulation pour étranger mineur (DCEM) : \n Un mineur étranger résidant en France, n'est pas soumis à l'obligation de détenir un titre de séjour.\n\t\t\tToutefois, pour  faciliter ses déplacements hors de France, il peut obtenir un document de circulation pour étranger mineur (DCEM).",
               "question_emb":[
                  -0.02115986868739128,
                  -0.0036648884415626526,
                  -0.03444172441959381,
                  0.007581798825412989,
                  -0.03003448247909546,
                  -0.00936505664139986,
                  -0.01120445691049099,
                  -0.028182614594697952
               ],
               "name":"F2718--autre-situ-uation-15--pieces-a-f-ournir-16--mineur-res-france-38.json",
               "arborescence":{
                  "audience":"Particuliers",
                  "theme":"Étranger",
                  "dossier":"Titres, carte de séjour et documents de circulation pour étranger en France",
                  "sous_dossier":"Document de circulation pour mineur étranger",
                  "titre":"Document de circulation pour étranger mineur (DCEM)"
               },
               "link":"https://www.service-public.fr/particuliers/vosdroits/F2718"
            }
         }
      ]
   }
}

so the problem comes from ES not Haystack as such. maybe the mapping ?

[tanaysoni]

Could you check if the mapping has the correct types? You can query it by http://{ES_HOST}:{ES_PORT}/{INDEX_NAME}/_mapping

[guillim]

Here is my mapping :

{
   "document":{
      "mappings":{
         "properties":{
            "arborescence":{
               "properties":{
                  "audience":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "dossier":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "sous_dossier":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "theme":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "titre":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  }
               }
            },
            "link":{
               "type":"keyword"
            },
            "name":{
               "type":"keyword"
            },
            "question":{
               "type":"text"
            },
            "question_emb":{
               "type":"dense_vector",
               "dims":512
            },
            "text":{
               "type":"text"
            }
         }
      }
   }
}

I am very used to ES, so maybe the problem comes from that ?

[tanaysoni]

Yes, it seems to be a mapping issue. The theme field is of type text but it'd be better as a keyword field for an exact match. Did you create this index using Haystack?

Maybe it could still work with the query Justice in lowercase, i.e., justice?!

[guillim]

It works with justice uncased !

[tanaysoni]

Hi @moise-g, @guillim I'm closing this issue as nested fields can be queried using the dot notation. Here's an example for reference:

from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from haystack import Document

document_store = ElasticsearchDocumentStore()

documents = [
    Document(text="Doc1", meta={"prop": {"nested_prop": "test"}}),
    Document(text="Doc2", meta={"nested_prop": "test"}),
]
document_store.write_documents(documents)

documents = document_store.get_all_documents(filters={"prop.nested_prop": ["test"]})  # query nested field
assert documents[0].text == "Doc1"