Elasticsearch-dsl-py: Document.search fails when using custom index= value

[vanzi]

Using

elasticsearch==6.4.0
elasticsearch-dsl==6.4.0

given a Document subclass called Profile, the following code fails

>>> connection = connections.create_connection('default', hosts=['elasticsearch'])
>>> Profile.search(index='default', using=connection).execute()
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/srv/venv/lib/python3.7/site-packages/elasticsearch_dsl/response/__init__.py", line 25, in __repr__
    return '<Response: %r>' % (self.hits or self.aggregations)
  File "/srv/venv/lib/python3.7/site-packages/elasticsearch_dsl/response/__init__.py", line 47, in hits
    hits = AttrList(map(self._search._get_result, h['hits']))
  File "/srv/venv/lib/python3.7/site-packages/elasticsearch_dsl/utils.py", line 38, in __init__
    l = list(l)
  File "/srv/venv/lib/python3.7/site-packages/elasticsearch_dsl/search.py", line 221, in _get_result
    if hasattr(doc_type, '_matches') and doc_type._matches(hit):
  File "/srv/venv/lib/python3.7/site-packages/elasticsearch_dsl/document.py", line 117, in _matches
    return fnmatch(hit.get('_index', ''), cls._index._name) \
  File "/usr/local/lib/python3.7/fnmatch.py", line 35, in fnmatch
    pat = os.path.normcase(pat)
  File "/usr/local/lib/python3.7/posixpath.py", line 54, in normcase
    s = os.fspath(s)
TypeError: expected str, bytes or os.PathLike object, not NoneType

this happens when index is passed to search as a param and Profile.Index is not set (which in our case it can't be as there are multiple indices that are used with this document type).

I did some digging in the code and it seems that when Request._get_result calls the Document._matches method it does not pass the index name, the document method always uses cls._index._name which is None.

If I understand this code correctly then even if Profile.Index was set to something, it would still not work correctly when index is overwritten via search param since the values could be different.

So far our workaround was to override the _matches method and get rid of that comparison on index name, but I guess it was there for a reason so that probably breaks some use case, though I'm not sure what that could be -- if anyone could shed some light on this that would help.

[HonzaKral]

Thank you for the report, this indeed looks like a bug. The workaround you have makes perfect sense.

If you override index the current logic would still work when the index name on the document is set to a wildcard, which is what you would most likely do in that case - have for example multiple indices for profiles like profile-default and profile-special and your profile class would then be:

class Profile(Document):
  class Index:
    name = "profile-*"

[HonzaKral]

Thanks again for the report, @vanzi, the fix will go out with the next release.

[monomonedula]

@HonzaKral
I got exactly same problem with python 3.7, using
elasticsearch==7.0.0 elasticsearch-dsl==7.0.0

[HonzaKral]

@monomonedula that was my bad, I fixed the code but haven't done a release since then. Just released 7.1.0 containing the fix. Apologies for the delay!