Elasticsearch-dsl-py: sort results to check if the field exits

[wdfsinap]

hi, when i sort the search results with sort(), some documents don't exit, so it goes error. a way is to use filter() to check if the documents have the filed to sort. but how to use filter to check if the field exits in the documents?

thanks

[HonzaKral]

There is the exists query/filter that does exactly that - https://www.elastic.co/guide/en/elasticsearch/reference/2.1/query-dsl-exists-query.html

To use it just do:

s = Search()
...
s = s.filter('exists', field='MY_FIELD_I_WANT_TO_SORT_BY')

Hope this helps.

[wdfsinap]

can i use this filter in ES.15?

[wdfsinap]

sorry ES 1.5

[HonzaKral]

yes

[wdfsinap]

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Q

client = Elasticsearch()

s = Search(using=client, index="wk").query().filter('exits',field='views')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "elasticsearch_dsl/search.py", line 224, in filter
    return self.query(Bool(filter=[Q(*args, **kwargs)]))
  File "elasticsearch_dsl/query.py", line 38, in Q
    return Query.get_dsl_class(name_or_query)(**params)
  File "elasticsearch_dsl/utils.py", line 196, in get_dsl_class
    raise UnknownDslObject('DSL class `%s` does not exist in %s.' % (name, cls._type_name))
elasticsearch_dsl.exceptions.UnknownDslObject: DSL class `exits` does not exist in query.

I use ES 1.5. what's the problem?

[njoannin]

@wdfsinap I think you made a typo: it should be "exists" and not "exits".

Also, I don't think you need the extra ".query()" in the middle.

Hope this helps!

[wdfsinap]

s = Search(using=client, index="pornwk").filter('exists',field='views').sort('-views')
response = s.execute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/elasticsearch_dsl/search.py", line 569, in execute
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 531, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[7B-RkEI5Rq2uLL2yqg3CPA][pornwk][0]: SearchParseException[[pornwk][0]: from[-1],size[-1],sort[<custom:"views": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@54bee3f8>!]: Parse Failure [Failed to parse source [{"sort": [{"views": {"order": "desc"}}], "query": {"bool": {"filter": [{"exists": {"field": "views"}}]}}}]]]; nested: QueryParsingException[[pornwk] bool query does not support [filter]]; }{[7B-RkEI5Rq2uLL2yqg3CPA][pornwk][1]: SearchParseException[[pornwk][1]: from[-1],size[-1],sort[<custom:"views": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@23d07f22>!]: Parse Failure [Failed to parse source [{"sort": [{"views": {"order": "desc"}}], "query": {"bool": {"filter": [{"exists": {"field": "views"}}]}}}]]]; nested: QueryParsingException[[pornwk] bool query does not support [filter]]; }{[7B-RkEI5Rq2uLL2yqg3CPA][pornwk][2]: SearchParseException[[pornwk][2]: from[-1],size[-1],sort[<custom:"views": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@4068da3f>!]: Parse Failure [Failed to parse source [{"sort": [{"views": {"order": "desc"}}], "query": {"bool": {"filter": [{"exists": {"field": "views"}}]}}}]]]; nested: QueryParsingException[[pornwk] bool query does not support [filter]]; }{[7B-RkEI5Rq2uLL2yqg3CPA][pornwk][3]: SearchParseException[[pornwk][3]: from[-1],size[-1],sort[<custom:"views": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@12c4a284>!]: Parse Failure [Failed to parse source [{"sort": [{"views": {"order": "desc"}}], "query": {"bool": {"filter": [{"exists": {"field": "views"}}]}}}]]]; nested: QueryParsingException[[pornwk] bool query does not support [filter]]; }{[7B-RkEI5Rq2uLL2yqg3CPA][pornwk][4]: SearchParseException[[pornwk][4]: from[-1],size[-1],sort[<custom:"views": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@2b8991bc>!]: Parse Failure [Failed to parse source [{"sort": [{"views": {"order": "desc"}}], "query": {"bool": {"filter": [{"exists": {"field": "views"}}]}}}]]]; nested: QueryParsingException[[pornwk] bool query does not support [filter]]; }]')

but still show error

[njoannin]

@wdfsinap Not sure why you get that error...

This part of the error message seems odd to me: Failed to execute phase [query], all shards failed.

I guess you'll have to wait for @HonzaKral for help on this one.

[wdfsinap]

s = Search(using=client, index="pornwk").sort('-views')
response = s.execute()
search without 'exists' filter works perfect. what's wrong with the filter?

[HonzaKral]

You are using a version of elasticsearch-dsl that is meant for elasticsearch 2.0 and newer, please use any of the release 0.x versions instead

[wdfsinap]

I use following method to install ES-dsl.
git clone https://github.com/elastic/elasticsearch-dsl-py
edit setup.py
elasticsearch>=2.0.0, <3.0.0 to elasticsearch-dsl<2.0.0
then install es-dsl by
python setup.py install
why i still install es-dsl 2.0?

[HonzaKral]

Master branch of elasticsearch-dsl is meant to work with new versions of elasticsearch only, it doesn't work with elasticsearch 1.5

To install proper version just run pip install "elasticsearch-dsl<1.0.0"

[wdfsinap]

works perfect after using pip install "elasticsearch-dsl<1.0.0"
thanks

[HonzaKral]

good to hear, thanks

[wdfsinap]

hi, how to do if I want to search a field not exists?

[HonzaKral]

s = s.exclude('exists', field='MY_FIELD_I_WANT_TO_SORT_BY')

[wdfsinap]

thanks. anyway to change a field's mapping type? like from Keyword() to Nested()
thanks

[HonzaKral]

Mappings cannot be changed once created, the way to do it is to delete the index, recreate with new mappings and then index the data again

[wdfsinap]

hi, when I try to search the query database with the field "links" empty or =[]

i= Search(using=client, index="goph").query(Q('match',links=[]))
j=i.execute()

errers:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch_dsl/search.py", line 698, in execute
    **self._params
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 655, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 185, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(500, u'illegal_state_exception', u"Can't get text on a START_ARRAY at 1:28")

how to get the right query?
thanks

[wdfsinap]

the field "links" is a nested type, I want to query the field's element's number.

[wdfsinap]

for example:

GET /goph/_search
{
  "query": {
    "nested":{
      "path":"links",
      "query":{
        "bool":{
          "must":{
            "term":{"links.name":"fembed"}
          }
      }
    }
    }
  }
}

I get 13 items. but when I use

GET /goph/_search
{
  "query": {
    "nested":{
      "path":"links",
      "query":{
        "bool":{
          "must_not":{
            "term":{"links.name":"fembed"}
          }
      }
    }
    }
  }
}

I get 0 item. how to get the rest items?

[wdfsinap]

@HonzaKral can you please help? thanks

[HonzaKral]

@wdfsinap I am sorry but I cannot help you without knowing your data. Also this is not the proper place for questions, since this is a question on how to use elasticsearch itself and not an issue with the python library I would recommend you ask your questions in the more appropriate elastic forum at https://discuss.elastic.co/

Thank you!

[sankalpgoibibo]

I was facing this error when I had import for from django.db.models import Q as well which led to Q object being ambiguous.