Elasticsearch-py: helpers.scan: TypeError: search() got an unexpected keyword argument 'doc_type'

[qilds123]

I'm using helpers.scan function to retrieve data. I passed in doc_type = log to it (following the online resource here). But I got this error:

<ipython-input-53-dffeaecb48f3> in export(self, outputFiles, host, indexDbName, docType, queryString, queryJson, size, fields, delimiter)
     41             w.writeheader()
     42             try:
---> 43                 for row in scanResponse:
     44                     for key,value in row['_source'].iteritems():
     45                         row['_source'][key] = unicode(value)

C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\helpers\actions.py in scan(client, query, scroll, raise_on_error, preserve_order, size, request_timeout, clear_scroll, scroll_kwargs, **kwargs)
    431     # initial search
    432     resp = client.search(
--> 433         body=query, scroll=scroll, size=size, request_timeout=request_timeout, **kwargs
    434     )
    435     scroll_id = resp.get("_scroll_id")



md5-74bebc22fdcc5578f71ec4ba94a7c930



C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\client\utils.py in _wrapped(*args, **kwargs)
     82                 if p in kwargs:
     83                     params[p] = kwargs.pop(p)
---> 84             return func(*args, params=params, **kwargs)
     85 
     86         return _wrapped

TypeError: search() got an unexpected keyword argument 'doc_type'

I use 'log' as the doctype and I'm using elasticsearch server 6.3.0
Did I set the doctype wrong? Thank you!

[CrimsonGlory]

got the same problem after the update from
elasticsearch==6.3.1 to elasticsearch==7.0.0

[fxdgear]

Thanks for the issue. I'll look into it.

FYI while using elasticsearch-py you should be matching major versions. I can't make any guarantee that using es-py==7 will work with older version of Elasticsearch.

There is a package released now you can install elasticsearch6 for a package that's designed to work with Elasticsearch 6.x.

https://github.com/elastic/elasticsearch-py#compatibility

[qilds123]

@fxdgear
Thanks for the reply
So you mean it may be the version issue but not the code
but is it still matter when I call the elasticsearch host rather than the localhost
Thanks a lot

[fxdgear]

@qilds123 well it IS the code, but starting in ES 7 doc_types are being phased out. The doc type _doc is default, and is used in all urls communicating with an ES 7 cluster. In ES8 the _doc doc_type is going away completely.

When I released v7 of this project, I change requirement of using doc_type as a required positional argument in each method to being an optional kwarg with a default of _doc. But turns out I removed it from search() but I had a gap in my testing :( SO it didn't get spotted before releasing 7.0.0.

[qilds123]

@fxdgear
So you mean maybe I choose the wrong doc type?
I first use _doc as doc type, but it can't work properly, then I checked the mapping of my json, the doc type is 'log', and when I run the python code, it really can create a csv file, but only have the column name without the data I want.

[fxdgear]

nope, it's my fault. I'm working on a fix now.

[qilds123]

@fxdgear Don't say that. I'm completely new to elasticsearch, maybe it's my fault.
Thanks a lot for helping me

[qilds123]

@fxdgear
I've already installed elasticsearch6
but the error still occurs

[sstoeckel]

any news here?

[fxdgear]

I pushed a new version, 7.0.1. I reworked some of the compatibility for ES 7.

[Haider8]

@fxdgear Hey, I am on version 7.0.1, but the error is still there for me. What can I do?

>>> es.search(index='my_index', doc_type='my_index',
... body={'query': {'match': {'text': 'this test'}}})
Traceback (most recent call last):
  File "<console>", line 2, in <module>
  File "/home/haider/Desktop/online courses/microblog/venv/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
>>> es.search(index='my_index', doc_type='my_index', body={'query': {'match': {'text': 'this test'}}})
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/haider/Desktop/online courses/microblog/venv/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'

And this is how I set the index:

>>> es.index(index='my_index', doc_type='my_index', id=1, body={'text': 'this is a test'})
{'_index': 'my_index', '_type': 'my_index', '_id': '1', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
>>> es.index(index='my_index', doc_type='my_index', id=2, body={'text': 'a second test'})
{'_index': 'my_index', '_type': 'my_index', '_id': '2', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}

[adtfrodo]

Hey
Any updates on the issue??

Cheers!!

[pietrolatorre]

any news? pls!

[browningtekum]

@fxdgear Same error. using django 1.11 python 2.7
Watching for any fixes

[Haider8]

You using python 2 😂😂😂. By the way, it worked when I remove doc_type
argument. Will see if I can carry on with this.

On Fri, May 24, 2019, 2:01 PM browningtekum <[email protected] wrote:

Same error. using django 1.11 python 2.7
Watching for any fixes

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch-py/issues/930?email_source=notifications&email_token=AKKLEUATCHHTXXN3MJG73L3PW6RV3A5CNFSM4HGHPJGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWERCMQ#issuecomment-495522098,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKKLEUCXTPUXHEF6JDWTQITPW6RV3ANCNFSM4HGHPJGA
.

[qilds123]

when i remove the doc_type then it can't print anything to the csv file

[fxdgear]

If you're using the 7.x version of this library with a cluster of elasticsearch pre 7 you will be having problems because the removal of doc_type from ES. I've cleaned up the code and pushed a 7.0.1 version of the client a little bit ago which should be working with all functionalities of Elasticsearch.

If you want to use the scan helper on ES 7 it does work, using the data created in the examples directory of this repo we can run a scan like so:

 from elasticsearch.helpers import scan

 results = scan(
        client,
        query={
            "query": {
                "bool": {
                    "must": {"match": {"description": "fix"}},
                    "must_not": {"term": {"files": "test_elasticsearch"}},
                }
            }
        },
        index="git",
    )
    for result in results:
    print_hit(result)

If you want to use the 7.x version of this library you must upgrade your cluster to 7.x
If you can't yet upgrade your cluster to 7.x, then please for now, keep using the 6.x version of this library.

[miaoweiwei]

My version of the elasticsearch service is 7.2.0, and the version of the elasticsearch in the python project is 7.0.2. This error has occurred,

es.search(index='my_index', doc_type='my_index',body={'query': {'match': {'text': 'this test'}}})
Traceback (most recent call last):
File "", line 1, in
File "D:MyprojectPythonFlaskFlask-Microblogvenvlibsite-packageselasticsearchclientutils.py", line 84, in _wrapped
return func(args, params=params, *kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'

[sethmlarson]

I believe this issue has been fixed in the recent releases of the client. Let me know and I will reopen if I'm mistaken.