Elasticsearch-py: TasksClient.get() requiring an argument on v7.5.1

[natefox]

Elasticsearch version (bin/elasticsearch --version): 7.1.1, 7.5.1

elasticsearch-py version (elasticsearch.__versionstr__): 7.5.1

Please make sure the major version matches the Elasticsearch server you are running.

Description of the problem including expected versus actual behavior:

Elasticsearch().tasks.get() fails using elasticsearch-py 7.5.1, whereas it succeeds using 7.1.1.

>>> import elasticsearch
>>> elasticsearch.__versionstr__
'7.5.1'
>>> elasticsearch.Elasticsearch().tasks.get()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nfox/myenv-7.5.1/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: get() missing 1 required positional argument: 'task_id'

>>> import elasticsearch
>>> elasticsearch.__versionstr__
'7.1.0'
>>> elasticsearch.Elasticsearch().tasks.get()
{'nodes': {'GkSSDiZPTBq4Tlv5xV9wtg': {'name': 'e6a01a4d549f', 'transport_address': '172.17.0.3:9300', 'host': '172.17.0.3', 'ip': '172.17.0.3:9300', 'roles': ['ingest', 'master', 'data', 'ml'], 'attributes': {'ml.machine_memory': '4129972224', 'xpack.installed': 'true', 'ml.max_open_jobs': '20'}, 'tasks': {'GkSSDiZPTBq4Tlv5xV9wtg:56': {'node': 'GkSSDiZPTBq4Tlv5xV9wtg', 'id': 56, 'type': 'direct', 'action': 'cluster:monitor/tasks/lists[n]', 'start_time_in_millis': 1582238526445, 'running_time_in_nanos': 8142200, 'cancellable': False, 'parent_task_id': 'GkSSDiZPTBq4Tlv5xV9wtg:55', 'headers': {}}, 'GkSSDiZPTBq4Tlv5xV9wtg:55': {'node': 'GkSSDiZPTBq4Tlv5xV9wtg', 'id': 55, 'type': 'transport', 'action': 'cluster:monitor/tasks/lists', 'start_time_in_millis': 1582238526442, 'running_time_in_nanos': 11192200, 'cancellable': False, 'headers': {}}}}}}

I've verified this against running both ES 7.5.1 and ES 7.1.1 (via docker on localhost)

Steps to reproduce:
run elasticsearch.Elasticsearch().tasks.get() while using elasticsearch-py 7.5.1

Ultimately, this is breaking curator's DeleteSnapshot as it checks for tasks.

[vchan2002]

Elasticsearch version (bin/elasticsearch --version): 7.6.0

elasticsearch-py version (elasticsearch.__versionstr__): 7.5.1

I am running into the same issue. I just upgraded my ES cluster from 6.8 to 7.6. I always have had a lambda that does a bunch of index maintenance, and the delete snapshot portion of it is broken. Other functions of my lambda (delete old indexes, generate snapshots)... etc... are fine.

[ERROR] TypeError: get() missing 1 required positional argument: 'task_id' Traceback (most recent call last): File "/var/task/handler.py", line 91, in serverlesscurator curator.DeleteSnapshots(snapshot_list).do_action() File "/var/task/curator/actions.py", line 1142, in do_action retry_interval=self.retry_interval, retry_count=self.retry_count): File "/var/task/curator/utils.py", lin [ERROR] TypeError: get() missing 1 required positional argument: 'task_id' Traceback (most recent call last): File "/var/task/handler.py", line 91, in serverlesscurator curator.DeleteSnapshots(snapshot_list).do_action() File "/var/task/curator/actions.py", line 1142, in do_action retry_interval=self.retry_interval, retry_count=self.retry_count): File "/var/task/curator/utils.py", line 1098, in safe_to_snap ongoing_task = find_snapshot_tasks(client) File "/var/task/curator/utils.py", line 1072, in find_snapshot_tasks tasklist = client.tasks.get() File "/var/task/elasticsearch/client/utils.py", line 84, in _wrapped return func(*args, params=params, **kwargs)

Thanks for pointing this out.... I don't recall this being an issue in the past, as I had always used a 6.x version of the client when I was running ES 6.3 thru 6.8. With the upgrade, I had to change requirements.txt to elasticsearch-curator==5.8.1, so it packaged elasticsearch-py to 7.5.1

[untergeek]

@sethmlarson Is this issue similar to what happened in #1141 ? Something with the new generated approach?

[natefox]

From what I can tell, https://github.com/elastic/elasticsearch/blob/7.6.0/rest-api-spec/src/main/resources/rest-api-spec/api/tasks.get.json (7.6.0 branch) needs another object under paths that would look like:

        {
          "path":"/_tasks",
          "methods":[
            "GET"
          ]
        }

I think adding that would remove the 'required' aspect of the task_id.

On a somewhat related note, I think that tasks.get.json file also needs a lot more under params based on the params listed in https://www.elastic.co/guide/en/elasticsearch/reference/master/tasks.html. I'm just very unsure if I'm looking in the right spot or if all those api files are somehow autogenerated and this isnt the root of the problem.

If this is the root of the problem, I'd happily make a PR.

[sethmlarson]

@natefox @untergeek It looks like the root of this issue is the YAML marking the task_id parameter as required via not having another path without the parameter as mentioned by @natefox. Another client generated via the YAML also has this issue. I'd open a PR upstream. :)

[karmi]

@natefox , I'm just driving by, but if you want to get a list of tasks — ie. without an ID —, you should perhaps use the tasks.list API?

[natefox]

Oh duh. Somehow looked right past that. So this tasks.list file is what's missing (a call to /tasks without requiring an ID). This tells me the issue is with the code auto generator and not elasticsearch's definitions.

[sethmlarson]

@natefox Hmm, when you say the issue is with the code generator what do you mean?
I took the comment as tasks.get() should have a required task_id and tasks.list() should be used if you don't have a task_id. Am I missing something?

[natefox]

@sethmlarson The docs do appear to indicate that list() should be used instead of .get() for this use case (listing tasks). However, it's a change from previous behavior and it even breaks one of Elastic's own software projects. Not that curator cant be fixed, but I wonder how many other people rely on this behavior in their own scripts?

https://github.com/elastic/curator/blob/master/curator/utils.py#L1167

[sethmlarson]

@natefox What I can do is add the feature back and emit a DeprecationWarning recommending switching to .list() because we don't want to continue with the old API which was incorrect. At least projects won't break then?

[natefox]

I think thats a fair approach while removing it for the 8.x.x release (at least moving to v8.0.0 conforms to semver's concept of breaking changes).

[sethmlarson]

This has been fixed in 7.6.0a1, full release will be coming sometime this week if all goes well.