Curator: AWS - Elasticsearch support - /_cluster/state/metadata/ HTTP 401

[ajohnstone]

Currently AWS Elasticsearch 2.3 doesn't support the following

2016-08-01 10:27:51,760 ERROR                curator.cli                    cli:242  Failed to complete action: delete_indices.  <class 'elasticsearch.exceptions.AuthenticationException'>: TransportError(401, u'{"Message":"Your request: \'/_cluster/state/metadata/.kibana-4,command.php,elastalert_status,logstash-2016.07.31,logstash-2016.08.01\' is not allowed."}')

A ticket has been opened internally for AWS, however curious if this can be resolved here alternatively? Or at least raise awareness of the issue.

[ajohnstone]

It seems these maybe a regression?

https://github.com/elastic/curator/issues/491
https://github.com/elastic/curator/issues/685

[untergeek]

If you read this comment in #685, you'll see that it's not a regression. Curator 4 _depends_ on the _cluster/state endpoint, and AWS ES does not provide it.

When I wrote Curator 4, which is a major change over Curator 3:

1. Due to new APIs, Curator 4 was not going to be reverse compatible with pre 2.0 versions of Elasticsearch.
2. AWS ES was tied to ES version 1.5
3. I hoped that if they ever released a newer version it would support the _cluster/state endpoint.

AWS ES now supports a newer version of Elasticsearch, but it still doesn't support the _cluster/state endpoint.

Why does Curator 4 depend on the _cluster/state endpoint, when Curator 3 did not? Because Curator 4 extends a lot of functionality that Curator 3 did not have, like the ability to filter by age with the index creation_date. The method in question is IndexList._get_metadata. Other code that depends on the cluster state includes the master_only testing and checking whether indices are open or closed. A workaround to not use cluster state would mean having to complicate the code, considerably, and it would result in a fragmented user experience, as it would mean that some users simply couldn't use some functionality. I got into that in Curator 3, and it was a pain to maintain and document, and I was constantly having to explain that feature X was not supported in use case Y. I want to keep the experience uniform, so I am using the simplest way forward, which is to use _cluster/state.

As mentioned in that linked comment, the best case scenario for you would be to use Curator 3 still, as it supports versions of ES up through 2.x. I also suggest asking AWS ES to change the policy of disallowing the _cluster/state endpoint, since it was their decision to remove that. Perhaps they could be persuaded to reverse that decision.

[ajohnstone]

Thanks @untergeek I've referenced this ticket in my support case with aws. So hopefully will be listening :)

[basex]

I'm having the same problem. Made a post on the AWS forums related with this https://forums.aws.amazon.com/thread.jspa?threadID=236643&tstart=0

I think there is no point in support the AWS Credentials in Curator 4 if there is no way this tool can work with AWS ES.

[malpani]

@untergeek I am from AWS ES and we want to ideally be able to support curator which is a super useful plugin. While exposing all the information in cluster/state will be harder for us, I would like to explore the path of opening this API and exposing enough information for curator to work.

While I know the best answer is to look up code, could you please share some relevant pointers on the portions of _cluster/state response which get used here?

[untergeek]

I think there is no point in support the AWS Credentials in Curator 4 if there is no way this tool can work with AWS ES.

@basex Not necessarily. This may be useful for people who want to run Curator remotely against self-hosted ES clusters residing in AWS

[untergeek]

@malpani

See https://github.com/elastic/curator/blob/v4.0.0/curator/indexlist.py#L146-L149

                self.client.cluster.state(
                    index=to_csv(l),metric='metadata'
                )['metadata']['indices']

and https://github.com/elastic/curator/blob/v4.0.0/curator/utils.py#L400

master_node_id = client.cluster.state(metric='master_node')['master_node']

The biggest part is really the index metadata in https://github.com/elastic/curator/blob/v4.0.0/curator/indexlist.py#L137-L166, where you see that Curator _currently_ uses (this may expand at any time):

• number_of_replicas
• number_of_shards
• state
• creation_date
• routing

Most of these are under cluster.state['metadata']['indices'][INDEXNAME]['settings']['index'], though the index state (open/closed) is at cluster.state['metadata']['indices'][INDEXNAME]['state']

[kimchy]

obviously, a reminder, Elastic official cloud offering, https://www.elastic.co/cloud, doesn't suffer from this limitation of AWS and curator works out of the box with it. This is being helped with the fact that both the devs behind ES and @untergeek work for Elastic and can make sure this happens. Obviously any fork of curator is not officially supported by Elastic.

[chetandhembre]

@untergeek have check out /_all api? I guess it provides all required details.

[untergeek]

@chetandhembre I can't find that API call in the elasticsearch-py.readthedocs.io documentation. I'm not looking to hack the call directly.

[chetandhembre]

@untergeek
self.client.indices.get('_all') is the api.
It wont give you index state and routing info, sadly.

[dpgaspar]

Testing AWS ES, already found this issue related with td-agent or logstash:

https://forums.aws.amazon.com/thread.jspa?messageID=694137#694137

Non related but...

Is this issue solved? since it's closed, it's not clear to me.

[untergeek]

It's currently not solvable due to AWS ES not supporting the required API call.

[dpgaspar]

Thanks @untergeek for the quick reply.

Does curator 3 work with AWS ES.

[untergeek]

Yes. It lacks some of the newer features in 4, but it should work with AWS ES.

[RyanBowlby-Reflektion]

I'm certain that tens of people have attempted to use curator 4 with AWS only to end up on this issue. I know it's quite squarely AWS who needs to implement use of _cluster/state, but in the meantime any chance you can add a banner warning to curator 4 mentioning it's not compatible? It'd save users a good bit of hassle.

[untergeek]

@RyanBowlby-Reflektion I'm sorry you had a bad experience. I keep putting notices everywhere I can think of.

This isn't a huge banner, but it's in the README:

It is also important to note that Curator 4 requires access to the /_cluster/state/metadata endpoint. Forks of Elasticsearch which do not support this endpoint (such as AWS ES, see #717) will not be able to use Curator version 4.

In the official documentation, I added the following warning to each of the AWS configuration flags here, here, and here.

This feature allows connection to AWS using IAM credentials, but Curator 4 does not currently work with AWS.

Which links to this FAQ response.

I also removed the AWS IAM flags from the configuration file example. Note the difference in the older 4.0 documentation here.

I'm not sure where else I can put notices to make it more plain.

[algesten]

Too small notice for me to not waste 2h just to end up at this dead end. How about making it clear in the version matrix in the README?

[jordansissel]

@algesten it looks like your request to make it clear in the README is resolved. At this time (about 7 days ago, by the commit logs), I see this in the readme table:

[jimmycuadra]

In case anyone lands on this issue wondering about Curator support for the new version 5 of the AWS Elasticsearch Service, take a look at this issue: https://github.com/elastic/curator/issues/880

[oji]

Hi, @untergeek, I'm trying to use curator 3.5.1 with AWS ES 2.3, but I can't find in the docs how to pass aws_key or aws_secret to curator 3.5. Can you give me any pointer on how to do this?

Thanks!

[untergeek]

@oji It was never officially supported.