Curator: Trying to run Curator on Amazon Linux 2

[pmcevoy]

## Expected Behavior

Trying to run simple Curator commands with curator action.yml or curator_cli

Actual Behavior

Seeing this in logs and cli output:

2019-09-02 15:48:34,564 DEBUG              curator.utils             get_client:813  Attempting to verify SSL certificate.
2019-09-02 15:48:34,571 DEBUG              curator.utils             get_client:855  Failed to import a module: /opt/elasticsearch-curator/lib/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_set_inheritable_async_safe

Steps to Reproduce the Problem

1. Spin up an AWS Amazon Linux 2 instance (eg amzn2-ami-hvm-2.0.20190618-x86_64-gp2)
2. Follow instructions here for enabling the yum repo: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html
3. I've tried a simple curator_cli show_indices (using a valid .curator/curator.yml file) and get the attached curator_cli.log

Specifications

• Version: Curator 5.7.6-1
• Platform: Amazon Linux 2, 20190618

Context (Environment)

Have not been able to get curator to run

Detailed Description

Given that Amazon Linux 2 does not ship with python3 (I know...) I have installed that using yum install python3 and with this result.

[ec2-user@ip-10-85-8-182 ~]$ sudo yum list installed | grep python3
python3.x86_64                        3.7.4-1.amzn2.0.1              @amzn2-core
python3-libs.x86_64                   3.7.4-1.amzn2.0.1              @amzn2-core
python3-pip.noarch                    9.0.3-1.amzn2.0.1              @amzn2-core
python3-setuptools.noarch             38.4.0-3.amzn2.0.6             @amzn2-core

curator.yml

client:
  hosts:
    - vpc-ding-XXXXXXXXXXXXXXXXXXX.eu-west-1.es.amazonaws.com
  port: 443
  url_prefix:
  use_ssl: True
  aws_region: eu-west-1
  aws_sign_request: True
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: false
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile:
  logformat: default
  blacklist: []

curator_cli.log

[untergeek]

This is an artifact of installing via RPM. Curator's RPM is not certified to run on Amazon Linux. You will have to install via pip, e.g. pip3 install -U --user elasticsearch-curator. You will almost certainly still see a similar error on Python 3.7.4 due to a PyYAML error, however. Curator 5.7.6 is only certified to run against Python 3.6. The upcoming release addresses this and will run with Python 3.8. You could build the master branch (or the 5.8 branch) with python 3.7 and it should run just fine.

[pmcevoy]

OK. That's a pity - I thought all required libraries were frozen into the RPM binaries (from docs)? Linux is not my primary platform so so I didn't really understand what that means.

Unfortunately too many dependency-yaks for me just to purge older indexes so I'll see if there is a way to script this.

[pmcevoy]

In the end, I had a much easier path using a docker image. Feels more sutainable longer term: use amazon-linux-extras to install docker, setup a cron to run:

@hourly /usr/bin/docker run -v ~ec2_user/curator:/curator/mounted bobrik/curator --config /curator/mounted/curator.yml /curator/mounted/action.yml