Curator: Dockerfile build from master has issues with pip requests module

[titpetric]

Actual Behavior

2019-09-04 07:25:48,007 INFO      Preparing Action ID: 1, "delete_indices"
2019-09-04 07:25:48,007 INFO      Creating client object and testing connection
2019-09-04 07:25:48,010 INFO      Instantiating client object
2019-09-04 07:25:48,010 ERROR     Unable to connect to Elasticsearch cluster. Error: Please install requests to use RequestsHttpConnection.
2019-09-04 07:25:48,010 CRITICAL  Curator cannot proceed. Exiting.

Expected Behavior

No error.

Detailed Description

I've created a docker image based of the master branch, and used two samples to provide /.curator/curator.yml (src) and pass a single action to curator (src - enabled as per docs). Running the container however produces an error, and doesn't complete succesfully.

I'm not particularly sure why this is happening, as the /curator/lib/requests location exists in the resulting docker image.

~
/curator $ find | grep req
./lib/requests
./lib/requests/_internal_utils.pyc
./lib/requests/certs.pyc
./lib/requests/status_codes.pyc
./lib/requests/hooks.pyc
./lib/requests/adapters.pyc
./lib/requests/models.pyc
./lib/requests/__version__.pyc
./lib/requests/exceptions.pyc
./lib/requests/cookies.pyc
./lib/requests/packages.pyc
./lib/requests/structures.pyc
./lib/requests/api.pyc
./lib/requests/__init__.pyc
./lib/requests/compat.pyc
./lib/requests/utils.pyc
./lib/requests/auth.pyc
./lib/requests/sessions.pyc
~

[titpetric]

I've tried editing Dockerfile to remove the multi-stage build part, to see if something wasn't copied over. I get the same result, so I'm thinking this is going to be some sort of dependency problem with requirements.txt

I also tried bumping the version of requests to >=2.22 (the latest version), same result. Any hints are appreciated.

The docker build on docker hub from #528 is functional however:

~
Digest: sha256:dacd0d7c3a2940dd88280378cf37bdbb21b1ae94d871cb564772c0020fac8bd4
Status: Downloaded newer image for untergeek/curator:5.7.6
2019-09-04 07:54:01,130 INFO Preparing Action ID: 1, "delete_indices"
2019-09-04 07:54:01,141 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 5 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2019-09-04 07:54:01,293 INFO Deleting 2 selected indices: ['logstash-2019.08.26-000001', 'apm-7.3.1-onboarding-2019.08.27']
2019-09-04 07:54:01,293 INFO ---deleting index logstash-2019.08.26-000001
2019-09-04 07:54:01,294 INFO ---deleting index apm-7.3.1-onboarding-2019.08.27
2019-09-04 07:54:01,473 INFO Action ID: 1, "delete_indices" completed.
2019-09-04 07:54:01,473 INFO Job completed.
~

At this point I'm assuming that the master is broken and outdated (<5.x). If that's the case, please close this issue.

[untergeek]

It's actually the way it's being frozen. See here:

RUN python3 setup.py build_exe

I don't see this as necessary, as it should still run just fine if installed via python3 setup.py install, though the entry point would have to change.

[untergeek]

I know this because I was trying to release 5.8.0 and my package builds (RPM/DEB) are suffering the same problem. I haven't had time to troubleshoot this yet, but I'm going to see what I can figure out.

[luanguimaraesla]

I had to use develop to make it run

FROM python:3.6-alpine3.6 as builder

RUN apk --no-cache upgrade && apk --no-cache add build-base tar musl-utils openssl-dev
RUN pip3 install setuptools cx_Freeze==6.0b1

COPY . .
RUN ln -s /lib/libc.musl-x86_64.so.1 ldd
RUN ln -s /lib /lib64
RUN pip3 install -r requirements.txt
RUN python3 setup.py develop

RUN apk --no-cache upgrade && apk --no-cache add ca-certificates && update-ca-certificates && apk --no-cache add wget
ENTRYPOINT ["python3", "run_curator.py"]

[pappnase99]

Ran into this issue with tag 5.6.1, is there a working version?

[untergeek]

apologies. I will try to build a 5.8.1 tag tonight

[untergeek]

I will commit the change to Dockerfile in master soon. The result is a few megabytes larger than previous releases, but it works now (I tested it myself).

I have pushed a working image to untergeek/curator:5.8.1.

[czerasz-flaconi]

@untergeek did You manage to commit the changes mentioned here:

I will commit the change to Dockerfile in master soon. The result is a few megabytes larger than previous releases, but it works now (I tested it myself).

I build the Docker image from v5.8.1:

CURATOR_VERSION='v5.8.1'
docker build "https://github.com/elastic/curator.git#${CURATOR_VERSION}" -t "${DOCKER_IMAGE_NAME}"

But still get:

2020-03-13 11:04:54,642 INFO      Preparing Action ID: 1, "delete_indices"
2020-03-13 11:04:54,642 INFO      Creating client object and testing connection
2020-03-13 11:04:54,645 INFO      Instantiating client object
2020-03-13 11:04:54,645 ERROR     Unable to connect to Elasticsearch cluster. Error: Please install requests to use RequestsHttpConnection.
2020-03-13 11:04:54,645 CRITICAL  Curator cannot proceed. Exiting.

[untergeek]

The change is in master. To use the current version, you'd have to clone the repository and just run docker build Dockerfile

When a new version gets released, it will be 5.8.2

[czerasz-flaconi]

Hi @untergeek , thanks for the quick response! Waiting for v5.8.2 then.

[jeromba6]

Hi @untergeek we are running in to the same problem any clue of the ETA of 5.8.2? Is this still actively maintained? Last tag is from september last year, about 11 months ago.

[nsharmacovs]

Hey guys any update on this issue. ?? Was anyone able to find work around? If yes, can you please list the steps?

[larsf]

doing a little debugging into this issue, I found that that reason curator failed with this error was that when the elasticsearch package tried to load the requests package - it failed to do so, because during the import of requests, the following exception occurs:

import requests

File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 121, in
from . import packages
File "/usr/lib/python2.7/site-packages/requests/packages.py", line 7, in
locals()[package] = __import__(package)
File "/usr/lib/python2.7/site-packages/idna/__init__.py", line 2, in
from .core import *
File "/usr/lib/python2.7/site-packages/idna/core.py", line 1, in
from . import idnadata
ImportError: cannot import name idnadata

looking at the frozen lib/idna directory, you see the following files:
core.pyc __init__.pyc intranges.pyc package_data.pyc uts46data.pyc

while the site-package/idna directory contains:
codec.py compat.py core.py idnadata.py __init__.py intranges.py package_data.py uts46data.py
codec.pyc compat.pyc core.pyc idnadata.pyc __init__.pyc intranges.pyc package_data.pyc uts46data.pyc.

It seems like cx_Freeze is having problems detecting dependencies that are imported like idnadata is:

in idna/core.py:
from . import idnadata

others have solved it by explicitly including the idnadata dependency as seen here (https://github.com/marcelotduarte/cx_Freeze/issues/389)
by adding includes =["idna.idnadata"] to the buildOptions.

from:
buildOptions = dict(
packages = [],
excludes = [],
include_files = [cert_file],
)

to
buildOptions = dict(
packages = [],
excludes = [],
includes = ["idna.idnadata"],
include_files = [cert_file],
)

This was with cx_Freeze 5.1.1 and python2.7 mind you

Of course this begs the questions - what other dependencies are missed??

[untergeek]

This sounds off. When I build, I use cx_Freeze v6.x, and Python 3.6+. Not sure where these lower requirements are coming from. Are they hardcoded in the Dockerfile?