Curator: Failed to import a module: No module named 'html.parser

[syst0m]

Using the untergeek/curator:5.7.6 docker image, curator fails silently on deleting indices action.
After enabling debug log, I see that it fails to load html.parser module.

Expected Behavior

Curator deletes the old indices.

Actual Behavior

Curator fails on loading html.parser module.

Steps to Reproduce the Problem

/ $ /curator/curator --config /tmp/config.yml /tmp/action_file.yml 
/ $ cat /tmp/curator.log 
2019-11-14 13:52:09,220 DEBUG                curator.cli                    run:110  Client and logging options validated.
2019-11-14 13:52:09,221 DEBUG                curator.cli                    run:114  default_timeout = 30
2019-11-14 13:52:09,221 DEBUG                curator.cli                    run:118  action_file: /tmp/action_file.yml
2019-11-14 13:52:09,226 DEBUG                curator.cli                    run:120  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'timeout_override': None, 'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'field': None, 'stats_result': None, 'epoch': None, 'exclude': False}]}}}
2019-11-14 13:52:09,226 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
2019-11-14 13:52:09,226 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'timeout_override': None, 'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'field': None, 'stats_result': None, 'epoch': None, 'exclude': False}]}}}
2019-11-14 13:52:09,227 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']")}
2019-11-14 13:52:09,227 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'timeout_override': None, 'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'field': None, 'stats_result': None, 'epoch': None, 'exclude': False}]}
2019-11-14 13:52:09,227 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2019-11-14 13:52:09,227 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'timeout_override': None, 'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'field': None, 'stats_result': None, 'epoch': None, 'exclude': False}]}
2019-11-14 13:52:09,230 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd4331a47b8>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd4331a4950>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd4331a4ae8>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd4331a4c80>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
2019-11-14 13:52:09,230 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}
2019-11-14 13:52:09,231 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function Filters.<locals>.f at 0x7fd4331a4378>
2019-11-14 13:52:09,231 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'field': None, 'stats_result': None, 'epoch': None, 'exclude': False}]
2019-11-14 13:52:09,231 DEBUG     curator.defaults.filtertypes                    age:55   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd433146158>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'timestring': Any(<class 'str'>, msg=None)}]
2019-11-14 13:52:09,232 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd433146158>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'timestring': Any(<class 'str'>, msg=None)}
2019-11-14 13:52:09,232 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'exclude': False}
2019-11-14 13:52:09,232 DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}
2019-11-14 13:52:09,232 DEBUG                curator.cli                    run:123  Full list of actions: {1: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}]}}
2019-11-14 13:52:09,232 DEBUG                curator.cli                    run:128  action_disabled = False
2019-11-14 13:52:09,232 DEBUG                curator.cli                    run:132  continue_if_exception = False
2019-11-14 13:52:09,232 DEBUG                curator.cli                    run:134  timeout_override = None
2019-11-14 13:52:09,232 DEBUG                curator.cli                    run:136  ignore_empty_list = True
2019-11-14 13:52:09,233 DEBUG                curator.cli                    run:138  allow_ilm_indices = False
2019-11-14 13:52:09,233 INFO                 curator.cli                    run:148  Preparing Action ID: 1, "delete_indices"
2019-11-14 13:52:09,233 INFO                 curator.cli                    run:162  Creating client object and testing connection
2019-11-14 13:52:09,233 DEBUG              curator.utils             get_client:809  kwargs = {'hosts': ['XXX:443'], 'aws_region': 'eu-west-1', 'aws_sign_request': True, 'url_prefix': '', 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'aws_token': None, 'master_only': False, 'aws_key': None, 'http_auth': None, 'aws_secret_key': None, 'certificate': None, 'client_key': None, 'client_cert': None, 'timeout': 30}
2019-11-14 13:52:09,281 DEBUG              curator.utils             get_client:858  Failed to import a module: No module named 'html.parser'

Specifications

• Version: 5.7.6
• Platform: docker

[untergeek]

Don't use the 5.7.6 image, which is outdated. Use the 5.8.1 image, which was pushed just last night to the Hub.

[syst0m]

@untergeek I tried it with the 5.8.1 image, and I get the same error.

[nukepuppy]

@untergeek same issue.. 5.8.1 image produces this error

[untergeek]

Hmmm. I'll have to see why it doesn't pick up the module. Bear with me on this. I do not have a quick or easy way to test an AWS connection (which is the only common thread here). It works against secured Elasticsearch otherwise, so this seems to be tied to boto and such.

[nukepuppy]

We had the cronjob succeeding but not doing its work... added debugging to the configs.. and that was the only error shown

[polothy]

so this seems to be tied to boto and such.

Thinking the same thing. Do you think it could be these lines? EG: boto is trying to import html.parser maybe?

https://github.com/elastic/curator/blob/bc0665d712ed076a7d6bb5aa0fe9ada8bc8e4de2/curator/utils.py#L865-L866

It seems like a likely place for the failure, if we look at the chain of calls here:

https://github.com/elastic/curator/blob/bc0665d712ed076a7d6bb5aa0fe9ada8bc8e4de2/curator/utils.py#L1012-L1015

• We see debug output for the call to process_ssl_args
• Don't see any exceptions being raised by process_aws_args
• Don't see the debug log message from try_aws_auth, so must have failed at try_boto_session?

Hope this helps and thanks so much for taking a look for us!

Cheers!

[untergeek]

Well, going down that path is a red herring. This is about Curator running on the Docker image not being able to find a dependency which _should_ have been copied over, or be present in the build.

It seems that something about this build process and/or copying the build directory to a child Docker image is causing Curator to not find the modules in the lib path. This works in the RPM/DEB builds. It successfully builds in the Docker build. What I can't tell is if the dependencies are being correctly copied over, or if it's a path discovery error or what.

[nukepuppy]

is there any sort of workaround for this? an explicit require to include or something of the sort?

[untergeek]

Not yet. I'm really frustrated with Docker and the ever pressing need to make images tiny. I will likely switch to a base CentOS image and simply install the current RPM in it as that shouldn't have the problems that this APK nonsense does. Yeah, it's bigger. But it will work, and you only have to download a base image once.

[nukepuppy]

@untergeek ditto.. Small price to pay for some confidence that things work

[nukepuppy]

@untergeek any luck? anything anyone can do to help?

[untergeek]

I have simply been too busy. I may have time later today, but can't promise anything.

[untergeek]

Sorry. I've been super busy, both with work and personal stuff. I will _maybe_ get a chance to engage with this over the holidays.

[untergeek]

That sure didn't happen. Had an unexpected funeral for my wife's grandmother, and subsequent bereavement leave. Not sure when I will be all caught up, since I'm barely back at work now.

[polothy]

Sorry to hear that, take care of yourself

[cromero-principe]

It seems there's some weird issues with six's import magic and python scripts packed with cx_Freeze. I was able to get past this issue by explicitly adding 'html.parser' to the packages list here:
https://github.com/elastic/curator/blob/a918e842463ee9bbaf0310866288c797d969176f/setup.py#L70-L71

and here: (I haven't tested this on Windows though)
https://github.com/elastic/curator/blob/a918e842463ee9bbaf0310866288c797d969176f/setup.py#L97-L98

[agimenez]

I can confirm that @cromero-principe's solution worked for me.

[jgournet]

Same: @cromero-principe solution worked for me - but I would really appreciate if this could be fixed by the maintainers.
@untergeek : any chance you can look at this ?

[untergeek]

I'm having some crazy stuff going on with new build changes. I hit a snag right before Christmas, and then took a break for COVID (my whole family got it), Christmas, and my wedding anniversary. I'm finally back in office today. Once I get some help with my build issues (entirely new package building process using Docker), I will look at making a release of a new semi-official Docker image. The truth is, the Curator build is working properly, with Python 3.8 on top of a built-from-source OpenSSL 1.1. Hopefully it will address what's going on. The current Docker build will not persist because of these issues. It will be replaced with one that works.