Aws-cli: cloud-init is failing due to urllib3 dependency

[aabhassharma]

Seeing this issue with awscli 1.16.41, which does seem to pin boto to a newer version, which in turn seems to pick the correct urllib3 (which is where the ordered_dict problem starts), but it's still broken.
To make some sense of what I mean above:

installed version of awscli: 1.16.41
relevant commit line in awscli: https://github.com/aws/aws-cli/commit/5a421214c3c99d51061d89ae2bd58f68ec2bda36#diff-2eeaed663bd0d25b7e608891384b7298R26
relevant commit in boto: https://github.com/boto/botocore/commit/c4d3b1a1ee33db3c4dc8fb1aeba6dc362cfa2cd5

Error from system log:
Traceback (most recent call last): File "/usr/bin/cloud-init", line 43, in <module> from cloudinit import netinfo File "/usr/lib/python2.7/dist-packages/cloudinit/netinfo.py", line 23, in <module> import cloudinit.util as util File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 59, in <module> from cloudinit import url_helper File "/usr/lib/python2.7/dist-packages/cloudinit/url_helper.py", line 27, in <module> import requests File "/usr/lib/python2.7/dist-packages/requests/__init__.py", line 58, in <module> from . import utils File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 25, in <module> from .compat import parse_http_list as _parse_list_header File "/usr/lib/python2.7/dist-packages/requests/compat.py", line 92, in <module> from urllib3.packages.ordered_dict import OrderedDict ImportError: No module named ordered_dict

Relevant discussion from urllib3:
https://github.com/urllib3/urllib3/issues/1456

[ghost]

I have also seen this issue. Installing the latest aws-cli on an image is preventing new AMIs based on that image from booting due to the above issue with urllib3.

[mflopez79]

I'm facing the same issue on Ubuntu 14.04 (AWS EC2), but it's reproducible on Vagrant. Boot a clean machine and cloud-init works (check with cloud-init init). Get awscli installed with pip and cloud-init shows the same error that @aabhassharma reported when cloud-init init is executed

[mflopez79]

Did some more checks, version 1.16.38 is the last one that doesn't break cloud-init. Starting from version 1.16.39 it breaks cloud-init imports.
What I did as a workaround was install version 1.16.38 as

pip install awscli==1.16.38

[joguSD]

This is an issue with how Python packages are installed and upstream in urllib3/requests.
AWS CLI v1.16.39 and greater bumped the version range for urllib3, so pip is pulling in the latest version of urllib3 which is incompatible with the version of requests that cloud-init is using.

You can use newer versions of the CLI, you'll just need to help pip by specifying that you want an older version of urllib3==1.23. You could also try pulling a newer version of requests, which supports the latest version of urllib3.

See: https://github.com/urllib3/urllib3/issues/1456

[ciandro]

I have forced a requests update to v2.20 which takes care of urllib3 1.24 and now all my automation is working.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[richid]

This issue should be reopened. I've spent many hours trying to track this down and I am completely horrified that installing a new version of awscli is responsible for breaking my instance launches.

[wvidana]

This is still going on. I had to specifically pip upgrade requests in my packer builds to avoid cloud-init failure

[joguSD]

@richid Unfortunately, there's nothing we can do to fix this for you. This is not an issue with the awscli, but an issue with installing any Python package globally that allows a newer version of urllib3 that conflicts with cloud-init. In general, installing Python packages globally into a shared environment can be problematic.

There's three options to get around this issue:

1) Carefully curate your global Python environment to ensure you're installing compatible versions of all your packages. In this case, installing an older version of urllib3 or a newer version of requests.
2) Properly isolate Python packages into virtual environments so tools can have their own set of isolated dependencies.
3) Use the bundled installer to install the AWS CLI which will handle setting up an isolated virtual environment to avoid conflicts like these.

[patrobinson]

This is one of the many reasons why awscli should just be a statically compiled binary. It's typically required to be installed on a system with various other requirements and as such is easily broken by requirements from other dependencies.

[mipearson]

@joguSD Is this official AWS advice, specifically option 3? This just bit us (twice) and we're preparing a general advice for other teams so that they can avoid their instances not booting.