Aws-cli: ImportError: cannot import name fix_s3_host

Created on 22 May 2015 · 16Comments · Source: aws/aws-cli

Version 1.7.29 appears to be broken. Running the aws command line give this error:

$ aws
Traceback (most recent call last):
File "/usr/local/bin/aws", line 27, in
sys.exit(main())
File "/usr/local/bin/aws", line 23, in main
return awscli.clidriver.main()
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 47, in main
driver = create_clidriver()
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 56, in create_clidriver
event_hooks=emitter)
File "/usr/local/lib/python2.7/dist-packages/awscli/plugin.py", line 44, in load_plugins
modules = _import_plugins(plugin_mapping)
File "/usr/local/lib/python2.7/dist-packages/awscli/plugin.py", line 61, in _import_plugins
module = __import__(path, fromlist=[module])
File "/usr/local/lib/python2.7/dist-packages/awscli/handlers.py", line 48, in
from awscli.customizations.s3endpoint import register_s3_endpoint
File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3endpoint.py", line 29, in
from botocore.utils import fix_s3_host
ImportError: cannot import name fix_s3_host

Version 1.7.28 does not have this problem.

closing-soon

Source

russellneufeld

Most helpful comment

Yeah that is strange. I was able to reproduce your issue. It looks like it is something on how pip figures out dependencies and the fact that there has not been a release of boto3 where we take on the alpha release version of botocore as a dependancy. To make your requirements.txt work, try adding the --pre argument to your pip install command.

Or edit your requirements.txt to (note that you exclude botocore, you can usually rely on the CLI to be using the latest version of botocore):

boto3

Let me know if that helps.

kyleknap on 27 May 2015

👍3

All 16 comments

How did you upgrade? It looks like you're not using the latest version of botocore, which should have been automatically upgraded when you ran "pip install --upgrade awscli". You can check via $ python -c "import botocore; print(botocore.__version__)" which should print 1.0.0a.

If for some reason it didn't pick up that dependency you can try manually installing it via pip install botocore==1.0.0a1.

jamesls on 22 May 2015

You are right that the version of botocore is one behind. I ran pip install -U botocore but it didn’t upgrade me to 1.0.0a1. I’m still at 0.109.0. Is there a chance that the latest botocore is not on pypi yet?

On May 22, 2015, at 2:02 PM, James Saryerwinnie [email protected] wrote:

How did you upgrade? It looks like you're not using the latest version of botocore, which should have been automatically upgraded when you ran "pip install --upgrade awscli". You can check via $ python -c "import botocore; print(botocore.version)" which should print 1.0.0a.

If for some reason it didn't pick up that dependency you can try manually installing it via pip install botocore==1.0.0a1.

—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-104730372.

russellneufeld on 22 May 2015

It's on pypi: https://pypi.python.org/pypi/botocore/1.0.0a1

You wouldn't happen to be in Australia by chance? We saw a similar issue with pypi a few months ago: https://github.com/aws/aws-cli/issues/973

jamesls on 22 May 2015

Nope, US. Weird that my ec2 instances aren’t seeing it. Maybe I’m hitting some mirror which hasn’t been updated yet?

On May 22, 2015, at 2:12 PM, James Saryerwinnie [email protected] wrote:

It's on pypi: https://pypi.python.org/pypi/botocore/1.0.0a1 https://pypi.python.org/pypi/botocore/1.0.0a1
You wouldn't to be in Australia by chance? We saw a similar issue with pypi a few months ago: #973 https://github.com/aws/aws-cli/issues/973
—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-104733741.

russellneufeld on 22 May 2015

Interesting. What region are your instances in? I'll see if I can repro.

jamesls on 22 May 2015

us-east-1

On May 22, 2015, at 2:30 PM, James Saryerwinnie [email protected] wrote:

Interesting. What region are your instances in? I'll see if I can repro.

—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-104738432.

russellneufeld on 22 May 2015

Just tried it on Amazon Linux in us-east-1, and I'm not seeing any issues. Perhaps you are in fact hitting some mirror or something that hasn't been updated yet:

$ sudo pip install --upgrade awscli
You are using pip version 6.0.8, however version 7.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting awscli
  Downloading awscli-1.7.29.tar.gz (334kB)
    100% |################################| 335kB 822kB/s
Collecting botocore==1.0.0a1 (from awscli)
  Downloading botocore-1.0.0a1.tar.gz (1.6MB)
    100% |################################| 1.6MB 151kB/s
Collecting bcdoc<0.15.0,>=0.14.0 (from awscli)
  Downloading bcdoc-0.14.0-py2.py3-none-any.whl
Collecting colorama<=0.3.3,>=0.2.5 from https://pypi.python.org/packages/source/c/colorama/colorama-0.3.3.tar.gz#md5=a56b8dc55158a41ab3c89c4c8feb8824 (from awscli)
  Downloading colorama-0.3.3.tar.gz
Collecting docutils>=0.10 from https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5=4622263b62c5c771c03502afa3157768 (from awscli)
  Downloading docutils-0.12.tar.gz (1.6MB)
    100% |################################| 1.6MB 150kB/s
Collecting rsa<=3.1.4,>=3.1.2 from https://pypi.python.org/packages/source/r/rsa/rsa-3.1.4.tar.gz#md5=b6b1c80e1931d4eba8538fd5d4de1355 (from awscli)
  Downloading rsa-3.1.4.tar.gz
Collecting jmespath==0.7.1 (from botocore==1.0.0a1->awscli)
  Downloading jmespath-0.7.1-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 from https://pypi.python.org/packages/any/p/python-dateutil/python_dateutil-2.4.2-py2.py3-none-any.whl#md5=c817289a1d95a355cc088e2292791a77 (from botocore==1.0.0a1->awscli)
  Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)
    100% |################################| 192kB 1.2MB/s
Collecting six<2.0.0,>=1.8.0 from https://pypi.python.org/packages/3.3/s/six/six-1.9.0-py2.py3-none-any.whl#md5=9ac7e129a80f72d6fc1f0216f6e9627b (from bcdoc<0.15.0,>=0.14.0->awscli)
  Downloading six-1.9.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.1.4,>=3.1.2->awscli)
  Downloading pyasn1-0.1.7.tar.gz (68kB)
    100% |################################| 69kB 5.5MB/s
Installing collected packages: pyasn1, six, python-dateutil, jmespath, rsa, docutils, colorama, bcdoc, botocore, awscli
  Running setup.py install for pyasn1
  Found existing installation: six 1.8.0
    Uninstalling six-1.8.0:
      Successfully uninstalled six-1.8.0

  Found existing installation: python-dateutil 2.1
    Uninstalling python-dateutil-2.1:
      Successfully uninstalled python-dateutil-2.1

  Found existing installation: jmespath 0.6.1
    Uninstalling jmespath-0.6.1:
      Successfully uninstalled jmespath-0.6.1

  Found existing installation: rsa 3.1.2
    Uninstalling rsa-3.1.2:
      Successfully uninstalled rsa-3.1.2
  Running setup.py install for rsa
    Installing pyrsa-encrypt-bigfile script to /usr/local/bin
    Installing pyrsa-encrypt script to /usr/local/bin
    Installing pyrsa-verify script to /usr/local/bin
    Installing pyrsa-sign script to /usr/local/bin
    Installing pyrsa-priv2pub script to /usr/local/bin
    Installing pyrsa-decrypt script to /usr/local/bin
    Installing pyrsa-decrypt-bigfile script to /usr/local/bin
    Installing pyrsa-keygen script to /usr/local/bin
  Found existing installation: docutils 0.11
    DEPRECATION: Uninstalling a distutils installed project (docutils) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling docutils-0.11:
      Successfully uninstalled docutils-0.11
  Running setup.py install for docutils
    changing mode of build/scripts-2.7/rst2html.py from 644 to 755
    changing mode of build/scripts-2.7/rst2s5.py from 644 to 755
    changing mode of build/scripts-2.7/rst2latex.py from 644 to 755
    changing mode of build/scripts-2.7/rst2xetex.py from 644 to 755
    changing mode of build/scripts-2.7/rst2man.py from 644 to 755
    changing mode of build/scripts-2.7/rst2xml.py from 644 to 755
    changing mode of build/scripts-2.7/rst2pseudoxml.py from 644 to 755
    changing mode of build/scripts-2.7/rstpep2html.py from 644 to 755
    changing mode of build/scripts-2.7/rst2odt.py from 644 to 755
    changing mode of build/scripts-2.7/rst2odt_prepstyles.py from 644 to 755
    changing mode of /usr/local/bin/rst2man.py to 755
    changing mode of /usr/local/bin/rst2s5.py to 755
    changing mode of /usr/local/bin/rst2html.py to 755
    changing mode of /usr/local/bin/rstpep2html.py to 755
    changing mode of /usr/local/bin/rst2odt_prepstyles.py to 755
    changing mode of /usr/local/bin/rst2pseudoxml.py to 755
    changing mode of /usr/local/bin/rst2latex.py to 755
    changing mode of /usr/local/bin/rst2odt.py to 755
    changing mode of /usr/local/bin/rst2xetex.py to 755
    changing mode of /usr/local/bin/rst2xml.py to 755
  Found existing installation: colorama 0.2.5
    DEPRECATION: Uninstalling a distutils installed project (colorama) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling colorama-0.2.5:
      Successfully uninstalled colorama-0.2.5
  Running setup.py install for colorama
  Found existing installation: bcdoc 0.13.0
    Uninstalling bcdoc-0.13.0:
      Successfully uninstalled bcdoc-0.13.0

  Found existing installation: botocore 0.95.0
    Uninstalling botocore-0.95.0:
      Successfully uninstalled botocore-0.95.0
  Running setup.py install for botocore
  Running setup.py install for awscli
    changing mode of build/scripts-2.7/aws from 644 to 755
    changing mode of build/scripts-2.7/aws.cmd from 644 to 755
    changing mode of build/scripts-2.7/aws_completer from 644 to 755
    changing mode of build/scripts-2.7/aws_zsh_completer.sh from 644 to 755
    changing mode of /usr/local/bin/aws to 755
    changing mode of /usr/local/bin/aws_completer to 755
    changing mode of /usr/local/bin/aws.cmd to 755
    changing mode of /usr/local/bin/aws_zsh_completer.sh to 755
Successfully installed awscli-1.7.29 bcdoc-0.14.0 botocore-1.0.0a1 colorama-0.3.3 docutils-0.12 jmespath-0.7.1 pyasn1-0.1.7 python-dateutil-2.4.2 rsa-3.1.4 six-1.9.0

jamesls on 22 May 2015

👍3

This is what I get:

$ sudo pip install -U botocore
/usr/local/lib/python2.7/dist-packages/pip-7.0.0-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Requirement already up-to-date: botocore in /usr/local/lib/python2.7/dist-packages
Requirement already up-to-date: jmespath==0.7.1 in /usr/local/lib/python2.7/dist-packages (from botocore)
Requirement already up-to-date: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python2.7/dist-packages (from botocore)
Requirement already up-to-date: six>=1.5 in /usr/local/lib/python2.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore)

$ pip freeze | grep botocore
botocore==0.109.0

On May 22, 2015, at 2:49 PM, James Saryerwinnie [email protected] wrote:

Just tried it on Amazon Linux in us-east-1, and I'm not seeing any issues. Perhaps you are in fact hitting some mirror or something that hasn't been updated yet:

$ sudo pip install --upgrade awscli
You are using pip version 6.0.8, however version 7.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting awscli
Downloading awscli-1.7.29.tar.gz (334kB)
100% |################################| 335kB 822kB/s
Collecting botocore==1.0.0a1 (from awscli)
Downloading botocore-1.0.0a1.tar.gz (1.6MB)
100% |################################| 1.6MB 151kB/s
Collecting bcdoc<0.15.0,>=0.14.0 (from awscli)
Downloading bcdoc-0.14.0-py2.py3-none-any.whl
Collecting colorama<=0.3.3,>=0.2.5 from https://pypi.python.org/packages/source/c/colorama/colorama-0.3.3.tar.gz#md5=a56b8dc55158a41ab3c89c4c8feb8824 (from awscli)
Downloading colorama-0.3.3.tar.gz
Collecting docutils>=0.10 from https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5=4622263b62c5c771c03502afa3157768 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 150kB/s
Collecting rsa<=3.1.4,>=3.1.2 from https://pypi.python.org/packages/source/r/rsa/rsa-3.1.4.tar.gz#md5=b6b1c80e1931d4eba8538fd5d4de1355 (from awscli)
Downloading rsa-3.1.4.tar.gz
Collecting jmespath==0.7.1 (from botocore==1.0.0a1->awscli)
Downloading jmespath-0.7.1-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 from https://pypi.python.org/packages/any/p/python-dateutil/python_dateutil-2.4.2-py2.py3-none-any.whl#md5=c817289a1d95a355cc088e2292791a77 (from botocore==1.0.0a1->awscli)
Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)
100% |################################| 192kB 1.2MB/s
Collecting six<2.0.0,>=1.8.0 from https://pypi.python.org/packages/3.3/s/six/six-1.9.0-py2.py3-none-any.whl#md5=9ac7e129a80f72d6fc1f0216f6e9627b (from bcdoc<0.15.0,>=0.14.0->awscli)
Downloading six-1.9.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.1.4,>=3.1.2->awscli)
Downloading pyasn1-0.1.7.tar.gz (68kB)
100% |################################| 69kB 5.5MB/s
Installing collected packages: pyasn1, six, python-dateutil, jmespath, rsa, docutils, colorama, bcdoc, botocore, awscli
Running setup.py install for pyasn1
Found existing installation: six 1.8.0
Uninstalling six-1.8.0:
Successfully uninstalled six-1.8.0

Found existing installation: python-dateutil 2.1
Uninstalling python-dateutil-2.1:
Successfully uninstalled python-dateutil-2.1

Found existing installation: jmespath 0.6.1
Uninstalling jmespath-0.6.1:
Successfully uninstalled jmespath-0.6.1

Found existing installation: rsa 3.1.2
Uninstalling rsa-3.1.2:
Successfully uninstalled rsa-3.1.2
Running setup.py install for rsa
Installing pyrsa-encrypt-bigfile script to /usr/local/bin
Installing pyrsa-encrypt script to /usr/local/bin
Installing pyrsa-verify script to /usr/local/bin
Installing pyrsa-sign script to /usr/local/bin
Installing pyrsa-priv2pub script to /usr/local/bin
Installing pyrsa-decrypt script to /usr/local/bin
Installing pyrsa-decrypt-bigfile script to /usr/local/bin
Installing pyrsa-keygen script to /usr/local/bin
Found existing installation: docutils 0.11
DEPRECATION: Uninstalling a distutils installed project (docutils) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling docutils-0.11:
Successfully uninstalled docutils-0.11
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755
changing mode of build/scripts-2.7/rst2latex.py from 644 to 755
changing mode of build/scripts-2.7/rst2xetex.py from 644 to 755
changing mode of build/scripts-2.7/rst2man.py from 644 to 755
changing mode of build/scripts-2.7/rst2xml.py from 644 to 755
changing mode of build/scripts-2.7/rst2pseudoxml.py from 644 to 755
changing mode of build/scripts-2.7/rstpep2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2odt.py from 644 to 755
changing mode of build/scripts-2.7/rst2odt_prepstyles.py from 644 to 755
changing mode of /usr/local/bin/rst2man.py to 755
changing mode of /usr/local/bin/rst2s5.py to 755
changing mode of /usr/local/bin/rst2html.py to 755
changing mode of /usr/local/bin/rstpep2html.py to 755
changing mode of /usr/local/bin/rst2odt_prepstyles.py to 755
changing mode of /usr/local/bin/rst2pseudoxml.py to 755
changing mode of /usr/local/bin/rst2latex.py to 755
changing mode of /usr/local/bin/rst2odt.py to 755
changing mode of /usr/local/bin/rst2xetex.py to 755
changing mode of /usr/local/bin/rst2xml.py to 755
Found existing installation: colorama 0.2.5
DEPRECATION: Uninstalling a distutils installed project (colorama) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling colorama-0.2.5:
Successfully uninstalled colorama-0.2.5
Running setup.py install for colorama
Found existing installation: bcdoc 0.13.0
Uninstalling bcdoc-0.13.0:
Successfully uninstalled bcdoc-0.13.0

Found existing installation: botocore 0.95.0
Uninstalling botocore-0.95.0:
Successfully uninstalled botocore-0.95.0
Running setup.py install for botocore
Running setup.py install for awscli
changing mode of build/scripts-2.7/aws from 644 to 755
changing mode of build/scripts-2.7/aws.cmd from 644 to 755
changing mode of build/scripts-2.7/aws_completer from 644 to 755
changing mode of build/scripts-2.7/aws_zsh_completer.sh from 644 to 755
changing mode of /usr/local/bin/aws to 755
changing mode of /usr/local/bin/aws_completer to 755
changing mode of /usr/local/bin/aws.cmd to 755
changing mode of /usr/local/bin/aws_zsh_completer.sh to 755
Successfully installed awscli-1.7.29 bcdoc-0.14.0 botocore-1.0.0a1 colorama-0.3.3 docutils-0.12 jmespath-0.7.1 pyasn1-0.1.7 python-dateutil-2.4.2 rsa-3.1.4 six-1.9.0
—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-104742018.

russellneufeld on 22 May 2015

If you're installing botocore directly, you'll need to either specify the specific version (botocore==1.0.0a1), or use the --pre option to pip because this is an alpha release of botocore 1.0.0.

However if you're upgrading the CLI you should be able to use the same commands you've been using, since we specifically call out the exact botocore version to use (https://github.com/aws/aws-cli/blob/develop/setup.py#L9)

jamesls on 22 May 2015

Hmm… I wonder if this is because I’m installing from a requirements.txt file. I ran pip install -U awscli and it worked as you said - it installed botocore 1.0.0a and awscli 1.7.29. However my bootstrap scripts actually have boto3, botocore, and awscli in a requirements.txt file, and I install that with pip install -U -r requirements.txt. I wonder if that’s the reason it’s failing.

On May 22, 2015, at 4:13 PM, James Saryerwinnie [email protected] wrote:

If you're installing botocore directly, you'll need to either specify the specific version (botocore==1.0.0a), or use the --pre option to pip because this is an alpha release of botocore 1.0.0.

However if you're upgrading the CLI you should be able to use the same commands you've been using, since we specifically call out the exact botocore version to use (https://github.com/aws/aws-cli/blob/develop/setup.py#L9 https://github.com/aws/aws-cli/blob/develop/setup.py#L9)

—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-104757295.

russellneufeld on 22 May 2015

@russellneufeld

What does your requirements.txt file look like? They may help us reproduce your issue.

kyleknap on 27 May 2015

I'm starting with ami-4a5b7022, which is an Ubuntu 14.04 LTS image for instance storage. I run "aptitude update" and "aptitude -y upgrade". Then "easy_install -U pip" to get the latest pip. Then I'm putting some python and puppet code on the machine. The puppet code ultimately calls pip with the -U option, and requirement.txt had a bunch of things in it, including
…
boto3
botocore
awscli
...

russellneufeld on 27 May 2015

…
boto3
botocore
awscli
...

On May 27, 2015, at 4:18 PM, Kyle Knapp [email protected] wrote:

@russellneufeld https://github.com/russellneufeld
What does your requirements.txt file look like? They may help us reproduce your issue.

—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-106060069.

russellneufeld on 27 May 2015

Or edit your requirements.txt to (note that you exclude botocore, you can usually rely on the CLI to be using the latest version of botocore):

boto3

Let me know if that helps.

kyleknap on 27 May 2015

👍3

I just installed with the --pre option as you suggested and it worked. Thanks for that!

On May 27, 2015, at 5:01 PM, Kyle Knapp [email protected] wrote:

Yeah that is strange. I was able to reproduce your issue. It looks like it is something on how pip figures out dependencies and the fact that there has not been a release of boto3 where we take on the alpha release version of botocore as a dependancy. To make your requirements.txt work, try adding the --pre argument to your pip install command.

Or edit your requirements.txt to (note that you exclude botocore, you can usually rely on the CLI to be using the latest version of botocore):

awscli
boto3
Let me know if that helps.

—
Reply to this email directly or view it on GitHub https://github.com/aws/aws-cli/issues/1339#issuecomment-106073219.