Aws-cli: CP Command via pip throwing KeyError on basic copy

[mlevyVHC]

Version 1.16.96

Running copy command from local file to S3 location similar to:
aws s3 cp deployment/file.zip "s3://bucket/file.zip"

Downgraded to 1.16.93 (picked random version from last few days) and it works fine. Debug command gave no additional output.

Outputting following error (I am not using anything related to sms-voice):
```Traceback (most recent call last):
File "/usr/local/bin/aws", line 27, in
sys.exit(main())
File "/usr/local/bin/aws", line 23, in main
return awscli.clidriver.main()
File "/usr/local/lib/python3.7/site-packages/awscli/clidriver.py", line 59, in main
rc = driver.main()
File "/usr/local/lib/python3.7/site-packages/awscli/clidriver.py", line 193, in main
command_table = self._get_command_table()
File "/usr/local/lib/python3.7/site-packages/awscli/clidriver.py", line 102, in _get_command_table
self._command_table = self._build_command_table()
File "/usr/local/lib/python3.7/site-packages/awscli/clidriver.py", line 122, in _build_command_table
command_object=self)
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 725, in emit
return self._events.emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(kwargs)
File "/usr/local/lib/python3.7/site-packages/awscli/customizations/sms_voice.py", line 21, in hide_sms_voice
command_table['sms-voice']._UNDOCUMENTED = True
KeyError: 'sms-voice'

[shawnjoseph]

related commit https://github.com/aws/aws-cli/commit/4537081328ed8ffa8c7dcfde711996450cf7754d

[klaytaybai]

@mlevyVHC , @shawnjoseph, @rsweis62, and @dvayns2 are you all experiencing this? I tried reproducing a couple times with the same version, but it worked just fine.

[LinaToquica]

I Downgraded to 1.16.93 and it works !

[klaytaybai]

We're still having trouble reproducing this. Can you please verify that your version is 1.16.96? Also please confirm that this is the exact output that you get when running aws s3 cp deployment/file.zip "s3://bucket/file.zip" --debug.

[klaytaybai]

Also, are you having issues with all or any other commands with this version?

[shawnjoseph]

From today using 1.16.96. Seems to be more than one command affected.

Requirement already satisfied: awscli in /var/lang/lib/python3.6/site-packages (1.16.96)
...
...
10:14:59 + aws --profile stage --region eu-west-1 lambda get-function --function-name stage-anx
10:15:00 Traceback (most recent call last):
10:15:00   File "/var/lang/bin/aws", line 27, in <module>
10:15:00     sys.exit(main())
10:15:00   File "/var/lang/bin/aws", line 23, in main
10:15:00     return awscli.clidriver.main()
10:15:00   File "/var/lang/lib/python3.6/site-packages/awscli/clidriver.py", line 59, in main
10:15:00     rc = driver.main()
10:15:00   File "/var/lang/lib/python3.6/site-packages/awscli/clidriver.py", line 193, in main
10:15:00     command_table = self._get_command_table()
10:15:00   File "/var/lang/lib/python3.6/site-packages/awscli/clidriver.py", line 102, in _get_command_table
10:15:00     self._command_table = self._build_command_table()
10:15:00   File "/var/lang/lib/python3.6/site-packages/awscli/clidriver.py", line 122, in _build_command_table
10:15:00     command_object=self)
10:15:00   File "/var/lang/lib/python3.6/site-packages/botocore/session.py", line 725, in emit
10:15:00     return self._events.emit(event_name, **kwargs)
10:15:00   File "/var/lang/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
10:15:00     return self._emitter.emit(aliased_event_name, **kwargs)
10:15:00   File "/var/lang/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
10:15:00     return self._emit(event_name, kwargs)
10:15:00   File "/var/lang/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
10:15:00     response = handler(**kwargs)
10:15:00   File "/var/lang/lib/python3.6/site-packages/awscli/customizations/sms_voice.py", line 21, in hide_sms_voice
10:15:00     command_table['sms-voice']._UNDOCUMENTED = True
10:15:00 KeyError: 'sms-voice'

[klaytaybai]

What is the output if you run aws --version? It would be good to ensure your botocore version is updated and compatible.

[klaytaybai]

Please try reinstalling the AWS CLI. You may need to instead use the bundled installer or use the CLI in a virtual environment with the correct dependencies because you are most likely having issues with botocore version compatibility.

[stealthycoin]

This is because your botocore version is not correct. The CLI takes an == dependency on a botocore version. This means if the botocore version is not EXACTLY right we do not guarantee it work work at all.

Version 1.16.96 of the CLI where this was released takes a dependency on botocore==1.12.86:
https://github.com/aws/aws-cli/blob/b3f9ade91c725935e3bcd7fd7bab81362ecca5a7/setup.py#L26

Somehow you have modified the version of botocore you have installed along side the CLI to not have sms-voice`. If you check the correct version of botocore, its there: https://github.com/boto/botocore/tree/48a9075edbc964aae1e9a04439714742aba37bb6/botocore/data/sms-voice/2018-09-05

This cannot happen if you use the MSI installer/Bundled installer for the CLI as they create isolated environments to put the dependencies in. This is typically only an issue if you install the CLI using pip manually into the global environment, and then also install other things into that environment that take conflicting dependencies on botocore.

I would be curious to see if anyone who is affected by this problem can provide me the output of pip freeze so that we can look at the dependency list and see what is commonly causing this issue.

To be clear downgrading here isnot an acutal solution. All that is doing is getting pip to install the correct version of botocore over what you had before. Upgrading again should work as well since it would be installing the right botocore version. Or you could manually install the correct version of botocore directly. This would likely break whatever else in your environment is installing the old version of botocore in the first place. The real solution is to separate your environments and python dependencies so they cannot conflict like this.

[jkieberking]

@stealthycoin we are getting the same error

alembic==1.0.0
asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==18.2.0
awscli==1.16.96
boto3==1.9.44
botocore==1.12.44
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
Click==7.0
colorama==0.3.9
coverage==4.5.1
cryptography==2.3.1
decorator==4.3.0
docutils==0.14
Flask==1.0.2
Flask-Migrate==2.2.1
Flask-SQLAlchemy==2.3.2
flex==6.13.2
genson==1.0.1
gevent==1.3.7
greenlet==0.4.15
idna==2.7
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
jsonpath-ng==1.4.3
jsonpointer==1.14
jsonschema==2.6.0
locustio==0.9.0
Mako==1.0.7
MarkupSafe==1.0
mimesis==2.1.0
monolog-python==0.1.0
more-itertools==4.3.0
msgpack==0.5.6
mysql-connector==2.1.6
parse==1.9.0
parse-type==0.4.2
pipenv==2018.11.26
pluggy==0.8.0
ply==3.11
py==1.7.0
pyasn1==0.4.5
pycparser==2.19
Pygments==2.2.0
PyMySQL==0.9.2
PyNaCl==1.3.0
pytest==4.0.0
python-dateutil==2.7.3
python-editor==1.0.3
pytz==2018.5
PyYAML==3.13
pyzmq==17.1.2
requests==2.19.1
RESTinstance==1.0.0rc2
rfc3987==1.3.8
robotframework==3.0.4
rsa==3.4.2
s3transfer==0.1.13
six==1.11.0
SQLAlchemy==1.2.12
strict-rfc3339==0.7
tzlocal==1.5.1
unittest-data-provider==1.0.1
urllib3==1.23
validate-email==1.3
virtualenv==16.3.0
virtualenv-clone==0.5.1
Werkzeug==0.14.1

running aws --version:

Traceback (most recent call last):
  File "/usr/local/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 59, in main
    rc = driver.main()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 193, in main
    command_table = self._get_command_table()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 102, in _get_command_table
    self._command_table = self._build_command_table()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 122, in _build_command_table
    command_object=self)
  File "/home/circleci/.local/lib/python3.6/site-packages/botocore/session.py", line 645, in emit
    return self._events.emit(event_name, **kwargs)
  File "/home/circleci/.local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/circleci/.local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/circleci/.local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/awscli/customizations/sms_voice.py", line 21, in hide_sms_voice
    command_table['sms-voice']._UNDOCUMENTED = True
KeyError: 'sms-voice'

[klaytaybai]

Right @jkieberking. I suppose it was a bad idea to ask for aws --version if you can't initialize it in the first place. That being said, this was the result we expected. Your botocore version is outdated at 1.12.44. It should be 1.12.86 to match the CLI. Please consider following @stealthycoin's advice to get both up to date.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.