Aws-cli: Using aws-cli in AKS fails during IMDS request

[tna0y]

Confirm by changing [ ] to [x] below:

• [x] I've gone though the User Guide and the API reference
• [x] I've searched for previous similar issues and didn't find any solution

Issue is about usage on:

• [ ] Service API : I want to do X using Y service, what should I do?
• [ ] CLI : passing arguments or cli configurations.
• [x] Other/Not sure.

Platform/OS/Hardware/Device
What are you running the cli on?
Docker(base image nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04) / Azure AKS NV6 instance
Kubernetes version: 1.15.10
aws-cli/2.0.16 Python/3.7.3 Linux/4.9.184-linuxkit botocore/2.0.0dev20

Describe the question
At the startup of my container, I want to fetch some static data by copying bucket's contents into a local directory.

It seems from debug output that aws-cli detects that it is running in EKS and goes to fetch a metadata from IMDS (there is an http server on 169.254.169.254:80, see https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service). However, api is different and the request ends up with 400 bad request. I guess it actually is some kind of collision.

So, is there a way to go over this without some hacks like adding NET_ADMIN capability to the container and messing with ip routing?

Logs/output
Get full traceback and error logs by adding --debug to the command.

cli is installed in Dockerfile via:

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
    unzip awscliv2.zip && \
    ./aws/install && \
    rm -f awscliv2.zip && \
    aws --version

Command itself
```

• aws --debug s3 cp --recursive s3://bucket /tmp/tmp.VnkPuxUFd4
  2020-05-23 22:37:17,093 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.0.16 Python/3.7.3 Linux/4.15.0-1071-azure botocore/2.0.0dev20
  2020-05-23 22:37:17,094 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 's3', 'cp', '--recursive', 's3://bucket', '/tmp/tmp.VnkPuxUFd4']
  2020-05-23 22:37:17,094 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,094 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,094 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,095 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,097 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,097 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
  2020-05-23 22:37:17,097 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler
  2020-05-23 22:37:17,097 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler
  2020-05-23 22:37:17,098 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler
  2020-05-23 22:37:17,100 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler
  2020-05-23 22:37:17,100 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler
  2020-05-23 22:37:17,100 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,100 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler
  2020-05-23 22:37:17,100 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler
  2020-05-23 22:37:17,101 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler
  2020-05-23 22:37:17,102 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.copy-props: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler
  2020-05-23 22:37:17,103 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler
  2020-05-23 22:37:17,105 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
  2020-05-23 22:37:17,107 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 400 11
  2020-05-23 22:37:17,108 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
  Traceback (most recent call last):
  File "awscli/clidriver.py", line 335, in main
  File "awscli/customizations/commands.py", line 197, in __call__
  File "awscli/customizations/commands.py", line 190, in __call__
  File "awscli/customizations/s3/subcommands.py", line 709, in _run_main
  File "awscli/customizations/s3/subcommands.py", line 480, in _run_main
  File "awscli/customizations/s3/subcommands.py", line 473, in get_client
  File "botocore/session.py", line 775, in create_client
  File "botocore/session.py", line 824, in _resolve_region_name
  File "botocore/session.py", line 234, in get_config_variable
  File "botocore/configprovider.py", line 281, in get_config_variable
  File "botocore/configprovider.py", line 378, in provide
  File "awscli/utils.py", line 59, in provide
  File "awscli/utils.py", line 64, in _get_instance_metadata_region
  File "awscli/utils.py", line 104, in retrieve_region
  File "awscli/utils.py", line 113, in _get_region
  File "botocore/utils.py", line 227, in _fetch_metadata_token
  botocore.utils.BadIMDSRequestError:
  2020-05-23 22:37:17,108 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

```

[we-taper]

Having experienced the same bug as well.

[tna0y]

@we-taper As a temporary solution, the following command can be executed on the host prior to using aws-cli:

# ip route add blackhole 169.254.169.254

And after you are done don't forget to put everyting back in place by:

# ip route del blackhole 169.254.169.254

Alternatively, you can manually set AWS_REGION environment variable to your preferred region like this:

export AWS_REGION="eu-west-1"

Or via cli option:

aws --region eu-west-1 whatever

Still an issue since cli does not provide problem description and requires manual tuning.

[KaibaLopez]

Hi @tna0y.
sorry for the late response, it sounds like this was an issue regarding IMDS v2 incompatibility with CLI's region discovery.
This has been patched very recently (like this week), would you mind updating to the latest CLI version and retrying?

[we-taper]

I found that it didn't work for my build :P

[theory]

Thanks for that find, @tna0y. I had the same issue come up, as you can see here, only identified after I added --debug and was able to search for botocore.utils.BadIMDSRequestError and find this issue. Running 2.0.24 in the amazon/aws-cli docker image.

[bsc8180]

I have the same issue on an Azure VM.

aws --version aws-cli/2.0.40 Python/3.7.3 Linux/5.3.0-1035-azure exe/x86_64.ubuntu.18
uname -a Linux lin00004V 5.3.0-1035-azure #36-Ubuntu SMP Thu Aug 6 09:21:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Performing either of the workarounds above (set AWS_REGION or ip route) works.

Works fine under git-bash on Win10 with no requirement to implement either of these work arounds.

aws-cli/2.0.43 Python/3.7.7 Windows/10 exe/AMD64

[kdaily]

Thanks for your input, all!

As noted above, this is due to a change in behavior between the V1 and V2 clients. As of AWS CLI v2 2.0.15, if a region is not specified, the IMDS service is used to try and determine it. Setting a region prevents this request from happening.

https://github.com/aws/aws-cli/pull/5207

I think that @tna0y is correct in that it's using Azure's metadata service, which is not desirable!

There is also an undocumented environment variable that can be set to disable this:

AWS_EC2_METADATA_DISABLED=true

I will open a separate documentation issue to make sure this is added to the list of AWS CLI environment variables.

[kdaily]

Opened the following documentation issue: https://github.com/aws/aws-cli/issues/5623

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.