Botocore: Regression in 1.13.21: cannot assume role using EC2 instance metadata

[jbergknoff-rival]

Hi, we just started running into issues assuming roles in Jenkins, where the principal credentials come from EC2 instance metadata. This appears to be a regression in 1.13.21.

Here's a reproduction, running in an EC2 instance with permission to assume the role specified in the aws-config file.

$ docker run -it --rm python:3.7.5-alpine3.10 sh
/ # cat > aws-config
[profile role-to-assume]
role_arn = arn:aws:iam::...:role/...
credential_source = Ec2InstanceMetadata

/ # pip install awscli
...
Successfully installed PyYAML-5.1.2 awscli-1.16.285 botocore-1.13.21 colorama-0.4.1 docutils-0.15.2 jmespath-0.9.4 pyasn1-0.4.8 python-dateutil-2.8.0 rsa-3.4.2 s3transfer-0.2.1 six-1.13.0 urllib3-1.25.7

/ # AWS_PROFILE=role-to-assume AWS_CONFIG_FILE=aws-config aws sts get-caller-identity

Error when retrieving credentials from Ec2InstanceMetadata: No credentials found in credential_source referenced in profile role-to-assume

/ # pip install botocore==1.13.20
...
ERROR: awscli 1.16.285 has requirement botocore==1.13.21, but you'll have botocore 1.13.20 which is incompatible.
Installing collected packages: botocore
  Found existing installation: botocore 1.13.21
    Uninstalling botocore-1.13.21:
      Successfully uninstalled botocore-1.13.21
Successfully installed botocore-1.13.20

/ # AWS_PROFILE=role-to-assume AWS_CONFIG_FILE=aws-config aws sts get-caller-identity
{
    "UserId": "...",
    "Account": "...",
    "Arn": "arn:aws:sts::...:assumed-role/.../botocore-session-1574199910"
}

[crgwbr]

This just broke a bunch of CI pipelines for me. Reverting from aws-cli 1.16.285 to 1.16.284 seems to be a temp fix.

[vurt007]

same ... just broke our deployment pinned to 1.13.20 for now

[scottschreckengaust]

https://github.com/aws/aws-cli/issues/4682

[joguSD]

Thanks for bringing this to our attention. We'll be reverting this ASAP and cutting an additional release today.

[itwasntandy]

You are seeing this as you're running in Docker, which is adding an extra network hop

from the blog post today on metadata changes

With IMDSv2, the PUT response containing the secret token will, by default, not be able to travel outside the instance. This is accomplished by having the default Time To Live (TTL) on the low-level IP packets containing the secret token set to “1,”

You can work around this for now, by modifying the hop limit for instance metadata service on a per instance basis.

aws ec2 modify-instance-metadata-options --instance-id i-1234567898abcdef0 --http-put-response-hop-limit 3

from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#configuring-instance-metadata-options

[joguSD]

Botocore v1.13.22 has been released and reverts to the previous behavior fixing the regression. We are still working on a proper fix to support the new IMDS behavior.