Amazon-vpc-cni-k8s: Configure aws sdk to use Region specific AWS STS service endpoints.

Created on 8 Oct 2019 · 4Comments · Source: aws/amazon-vpc-cni-k8s

Issue Description

When worker nodes are launched within private subnets with no access to the internet.
VPC is configured with the VPC private endpoints for services EC2, ECR, S3 and STS.
Amazon VPC CNI Plugin for K8S is configured with the service account and mapped to IAM Role for restricted access.
In this case, while aws-node pod calling AWS EC2 API (DescribeNetworkInterfaces) fails since it can not reach sts.amazonaws.com global endpoint for AWS STS service to get the token to authenticate the API call.

[ERROR] Failed to get ENI eni-xxxxxxxx information from EC2 control plane WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: dial tcp 52.94.241.129:443: i/o timeout
2019-10-08T10:47:49.502Z [WARN] Error trying to set up ENI eni-xxxxxxxx: failed to retrieve ENI eni-xxxxxxxx IP addresses: failed to find ENI addresses for ENI eni-xxxxxxxx: failed to describe network interface: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: dial tcp 52.94.241.129:443: i/o timeout

If AWS SDK can be configured to use the region specific endpoint for STS (example- sts.eu-west-1.amazonaws.com), it can resolve the endpoint with Private IP's from the VPC Private Interface endpoint or to the public IP's when VPCE is not configured.

Note

AWS STS Private Endpoint is not required when pod's are not configured with the fine grained permissions through K8S serviceaccount.

enhancement feature request prioritP1

Source

yeolahm

👍3

All 4 comments

aws-sdk-go has the changes required to get rid of this annoying behaviour merged in v1.25.18:
https://github.com/aws/aws-sdk-go/releases/tag/v1.25.18

New configurable option was introduced to regulate the STS endpoint resolution for legacy regions.

AWS_STS_REGIONAL_ENDPOINTS=regional for environment, or sts_regional_endpoints=regional in shared config file.

Looks like just a version bump...

maxstepanov on 25 Dec 2019

@jaypipes, we need to re-open this issue.

We still need a way to set AWS_STS_REGIONAL_ENDPOINTS=regional environment variable to make it regional. Also, just setting it to regional might not work in region is Hong Kong and Baharin.

For more info - documentation