Aws-sdk-js: Assume IAM Role for a Service Account not working

[marcincuber]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone though Developer Guide and API reference
• [x] I've checked AWS Forums and StackOverflow for answers
• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug
Application has been upgraded to the latest version of aws-sdk and temporary credentials are not available when creating e.g. const secretsManagerClient = new AWS.SecretsManager(awsConfig.secretsManagerConfig). I am using latest version in order be able to use token_file_web_identity_credentials.

Application is running in kubernetes 1.14.

There is an open issue in the mentioned application. external-secrets-issue

SDK version number
v2.528.0

Observed behavior
Node application is correctly receiving desired variables i.e. AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, however assumed role credentials are not available.

So, credentials that are being used are taken from the node's instance profile whereas they should be available after assuming AWS_ROLE_ARN, log:
"level":50,"time":1568621518210,"pid":18,"hostname":"external-secrets-7595968467-2cfb2","message":"User: arn:aws:sts::01111111111:assumed-role/ceng-eks-test-worker-node/i-01f479d9225b21219 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:eu-west-1:01111111111:secret:newrelic-keys-f9UfVW","code":"AccessDeniedException","time":"2019-09-16T08:11:58.210Z","requestId":"d6ab153d-96c3-48c4-bc11-e9c89d68b07c","statusCode":400,"retryable":false,"retryDelay":69.02564110661021,"msg":"failure while polling the secret newrelic-account-key","stack":"AccessDeniedException: User: arn:aws:sts::01111111111:assumed-role/ceng-eks-test-worker-node/i-01f479d9225b21219 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:eu-west-1:01111111111:secret:newrelic-keys-f9UfVW\n at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/json.js:51:27)\n at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)\n at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /app/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)\n at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)","type":"Error","v":1}

Expected behavior
A clear and concise description of what you expected to happen.
I expect that if variables AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and AWS_REGION, credentials will be assume and then automatically passed into e.g. AWS.SecretsManager.

Based on the description in token_file_web_identity_credentials.js
I wouldn't need to do anything extra in terms of setup.

Any chances to point me in the right direction or provide some help? Thanks

[trivikr]

Hi @marcincuber,

We'd done extensive testing internally before merging TokenFileWebIdentityCredentials

You can view CredentialProviderChain at https://github.com/aws/aws-sdk-js/blob/5d8a5ead33431b1303cf63bf01db26c52c28ce69/lib/node_loader.js#L61-L69

Can you confirm that none of the other Credential Providers taking precedence over TokenFileWebIdentityCredentials?

[farshad-hobsons]

Hi @trivikr,

We tested this by building kubernetes-external-secrets release 1.3.1 with latest aws-sdk-js (2.543.0) and we encountered the following error:

{"level":50,"time":1570455016268,"pid":16,"hostname":"um-kubernetes-external-secret-kubernetes-external-secrets-2lq5t","type":"Error","stack":"CredentialsError: Missing credentials in config\n at IncomingMessage.<anonymous> (/app/node_modules/aws-sdk/lib/util.js:895:34)\n at IncomingMessage.emit (events.js:194:15)\n at IncomingMessage.EventEmitter.emit (domain.js:441:20)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","message":"Missing credentials in config","retryable":false,"time":"2019-10-07T13:30:16.268Z","code":"CredentialsError","originalError":{"message":"Could not load credentials from any providers","retryable":false,"time":"2019-10-07T13:30:16.268Z","code":"CredentialsError"},"msg":"Missing credentials in config","v":1}

To reproduce, you can try the helm chart that we used: https://github.com/Hobsons/kubernetes-external-secrets/tree/v1-3-1-with-updates/charts/kubernetes-external-secrets with the following instructions:

1. Build https://github.com/Hobsons/kubernetes-external-secrets/blob/v1-3-1-with-updates/Dockerfile with aws-sdk-js version 2.543 and push to a cluster-accessable registry
2. Deploy helm chart with appropriate IRSA configs.
   helm --tiller-namespace ${NAMESPACE} --namespace ${NAMESPACE} upgrade um-kubernetes-external-secret --set serviceAccount.name=${EXISTING_IRSA_COMPATIBLE_SA} --set env.AWS_REGION=us-east-1 --set serviceAccount.create=false --set image.repository=${CLUSTER_IMAGE_REPO} --set image.tag=1.3.1-updates --set image.imagePullSecrets=${CLUSTER_IMAGE_SECRET} --version 1.0.1 --debug -i ./charts/kubernetes-external-secrets

Variable Definitions:
${NAMESPACE}=namespace in which you want to deploy
${EXISTING_IRSA_COMPATIBLE_SA}=an existing service account that has the annotation eks.amazonaws.com/role-arn: arn:aws:iam::YOURAWSACCOUNT:role/ROLE2ASSUME
CLUSTER_IMAGE_REPO= Docker registry that you pushed the built image to
CLUSTER_IMAGE_SECRET= k8s secret of type kubernetes.io/dockerconfigjson that can pull images from CLUSTER_IMAGE_REPO

On creating an object of type externalSecret we see the error above

We also tried to test with the latest version(1.5.0) of kubernetes-external-secrets. 1.5.0 has some assume role functionality based on a namespace annotations so we are more confident with our error reproduction with version 1.3.1. The error that we encountered with that version 1.3.1 was: {"level":50,"time":1570041009381,"pid":17,"hostname":"um-kubernetes-external-secret-kubernetes-external-secrets-klhlk","type":"Error","stack":"CredentialsError: Missing credentials in config\n at Inc omingMessage.<anonymous> (/app/node_modules/aws-sdk/lib/util.js:865:34)\n at IncomingMessage.emit (events.js:194:15)\n at IncomingMessage.EventEmitter.emit (domain.js:441:20)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","message":"Missing credentials in config","retryable":false,"time":"2019-10-02T18:30:09.378Z","code":" CredentialsError","originalError":{"message":"Could not load credentials from any providers","retryable":false,"time":"2019-10-02T18:30:09.378Z","code":"CredentialsError"},"msg":"failure while polling t he secret um-test-service","v":1}

Sidenote: kubernetes-external-secrets version 1.3.1 with kiam integration works without a hitch!

[trivikr]

@ajredniwja will look into the test case to repro and verify with documentation as part of triaging.

[ajredniwja]

@farshad-hobsons can you try providing the credentials explicitly using

const secretsManagerClient = new AWS.SecretsManager({credentials: new AWS.TokenFileWebIdentityCredentials()});

when initiating the SecretsManager client to make sure EKS credentials are working fine?

[farshad-hobsons]

Thank you @ajredniwja & @trivikr for providing help on this.

@ajredniwja , on adding explicit TokenFileWebIdentityCredentials, we encountered the following error:
{"level":50,"time":1570628658062,"pid":18,"hostname":"um-kubernetes-external-secret-kubernetes-external-secrets-4sks5","type":"Error","stack":"CredentialsError: Missing credentials in config\n at Object.openSync (fs.js:439:3)\n at Object.readFileSync (fs.js:344:35)\n at TokenFileWebIdentityCredentials.load (/app/node_modules/aws-sdk/lib/credentials/token_file_web_identity_credentials.js:163:28)\n at TokenFileWebIdentityCredentials.coalesceRefresh (/app/node_modules/aws-sdk/lib/credentials.js:205:12)\n at TokenFileWebIdentityCredentials.refresh (/app/node_modules/aws-sdk/lib/credentials/token_file_web_identity_credentials.js:119:10)\n at TokenFileWebIdentityCredentials.get (/app/node_modules/aws-sdk/lib/credentials.js:122:12)\n at getAsyncCredentials (/app/node_modules/aws-sdk/lib/config.js:369:24)\n at Config.getCredentials (/app/node_modules/aws-sdk/lib/config.js:389:9)\n at Request.VALIDATE_CREDENTIALS (/app/node_modules/aws-sdk/lib/event_listeners.js:81:26)\n at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:102:18)","message":"Missing credentials in config","errno":-13,"syscall":"open","code":"CredentialsError","path":"/var/run/secrets/eks.amazonaws.com/serviceaccount/token","time":"2019-10-09T13:44:18.062Z","originalError":{"message":"Could not load credentials from TokenFileWebIdentityCredentials","errno":-13,"syscall":"open","code":"CredentialsError","path":"/var/run/secrets/eks.amazonaws.com/serviceaccount/token","time":"2019-10-09T13:44:18.062Z","originalError":{"errno":-13,"syscall":"open","code":"EACCES","path":"/var/run/secrets/eks.amazonaws.com/serviceaccount/token","message":"EACCES: permission denied, open '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'"}},"msg":"Missing credentials in config","v":1}

The above error led us to think that there was some form of permission issue and we quickly discovered that we have this https://github.com/aws/amazon-eks-pod-identity-webhook/issues/8 issue. We then successfully tested the workaround in https://github.com/kubernetes-incubator/external-dns/pull/1185#issuecomment-530439786 to set securityContext.fsGroup=65534 and also successfully tested it by dropping explicit TokenFileWebIdentityCredentials.

My conclusion is that this is not an aws-sdk-js bug, rather it is an upstream kubernetes issue of hardcoding 0600 (even 1.16 kubernetes has that)

The implication is that: until upstream is fixed, non-root account in containers might need the workaround of securityContext.fsGroup=65534. I also suggest that if this detail is mentioned in Amazon EKS IRSA docs, then it would be benefical for end-users for awareness purposes.

[ajredniwja]

@farshad-hobsons, I can reach to the docs team for adding some details. Closing this issue now, please reach out if you have any additional questions.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.