Aws-sdk-java-v2: Java SDK does not support EKS IAM for service accounts

Created on 13 Oct 2019 · 16Comments · Source: aws/aws-sdk-java-v2

EKS IAM Service Account Role introduces a new environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and based on the documentation on these two pages, the Java SDK should use "AWS_WEB_IDENTITY_TOKEN_FILE" for credentials if exists.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

I have tried the latest Java SDK 2.9.19 and it doesn't seem to work. There is a similar discussion on DotNet SDK at here:

https://github.com/aws/aws-sdk-net/issues/1413

I couldn't find docs saying "AWS_WEB_IDENTITY_TOKEN_FILE" is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.

Expected Behavior

Based on the EKS doc link above, Java SDK should recognize the environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.

Current Behavior

I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run aws s3 ls and it worked, which means the token is correct.

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 467C66AAEAD354A9)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listBuckets(DefaultS3Client.java:2035)
        at com.example.s3.S3BucketOps.main(S3BucketOps.java:63)

Steps to Reproduce (for bugs)

The code I am using to test is from sample S3 code at here:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java

I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn't work.

Your Environment

AWS Java SDK version used: 2.9.19
JDK version used: 1.8
Operating System and version: Linux running from a container (pod on EKS)

bug

Source

starchx

👍5

Most helpful comment

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

starchx on 18 Oct 2019

👍7 ❤5

All 16 comments

Having the same issue using the AWS Java SDK version: 1.11.653.

JDK version used: 11
Operating System and version: Linux running from a container (pod on EKS)
Kubernetes version 1.14

According to the documentation this version of the SDK should also work fine. My application uses the role of the node instead of the one I have added through a service account.

endre-synnes on 17 Oct 2019

👍1

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

starchx on 18 Oct 2019

👍7 ❤5

@starchx Thank you! That solved the issue😄 Hope that the WebIdentityTokenFileCredentialsProvider will be added as a default soon!

endre-synnes on 21 Oct 2019

Hi all, this should be part of the default chain. We will prepare a change for this.

dagnir on 24 Oct 2019

👍5

@endre-synnes I am also experiencing the same problem despite the documentation saying it should work. We should file an issue at https://github.com/aws/aws-sdk-java/issues

thundergolfer on 29 Oct 2019

😄1

Fixed via #1501 and released as part of 2.10.11

zoewangg on 7 Nov 2019

Hi,
It still doesn't work for me with 2.10.57. The sdk uses node role instead of service account role.

UPDATE
It works. I just had to add

      securityContext:
        fsGroup: 65534

to the pod spec in order to make sure containers have access to the service token file.

thereisnospoon on 25 Feb 2020

Hi, It still doesn't work for me with 2.10.72.
It seems that java sdk doesn't get credentials from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN envs.
(FYI, I already added security context to pods to be able to access the token file)

imcheck on 19 Mar 2020

👍3

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

thereisnospoon on 19 Mar 2020

👍3 😕2

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sts</artifactId>
        </dependency>

It just works :)

jiang-wei on 20 Apr 2020

👍1

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

gargkshitiz on 20 Apr 2020

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

S3Client s3 = S3Client.builder()
                              .region(region)
                              .build();

and include sts with the latest version from maven repo, today it is 2.11.14

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sts</artifactId>
        </dependency>

jiang-wei on 20 Apr 2020

👍3

Thank you Jiang. This was conflicting with aws sdk jar I was using. Latest versions of sdk and sts helped me.

gargkshitiz on 20 Apr 2020

This is an interesting discussion and somewhat related to a use case I am trying to figure out. My service needs to access resources in a different AWS account from EKS, so I want to use profiles and assume role using a config like this:

[profile eks-account-role]
role_arn = <eks-account-role-arn>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile other-account-role]
role_arn = <other-account-role-arn>
source_profile = eks-account-role
role_session_name = x-account-session

Using the following code to access the STS client (for other purposes than assuming roles).

stsClient = StsClient.builder().build();

Then I set the environment variable AWS_PROFILE=other-account-role in the container.

But when I go to run it, the service is still running as the eks-account-role. I'm assuming the default credential provider chain will pick up the AWS profile defined in the config and use that method for authentication, but it doesn't seem to be happening. However, when I try this with the AWS CLI it works as expected (I have properly setup the other role to be assumed by the eks role).

I have found a workaround where if I specify to use profile credentials explicitly it works as expected.

stsClient = StsClient.builder()
                .credentialsProvider(ProfileCredentialsProvider.create())
                .build();

Now this isn't a big deal as it is a small amount of additional code, but I am curious what the SDK is doing under the hood and why it is different from the CLI. A couple things to consider, EKS seems to be injecting two environment variables into the container that could be effecting the behavior AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. However, I am running the CLI from within the container with the same variables present.

Thoughts?

campbel on 10 Jun 2020

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included
      <dependency>
          <groupId>software.amazon.awssdk</groupId>
          <artifactId>sts</artifactId>
      </dependency>
It just works :)

Surprised that this works, but it does. Any idea why the separate declaration of sts is required?

rod-dvla on 9 Dec 2020

👍2

I still get an error trying to connect to Secrets Manager from EKS using Service Account role.
I'm using sdk v2.16.52.

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 13e02bf6-082c-47a2-ace4-900b02792b72, Extended Request ID: null), ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.secretsmanager.DefaultSecretsManagerClient.getSecretValue(DefaultSecretsManagerClient.java:1098) at com.hlpp.csw.util.SecretManagerUtil.getSecretFromSM(SecretManagerUtil.java:51) at com.hlpp.csw.config.ShiroCustomEnvironment.setIniOverrides(ShiroCustomEnvironment.java:23) at com.hlpp.csw.config.ShiroCustomEnvironment.init(ShiroCustomEnvironment.java:15) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:45) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:40) at org.apache.shiro.web.env.EnvironmentLoader.createEnvironment(EnvironmentLoader.java:313) at org.apache.shiro.web.env.EnvironmentLoader.initEnvironment(EnvironmentLoader.java:139) at org.apache.shiro.web.env.EnvironmentLoaderListener.contextInitialized(EnvironmentLoaderListener.java:58) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:692) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1023) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:824) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1611) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:829) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:433) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)