Aws-cli: Docker login fails returning "The stub received bad data" on Windows when using awscli v2.

[overbit]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone though the User Guide and the API reference
• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug
Authenticate Docker to an Amazon ECR registry with get-login-password fails with the following error:
Error saving credentials: error storing credentials - err: exit status 1, out: 'The stub received bad data.'

SDK version number
aws-cli/2.0.56 Python/3.7.7 Windows/10 exe/AMD64

Platform/OS/Hardware/Device
Windows 10 1909
Docker Desktop 2.4.0

To Reproduce (observed behavior)

1. Have an ECR repository setup
2. Authenticate docker to pull and push images from the repository using the authorization token:
   aws ecr get-login-password --region <specific-region> | docker login --username AWS --password-stdin <myaccount>.dkr.ecr. <specific-region>.amazonaws.com

Expected behavior
Login should be successful

Additional context
I'm currently using an aws profile configured with AWS SSO

aws ecr get-login-password --region <specific-region> command alone succeed returning a token.
This token might be too big for docker login to accept?

[paulriley]

I ran into the same problem. Reverting to Docker Desktop 2.3.0.5 made it go away, so I think this must be a new problem with 2.4.0.0.

[kdaily]

Thanks for the report. A quick look around shows a StackOverflow post and the same error with the Azure CLI with this scenario, and a (potentially insecure) solution:

• https://github.com/Azure/azure-cli/issues/4843
• https://stackoverflow.com/questions/60807697/get-ecrlogincommand-and-docker-login-error-storing-credentials-the-stub-receiv

Looking at the post from the Azure CLI, it was addressed on the server side by limiting the token length. I'll pass this information on to the ECR team for review, as I don't think it's something that the CLI can address directly.

[maurera]

I have the same issue, also using AWS SSO

Workaround attempts:

• I tried uninstalling Docker Desktop 2.4.0.0 and installing version 2.3.0.5, as suggested by @paulriley . This was unsuccessful.
• I tried the answer to the stackoverflow question posted by @kdaily (modifying .dockerconfig.json and separately, deleting docker-credential-wincred.exe and docker-credential-desktop.exe). This was unsuccessful.

As it stands, I'm still unable to push to ecr

[maurera]

I just had a call with AWS support. As far as I understand it, when you run aws ecr get-login, you're requesting a string authentication token from AWS (IAM under the hood). There's no limit on the length of this string, but it's typically shorter than 2500 characters.

When you run docker login... it uses Windows Credentials Manager, which doesn't support secrets of greater than 2,500 characters (see AWS support thread here: https://forums.aws.amazon.com/thread.jspa?threadID=324928). The AWS ECR guys blame AWS IAM and AWS IAM blames Windows.

With that said, we found a workaround, by using amazon-ecr-credential-manager, which seems to override Windows Credentials Manager under the hood. Follow the instructions here https://github.com/awslabs/amazon-ecr-credential-helper. This has the added benefit of automating the aws ecr get-login step, so you can go straight from you docker tag step to docker push step.

[overbit]

@maurera Thanks for the help. Unfortunately, it doesn't seem that _amazon-ecr-credential-manager_ supports AWS SSO at the moment.
https://github.com/awslabs/amazon-ecr-credential-helper/issues/229

[kdaily]

Hi @maurera, thanks for passing along that information. I'm going to mark this as a potential update for our documentation troubleshooting section.

Sorry that the solution provided doesn't work for you, @overbit. You can add in a comment on the issue you linked.

[maurera]

FYI - the ECR Credential helper initially helped in my case, but now I get the following error after running docker push:

denied: Your authorization token has expired. Reauthenticate and try again

[kdaily]

Hi @maurera, sorry to hear that. This seems like an issue with the credential helper, so I would suggest following up in their repository.

[maurera]

Note - I tried two workarounds.

First (didn't work). I launched a WSL2 Ubuntu Linux instance and tried aws ecr get-login-password from there. This didn't work (it seems like it's still connecting with the docker engine running in Windows, which connects to Windows Credential Manager.)

Second (works). I spun up an EC2 instance, copied over my project, and docker pushd from there.

This seems like a major issue. People who develop in Windows can't push Docker containers to AWS right now.

[paulriley]

@maurera I had the same issue with WSL. Fixed it by deleting the docker.config from Windows and restarting everything. docker-login started using the fallback of storing creds in the docker.config on WSL. Go figure.

I did, at one point, have docker push working in WSL with the ECR credentials plugin. Then it started failing again. I haven't had the time to really hammer out the problem, but I will. In the meantime, I hope that helps you get the work-around working in WSL.

[kdaily]

Hi @maurera and @paulriley, thanks for the further details and feedback on the impact of this issue. I agree that it is a major issue impacting Windows developers. I have let the ECR team know about it. I would recommend opening (or re-opening) any issue you have with AWS Support to escalate it further.

The get-login-password is a wrapper around the boto3 ECR client get_authorization_token command, which is a direct call to the ECR API GetAuthorizationToken action:

• https://github.com/aws/aws-cli/blob/v2/awscli/customizations/ecr.py#L40
• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecr.html#ECR.Client.get_authorization_token
• https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_GetAuthorizationToken.html

The length of the token you get back is dependent on the type of IAM principal account. Unfortunately there isn't anything that the AWS CLI can do to change this behavior.

[overbit]

@kdaily is it worth to open a new issue about the length of the token received by ECR API? If yes, what will be the best repository to open the issue against?

[dougrday]

This seems to only happen with accounts that have AWS SSO enabled.

The workaround we've found from StackOverflow is to open the %userprofile%/.docker/config.json file, and remove the credStore and credsStore properties.

The problem is it's only temporary. Each day the engineers need to run aws sso login, and each day they need to open the above file and remove those values before calling aws ecr get-login-password | docker login --username AWS --password-stdin <registry>

I can confirm that aws ecr get-login-password returns a string greater than 2,500 characters when AWS SSO is enabled.

[paulriley]

SSO, maybe. But not specifically AWS.

We're using Azure SSO to access AWS and seeing the same problem.

It's possible the long keys are for accounts that have assumed roles. But, as I see it, the key length is not the problem, it's wincred's inability to handle such keys.

[dougrday]

I've created an issue over at Docker to indicate the bug with the Windows credential helpers.
https://github.com/docker/docker-credential-helpers/issues/190

[samuel-jabamani]

Issue :

Error saving credentials: error storing credentials - err: exit status 1, out: The stub received bad data.
exit status 1

Analysis:

saml2aws exec -a devstage "aws ecr get-login-password --region eu-central-1"

The above command returns a big password. Wincred seems to work with passwords of size 2500 characters or less.But the above command returns password with 2500 characters or more, in my case it was 2580 characters.So AWS ECR login fails.The docker desktop version I have in my windows10 is 2.5.0.0 .To solve this problem we have to do two things.

Solution :

1) Rename docker-credential-wincred.exe in C:Program FilesDockerDockerresourcesbin to say docker-credential-wincred123.exe
2) Go to C:Users\\.dockerconfig.json and remove the "credsStore": "wincred" line and save it.Dont forget to remove the , at the previous line too.

  Note : config.json will be created after the first successful login.So if you don't find config.json in C:\Users\\<user.name>\\.docker, do a normal docker login with your docker account and password and the file will be created. Then go and do the point no 2 above and logon to ECR.

Now we will be able to logon to AWS ECR successfully in a command prompt with a warning.

Hope this helps !!.

[dougrday]

I've created a workaround project that avoids the Windows Credential Manager, as it's completely blocking for us.

Use with caution

https://github.com/dougrday/docker-credential-plaintext

In our case, this workaround is fine as we're using SSO, and our tokens are short-lived (4 hours). If Docker Desktop resolves the issue with Windows Credential Manager, we'll stop using the workaround.

[mklinke]

FWIW, I (re)moved the ~/.docker/config.json and then the login succeeds as described in the docs. I'm running this in WSL2.

aws-cli/2.1.3 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.20

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Oct 14 19:43:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0-rc1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       131bf7e
  Built:            Tue Nov 17 22:52:57 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.4.1
  GitCommit:        c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0