Jx: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry

[mjhoffman65]

Summary

Build fails while attempting to pull images from Docker Hub.

Steps to reproduce the behavior

Run any build with a Dockerfile that has a base image from Docker Hub (or anywhere besides ECR) in an EKS 1.13 cluster with ECR enabled for classic jenkins.

Expected behavior

The docker build will pull the base image from Docker Hub without error when running the skaffold build command.

Actual behavior

Build fails when building the docker images with skaffold with an error stating docker-credential-ecr-login can only be used with ECR.

Building [stream-worker]...
time="2019-06-28T15:23:11Z" level=warning msg="Error processing base image (python:3.6) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fpython%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-06-28T15:23:11Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain. Dependencies may be incomplete."
Sending build context to Docker daemon 

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.71
Kubernetes cluster v1.13.7-eks-c57ff8
kubectl            v1.15.0
helm client        v2.10.0+g9ad53aa
helm server        v2.10.0+g9ad53aa
git                git version 2.17.1
Operating System   Ubuntu 18.04.2 LTS

Jenkins type

• [ ] Next Generation (Tekton + Prow)
• [X] Classic Jenkins
• [ ] Serverless Jenkins (JenkinsFileRunner + Prow)

Kubernetes cluster

EKS 1.13 created with eksctl and ECR.

Operating system / Environment

Ubuntu 18.04.2 LTS

[mjhoffman65]

I was able to fix this manually by updating the jenkins-docker-cfg secret to:

{
    "credHelpers": {
        "<YOUR ECR HERE>": "ecr-login"
    }
}

Would love to contribute the fix to this issue, but might need some help getting started

[mar-rih]

I was able to fix this manually by updating the jenkins-docker-cfg secret to:

{
    "credHelpers": {
        "<YOUR ECR HERE>": "ecr-login"
    }
}

Would love to contribute the fix to this issue, but might need some help getting started
if i need to change from ECR to Docker hub how can i do it

[deanesmith]

@deanesmith follow up

[ScottyVG]

if i need to change from ECR to Docker hub how can i do it

I also have the same question as @mar-rih

[mjhoffman65]

if i need to change from ECR to Docker hub how can i do it

I also have the same question as @mar-rih

For a private dockerhub registry? It should just work with public images

[thienlh]

I updated my jx/jenkins-docker-cfg/config.json to

{
    "credsStore": "ecr-login",
    "credHelpers": {
        "ecr.eu-central-1.amazonaws.com": "ecr-login"
    }
}

and the problem is solved. Hope that help.
jx version output

NAME               VERSION
jx                 2.0.702
jenkins x platform 2.0.1141
Kubernetes cluster v1.14.6-eks-5047ed
kubectl            v1.15.3
helm client        v2.14.3+g0e7f3b6
helm server        v2.14.3+g0e7f3b6
git                2.23.0
Operating System   Mac OS X 10.15 build 19A546d

[stepanselyuk]

I also updated my k8s secret, with content:

{
  "credsStore": "ecr-login",
  "credHelpers": {
        "ecr.us-west-2.amazonaws.com": "ecr-login"
  }
}

Even restarted Jenkins, but it did not help and I'm still getting error:

ONBUILD instructions: parsing ONBUILD instructions: processing base image (composer:1) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fcomposer%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-10T12:40:53Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"

jx version output

NAME               VERSION
jx                 2.0.705
jenkins x platform 2.0.1119
Kubernetes cluster v1.13.10-eks-5ac0f1
kubectl            v1.15.3
helm client        v2.14.3+g0e7f3b6
helm server        v2.14.3+g0e7f3b6
git                2.23.0
Operating System   Mac OS X 10.14.6 build 18G95

[stepanselyuk]

I updated config.json content as

{
  "credHelpers": {
        "ecr.us-west-2.amazonaws.com": "ecr-login"
  }
}

Removed "credsStore": "ecr-login",, and that error dissapered, but I got a new one:

time="2019-09-10T13:16:52Z" level=fatal msg="build failed: build failed: building [redlotus-corp/cdp-web]: build artifact: unable to stream build output: Error parsing reference: \"composer:1 AS build-env\" is not a valid repository/tag: invalid reference format"

[thienlh]

@stepanselyuk I also updated all of the Jenkins plugins (with a restart), except ones that can't be upgrade.

[stepanselyuk]

Last error seems caused by an older version of Docker installed on the nodes. I use amiFamily: Ubuntu1804 in eksctl config.

docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.9.4
 Git commit:   f5ec1e2
 Built:        Thu Jul 11 12:25:38 2019
 OS/Arch:      linux/amd64

Multi-stage builds were introduced in 17.06 https://blog.docker.com/2017/07/multi-stage-builds/

[stepanselyuk]

Updated docker version on the nodes and re-run build, next I've got an error related to push to ECR.
Fixed by using this configuration:

{
  "credHelpers": {
        "{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
  }
}

"credsStore": "ecr-login", not needed cause it make these credentials use by default to any registry.

If you need to use a private registry (e.g. private Docker Hub), you can use a configuration like this:

{
"auths": {
     "index.docker.io/v1": {"auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } 
  },
  "credHelpers": {
        "{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
  }
}

Where is XXXX.. is your token.

[stepanselyuk]

On Ubuntu Nodes you can update your Docker version by executing commands below (from Ubuntu user), and it's not recommended for production, cause you need to have more control what to do.

sudo apt-get remove -y docker docker-engine docker.io containerd runc
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

More information on this topic: https://docs.docker.com/install/linux/docker-ce/ubuntu/

It is possible that you will need to update Jenkins plugins again, but I'm not sure what caused that in my case, cause first I've upgraded EKS cluster to 1.14 version and then upgraded Docker on Nodes.

[dragoonis]

I have this precise problem for a customer and I'm unable to provide them the EKS-backed CI/CD solution until this is patched.

I've followed the above recommendations to change the config.json secret, which has not worked for me at all.

I have also destroyed my EKS cluster and rebuild it and re-installed JX, and I have the exact same outcome.

I have updated my jx CLI (brew upgrade jx) and also pulled down the latest cloud-environments repository and such when prompted during cluster and JX installations, so I'm confident this hasn't been patched yet. If there's some kind of canary build I can use/test with I'd be more than happy to give it a spin! 👍 :)

Is someone actively working on this fix? I'd like to be able to update my customer on an ETA to see if they want to wait for this EKS fix or come up with a workaround.

Thanks! and keep up the great work 👍

/cc @martin-patsov

[stepanselyuk]

@dragoonis for me this configuration works:

{
  "credHelpers": {
        "{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
  }
}

Please be sure you properly encoded this content to base64 and placed in jenkins-docker-cfg secret.

cat <<EOF | base64
{
  "credHelpers": {
        "{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
  }
}
EOF

What exactly error do you have in the logs?

[dragoonis]

@stepanselyuk I had not put in the ACCOUNT_ID part before. I will try that.

However, I still believe that when making requests to Docker Hub (anonymously ofcourse) to pull in official base images, that it will say docker-credential-ecr-login can only be used with Amazon Elastic Container Registry meaning it's trying to send ecr-login to Docker Hub.

I will get back to you with trying the ACCOUNT_ID

[stepanselyuk]

@dragoonis please be sure you don't use "credsStore": "ecr-login", cause that defaults all registries hosts to be used with ecr-login, and ecr-login can be used with ECR only as I see in the error message.

If you don't use ACCOUNT_ID, you won't be able to push images into ECR.

[dragoonis]

It's worked! partially..

TLDR; Docker pull (base image) from Docker Hub is now working, but only for 1 of 2 images. Docker push to ECR is working.

The one that works (company-api-nginx) manages to pull from Docker Hub and be built and pushed to ECR.
Second one (company-api-php-fpm) fails to pull from Docker Hub

build

 - company-api-nginx: Not found. Building
 - company-api-php-fpm: time="2019-09-23T20:35:36Z" level=warning msg="error checking cache, caching may not work as expected: getting hash for artifact company-api-php-fpm: getting dependencies for company-api-php-fpm: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (php:7-fpm-alpine) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-23T20:35:35Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"

push for nginx

Successfully tagged xxxxxxx.dkr.ecr.eu-west-2.amazonaws.com/company/company-api-nginx:0.0.0-SNAPSHOT-PR-628-3
The push refers to repository [xxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com/company/company-api-nginx]
e929d9605539: Preparing
72c96a498309: Preparing
a0a699786342: Preparing
8b5b6870672a: Preparing
03fb8dd8a4de: Preparing
0665fde51bac: Preparing
3e76d2df1790: Preparing
03901b4a2ea8: Preparing
0665fde51bac: Waiting
3e76d2df1790: Waiting
03901b4a2ea8: Waiting
72c96a498309: Pushed
e929d9605539: Pushed
a0a699786342: Pushed
3e76d2df1790: Layer already exists
03901b4a2ea8: Layer already exists
03fb8dd8a4de: Pushed
0665fde51bac: Pushed
8b5b6870672a: Pushed
0.0.0-SNAPSHOT-PR-628-3: digest: sha256:00a6f1813538ceb9e6296c8741d987e48d010b4b5940b05e80c96b53ceb2a36c size: 1986

no push for php-fpm as it fails at build stage

Building [company-api-php-fpm]...
Sending build context to Docker daemon 

time="2019-09-23T20:35:57Z" level=fatal msg="build failed: build failed: building [company-api-php-fpm]: build artifact: docker build: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/build?buildargs=null&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=docker%2Fphp-fpm%2FDockerfile&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=xxxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com%2Fcompany%2Fcompany-api-php-fpm%3A0.0.0-SNAPSHOT-PR-628-3&target=&ulimits=null&version=: creating docker context: getting relative tar paths: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (php:7-fpm-alpine) for ONBUILD triggers: getting remote config: getting image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-ecr-login: exit status 1; output: 2019-09-23T20:35:57Z [ERROR] Error parsing the serverURL: https://index.docker.io, error: docker-credential-ecr-login can only be used with Amazon Elastic Container Registry.\ncredentials not found in native keychain"

[stepanselyuk]

@dragoonis if it worked partially, I think it is possible that erroneous build was affected by old configuration. Please try again (the PR rebuild), and if all the same, I think better to check actual Docker config on each node. Maybe it worth also to restart Jenkins.

[lacatus]

https://aws.amazon.com/es/blogs/compute/authenticating-amazon-ecr-repositories-for-docker-cli-with-credential-helper/

We switched to

{ "credHelpers": { "<YOUR ECR HERE>": "ecr-login" } }

and worked like charm, guessing that the node expires the token after several days...

[yuwenhui]

I was fixed this problem by:

1. View the old jenkins-docker-cfg secret:

kubectl get secret jenkins-docker-cfg -o yaml

I got the result as follow:

apiVersion: v1
data:
  config.json: <Secret DATA>
kind: Secret
metadata:
...

Then base64 encode the <Secret DATA> use echo '<Secret DATA> '' | base64 --decode, and i saw the secret result was:

{
  "credsStore": "ecr-login"
}

1. Delete the old jenkins-docker-cfgby using kubectl delete secret jenkins-docker-cfg;
2. Create config.json with:

{
  "credHelpers": {
        "{ACCOUNT_ID}.dkr.ecr.{ECR_REGION}.amazonaws.com": "ecr-login"
  }
}

1. Recreate the jenkins-docker-cfg secret by using:

kubectl create secret generic jenkins-docker-cfg  --from-file=./config.json

I using AWS EKS and jenkins-x, with jx version:

NAME               VERSION
jx                 2.0.861
jenkins x platform 2.0.1441
Kubernetes cluster v1.14.6-eks-5047ed
kubectl            v1.12.7
helm client        Client: v2.14.1+g5270352
git                2.14.5
Operating System   Unknown Linux distribution Linux version 4.14.114-82.97.amzn1.x86_64 (mockbuild@gobi-build-64003) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Sun Apr 28 07:27:43 UTC 2019

[miguelaustro]

thanks @yuwenhui

Another version, In two steps :)

$ ERC=$(echo '{ "credsStore": {"999999999999.dkr.ecr.ap-southeast-2.amazonaws.com": "ecr-login"} }'| base64 -w 0)
$ kubectl patch secret jenkins-docker-cfg -p="{\"data\": { \"config.json\": \"$ERC\"}}"

[dragoonis]

I'm glad this is now patched.