Jx: skaffold build failed :Error creating credential store helper: Failed to create secondary file store dir /home/jenkins/.docker/acr

[zgorizzo69]

Summary

After upgraded the platform and jx my pipeline for the angulario jenkins javascript application example doesn't work anymore even after deleting it and redo a jx import. The error is
Error creating credential store helper: Failed to create secondary file store dir /home/jenkins/.docker/acr

time="2018-07-26T10:04:27Z" level=fatal msg="build step: building [changeme]: pushing: getting auth config for 10.0.66.232:5000/zgorizzo69/website-cogarius:0.0.33: getting auth config: error getting credentials - err: exit status 1, out: ``"

script returned exit code 1

Steps to reproduce the behavior

do a jx import on the angular.io jenkins-x kickstart application

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.117
jenkins x platform 0.0.1912
kubernetes cluster v1.9.9
kubectl            v1.10.5
helm client        v2.10.0-rc.1+gaa98e7e
helm server        v2.10.0-rc.1+gaa98e7e
git                git version 2.17.1

Kubernetes cluster

What kind of Kubernetes cluster are you using & how did you create it?
on azure (aks) I did a
jx create cluster aks --domain=blocklistener.com --tls-acme=true --http=false --nodes=3

Operating system / Environment

ubuntu 18.04

Expected behavior

I expect that this step execute correctly

 sh 'export VERSION=$PREVIEW_VERSION && skaffold build -f skaffold.yaml'

Actual behavior

the step export VERSION=cat VERSION && skaffold build -f skaffold.yaml failed

— Shell Script
Running shell script

++ cat VERSION

• export VERSION=0.0.33

• VERSION=0.0.33

• skaffold build -f skaffold.yaml

Starting build...

Building [changeme]...

Error creating credential store helper: Failed to create secondary file store dir /home/jenkins/.docker/acr

Sending build context to Docker daemon 533.8kB
...
Sending build context to Docker daemon 236.2MB

Step 1/5 : FROM abiosoft/caddy

---> 0a6415c06a4e

Step 2/5 : COPY . /srv

---> 56a2cd01185d

Step 3/5 : COPY Caddyfile /etc/Caddyfile

---> 525ee2899cc3

Step 4/5 : ENV PORT 80

---> Running in c3d10e8c91a4

---> 40d5c81552c4

Step 5/5 : EXPOSE 80

---> Running in 8245ce2ce0bf

---> 0b30e550ce1f

Successfully built 0b30e550ce1f

Error creating credential store helper: Failed to create secondary file store dir /home/jenkins/.docker/acr

time="2018-07-26T10:04:27Z" level=fatal msg="build step: building [changeme]: pushing: getting auth config for 10.0.66.232:5000/zgorizzo69/website-cogarius:0.0.33: getting auth config: error getting credentials - err: exit status 1, out: ``"

script returned exit code 1

[sapedevops]

Any updates or eta for its fix?

[ilyaa1]

This happens to me on AKS both when I use the built-in registry _and_ when I have a dedicated ACR registry provisioned beforehand (with all permissions in place).

I wonder if there's a step missing in setting up the storage on Azure.

[ilyaa1]

Any updates on this? Even with some hacking and overrides, I basically couldn't get JX to be usable end-to-end on AKS.

[zgorizzo69]

I have updated jx and platform ans till the same issue

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.175
jenkins x platform 0.0.2191
kubernetes cluster v1.9.9
kubectl            v1.10.5
helm client        v2.10.0+g9ad53aa
helm server        v2.10.0+g9ad53aa
git                git version 2.17.1

[bapetc]

Same issue here.
Any update?

[manishgoyaler]

Same issue here.
Any update?

[tgarlot]

Will have a look at this issue and try to fix it (having the same one...)

[tgarlot]

I was able to "make it work" by following the explanation on this page.

https://jenkins-x.io/architecture/docker-registry/

After having set the env variable DOCKER_REGISTRY and use jx create docker auth to set my ACR credentials (email is mandatory but can be anything), my buildpipeline (based on spring) completed successfully

Will still continue to have a look if this can somehow be automated during jx create cluster aks by adding few parameters for the credentials

[jmlambert78]

on AKS, I tried to make use of the internal registry, but I get difficulty to specify its URL in the DOCKER_REGISTRY key (in jenkins configuration)
While using only the value defined after JX install, it was blocking to :
Error creating credential store helper: Failed to create secondary file store dir /home/jenkins/.docker/acr
time="2018-09-25T15:13:16Z" level=fatal msg="build step: building [changeme]: pushing: getting auth config for 10.0.164.227:5000/jmlambert78/golang-htto-aks:0.0.1: getting auth config: error getting credentials - err: exit status 1, out: ``"

I tried to use the dockerhub url : index.docker.io but this seems not working properly.
I saw the need to have a "jx create docker auth" that I added but this seems not functional.
I am back to use the local registry deployed on AKS but I get a difficulty to specify the url :
I tried the following https url : but get the following msg:
Get https://10.0.164.227:5000/v1/_ping: http: server gave HTTP response to HTTPS client"
Then I changed to http :
time="2018-09-25T16:15:42Z" level=fatal msg="build step: building [changeme]: tagging: Error parsing reference: \"http://10.0.164.227:5000/jmlambert78/golang-htto-aks:0.0.15\" is not a valid repository/tag: invalid reference format"

Question : I do not see which syntax to really use for this URL and if it is necessary to have a docker auth token as well fo rthe local registry.
I may provide the logs of any if necessary.
BRs
@rawlingsj : I am back on jenkinsX after having used fabric8 some time ago...
The outcome is very nice,
BRs

[cnorburn]

Any progress on this?
I've got a fresh install on AKS, am facing the same issue.
Thanks

[jmlambert78]

In aim to solve, I deployed an ACR with a service principal and created the jx create docker auth token with the SP id & secret. After that it worked. But I still have the question on why I do not access correctly to the locally deployed registry pod.

[cnorburn]

Thanks @jmlambert78, you mentioned in your previous post that jx create docker auth didn't seem functional.
I seem to have the same problem
How did you manage to get this to work? Did you run it locally (your laptop) or via ssh on an AKS box?
Thanks

[jmlambert78]

In fact it works with ACR you need the Service Principal + Secret that you get after the creation of the SP.
YOu put the ID of the SP as username & secret as password
jx create docker auth --host "yourregistry.azurecr.io" --user "ffe9xxxx-ff52-xxxx-9fxx-xxxxxxxxx0" --secret "HRh9d2DxLGbIAxxxxxxxxxxxxxxxxxxx" --email "youremail"
and in the jenkins, you specify the DOCKER_REGISTRY entry with your acr entry name.
yourregistry.azurecr.io
thats all folks..

[cnorburn]

Thanks @jmlambert78, where are you running jx create docker auth?
When I run it both locally and on a vm nothing appears to happen, there's no output from the command, even with --verbose. It just returns to the prompt.
I suspect this is my problem (ACR and SP have been setup)

[jmlambert78]

from the machine where you run jx, in my case, I run on my PC under opensuse and use all kubectl, jx and all harness from there.
I use the jx command there.
The SP you should take the application ID of the SP as well as the generated secret.
For sure you should define the SP to have a CONTRIBUTOR role on the registry at least.

[zgorizzo69]

ACR

Yes indeed it's cool that you have found a way to make it work with acr but your question is very relevant .Why can't we use the local registry too :)

[yuwzho]

@tgarlot @jmlambert78 According to your guide, I can make the Jenkins build pipeline successfully work, but the deployment to kubernetes goes fail because of no credential.

The jx create docker auth just apply a secret to the jx namespace in k8s, so that the Jenkins can successfully push image to the ACR.
However, if we apply deployment to staging/production, the namespace cannot get the secrets from namespace jx, and then cannot pull image from the private registry ACR.

Currently I manually create the secret in the jx-staging/jx-production namespace, and modify the template/deployment.yaml in the code repo. It just gets work. But I have no idea with the PR namespace and don't think this a standard way to solve the question.

So, do you know how to work with such a private registry like ACR?

[yuwzho]

I find the jenkins-docker-cfg is saved with value like this:

https://github.com/jenkins-x/cloud-environments/blob/4d3c665aac7cb0379c5597e040e9a884d6268fe6/env-aks/myvalues.yaml#L6-L9

Then I replace the config.json with only {} by

kubectl delete secret jenkins-docker-cfg
kubectl create secret generic jenkins-docker-cfg --from-file=./config.json

Seems the credential is no longer the problem, but it runs into issue described in #659

[yuwzho]

I solve this problem in ARM level according to https://github.com/jenkins-x/jx/issues/1805#issuecomment-437261518