Skaffold: Private base images don't work with skaffold properly

[dcherniv]

Skaffold cannot pull from private registry. Docker daemon can.
OS and docker daemon:

root@debbie:~# uname -a
Linux debbie 4.14.0-3-amd64 #1 SMP Debian 4.14.17-1 (2018-02-14) x86_64 GNU/Linux
root@debbie:~# lsb_release  -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux testing (buster)
Release:        testing
Codename:       buster
root@debbie:~# docker -v
Docker version 17.12.1-ce, build 7390fc6
root@debbie:~# 

Skaffold:

root@debbie:~# skaffold version
v0.2.0
root@debbie:~# 

Skaffold output:

dcherniv@debbie:~REDACTED$ skaffold run
Starting build...
Found minikube or Docker for Desktop context, using local docker daemon.
WARN[0000] Could not get minikube docker env, falling back to local docker daemon 
Sending build context to Docker daemon  814.1kB
Step 1/22 : FROM REGISTRY/REDACTED/REDACTED:latest
ERRO[0000] run: running skaffold steps: run: build step: running build: unauthorized: The client does not have permission for manifest 

Docker output:

dcherniv@debbie:~REDACTED$ docker pull REGISTRY/REDACTED/REDACTED:latest
latest: Pulling from REGISTRY/REDACTED/REDACTED
6d987f6f4279: Pull complete 
dab216b6afad: Pull complete 
c9580cbb00a7: Pull complete 
Digest: sha256:340d53a95a520fb94b4eda1718780bd841a7d8ae484181e700163da75dfd742b
Status: Downloaded newer image for REGISTRY/REDACTED/REDACTED:latest
dcherniv@debbie:~/REDACTED$ 

After the image is downloaded to local, skaffold run succeeds. Output ommited for brevity.

What gives? my docker config is in ~/.docker/config.json

Auth is base64 encoded:

        "auths": {
                "https://REGISTRY": {
                        "auth": "REDACTEDBASE64",
                        "email": "[email protected]"
                },
          [....]
        }

In private docker registry logs i see requests from skaffold coming in as anonymous/unauthenticated, where with docker pull i see the request having a proper username.
Private registry is running on artifactory.

[r2d4]

Thanks for the detailed issue! This is broken across all the private repositories - it appears I've added an embarassing TODO here where the auth should go. We get the right "auth configs" for push, but not for pull.

I think it should be as simple as implementing a function similar to the one in the docker CLI

1. parse the image name for registry
2. get the auth for that registry

docker cli example:
https://github.com/docker/cli/blob/c0ffb9491cdffb628e18bb491b566255987fd28d/cli/trust/trust.go#L301-L333

skaffold's TODO:
https://github.com/GoogleCloudPlatform/skaffold/blob/master/pkg/skaffold/docker/image.go#L53-L56

Happy to help out if you're interested in contributing this fix yourself, or I'll get to it soon since it seems pretty high priority.

[dgageot]

@r2d4 I'm afraid there's nothing we can do expect uncomment your line. It's slow, not really because of docker, but because of any credential helper you might have installed.

[dlorenc]

https://github.com/GoogleCloudPlatform/docker-credential-gcr

The source is here. We could probably try to speed it up.

[dgageot]

@dlorenc I could push some code that parses all the auth tokens but doesn't query the credential helpers yet. It would already solve a lot of issues.

[dgageot]

@dcherniv have you tested master branch?

[dgageot]

@dcherniv did you get a chance to test your use case with 0.3?

[dcherniv]

@dgageot will test in a couple of days. My apologies been a bit busy.

[dcherniv]

@dgageot I just tested it and it appears to work. I have a question about how it's implemented.
As far as i can see my docker build step happens on the minikube VM, but the VM is not authenticated to the registry only my host machine is.
Does it read my ~/.docker/config.json and passes it to minikube?
I also see this pod struggling to start. Not sure if it is intentional.

registry-creds-p5trb                    0/1       ContainerCreating   0          13m

 conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-04-03T22:49:11Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-04-03T22:49:11Z
    message: 'containers with unready status: [registry-creds]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-04-03T22:49:11Z
    status: "True"
    type: PodScheduled

[r2d4]

@dcherniv it detects the minikube context and does the equivalent of a minikube docker-env without doing so.

Your second issue seems to be with the registry-creds addon, which is probably better suited for https://github.com/upmc-enterprises/registry-creds or the minikube issue tracker

[r2d4]

The auth resolution happens on the host with the help of credential helpers, and is sent to the minikube docker daemon, much like the docker cli does and communicates with the docker daemon.

I'm going to close this since the initial issue seems to be solved

[james-masson]

This is still broken for me - built skaffold from master today.

Without local image

JEG-CON-GEL0068:service james.masson$ skaffold dev
Starting build...
Sending build context to Docker daemon  303.6MB
Step 1/4 : FROM eu.gcr.io/<private>/<private>-java8:1.0.8
WARN[0024] run: build: build step: running build: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Watching for changes...

Manual pull of image

JEG-CON-GEL0068:service james.masson$ docker pull eu.gcr.io/<private>/<private>-java8:1.0.8
1.0.8: Pulling from <private>/<private>-java8
e12c678537ae: Pull complete
8d9ed335b7db: Pull complete
5a6dcc67a16a: Pull complete
6fcd5d005ecf: Pull complete
0e13ccade694: Pull complete
851ab8636be5: Pull complete
449966edb374: Pull complete
b02bf6cc8e48: Pull complete
666617777f1a: Pull complete
5cf04f37ded3: Pull complete
Digest: sha256:752254105475f1d0f42bbb85842fae9ce40933d86a4b7a11cfa01e8dbf6cc7d9
Status: Downloaded newer image for eu.gcr.io/<private>/<private>-java8:1.0.8

success with skaffold

JEG-CON-GEL0068:service james.masson$ skaffold dev
Starting build...
Sending build context to Docker daemon  303.6MB
Step 1/4 : FROM eu.gcr.io/<private>/<private>-java8:1.0.8
 ---> 7984842b12c0
Step 2/4 : MAINTAINER Data2 Services
 ---> Running in 8c9a910aa61c
 ---> 09a693e7dcf3
<More steps>
Successfully built 1da4583598af

[sboardwell]

I can confirm this is happening for me as well with Skaffold v20.0 and a private GCR.

[tjerkw]

For my the prepulling of cacheFrom images also fails for private registries:

WARN[0005] Error processing base image (quay.io/tiqets/some-image:develop) for ONBUILD triggers: getting remote config: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
. Dependencies may be incomplete.

I think this is related?

[dgageot]

@dcherniv @tjerkw @sboardwell Have you tried with a recent version of Skaffold?

[sboardwell]

@dgageot - no, not yet, I will check and get back to you. Thanks for answering.

[nkubala]

@sboardwell I think you want to flag your registry as insecure when you run skaffold.

skaffold dev --insecure-registry <your_registry>

[sboardwell]

@sboardwell I think you want to flag your registry as insecure when you run skaffold.

skaffold dev --insecure-registry <your_registry>

The registry is not insecure (as in available over http), it is just private. I can try that as well but it would be an incorrect flag so not very intuitive IMO, if that is in fact the solution.

[nkubala]

ah sorry I misunderstood. I don't think this flag will fix your issue, it's just for flagging connections as insecure (http vs. https). try out the latest version of skaffold though and see if you're still seeing the issue

[tejal29]

ok, i tried this on latest master and do see a failure. I am not sure if this is the original issue.

I made the following changes:

tejaldesai:getting-started (master)$ git diff
diff --git a/examples/getting-started/skaffold.yaml b/examples/getting-started/skaffold.yaml
index b98ebe9d..c395a68b 100644
--- a/examples/getting-started/skaffold.yaml
+++ b/examples/getting-started/skaffold.yaml
@@ -1,9 +1,15 @@
 apiVersion: skaffold/v1beta13
 kind: Config
 build:
+  local:
+    push: true
   artifacts:
   - image: gcr.io/k8s-skaffold/skaffold-example
 deploy:
   kubectl:
     manifests:
       - k8s-*

Here is the error i see

tejaldesai:getting-started (master)$ skaffold run --default-repo myprivate:5001/test --tail
Generating tags...
 - myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example -> myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example:v0.33.0-205-gffd06082-dirty
Tags generated in 21.84194ms
Starting build...
Found [minikube] context, using local docker daemon.
Building [myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM scratch
 ---> 
Step 2/3 : COPY hello /
 ---> Using cache
 ---> f9c1d28b5440
Step 3/3 : CMD ["/hello"]
 ---> Using cache
 ---> de2dbb1b3aa2
Successfully built de2dbb1b3aa2
Successfully tagged myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example:v0.33.0-205-gffd06082-dirty
The push refers to repository [myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example]
FATA[0000] failed to build: build failed: building [myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example]: build artifact: Get https://myprivate:5001/v2/: dial tcp: lookup myprivate on 192.168.122.1:53: no such host 

If, i remove the push:true, the build passes but fails in deploy.

tejaldesai:getting-started (master)$ skaffold run --default-repo myprivate:5001/test --tail
Generating tags...
 - myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example -> myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example:v0.33.0-205-gffd06082-dirty
Tags generated in 16.337179ms
Starting build...
Found [minikube] context, using local docker daemon.
Building [myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM scratch
 ---> 
Step 2/3 : COPY hello /
 ---> Using cache
 ---> f9c1d28b5440
Step 3/3 : CMD ["/hello"]
 ---> Using cache
 ---> de2dbb1b3aa2
Successfully built de2dbb1b3aa2
Successfully tagged myprivate:5001/test/gcr_io_k8s-skaffold_skaffold-example:v0.33.0-205-gffd06082-dirty
Build complete in 60.735854ms
Starting test...
Test complete in 9.217µs
Starting deploy...
kubectl client version: 1.11+
kubectl version 1.12.0 or greater is recommended for use with Skaffold
FATA[0000] reading manifests: kubectl create: Running [kubectl --context minikube create --dry-run -oyaml -f /usr/local/google/home/tejaldesai/go/src/github.com/GoogleContainerTools/skaffold/examples/getting-started/k8s-pod.yaml]: stdout , stderr: The connection to the server 192.168.39.53:8443 was refused - did you specify the right host or port?
, err: exit status 1: exit status 1 

[mattnworb]

I think the problem is that, at least on my system, GetAllAuthConfigs is returning an empty map: https://github.com/GoogleContainerTools/skaffold/blob/4ba3d06e21ef9ca34c9e21c1550c177360013a4a/pkg/skaffold/docker/image.go#L136-L141

If I change the implementation to call cf.GetAllCredentials() instead of cf.GetCredentialsStore("").GetAll(), then my skaffold build - which needs to pull a private gcr.io image - works fine.

this is with a simple ~/.docker/config.json of:

{
  "credHelpers" : {
    "marketplace.gcr.io" : "gcloud",
    "asia.gcr.io" : "gcr",
    "us.gcr.io" : "gcr",
    "staging-k8s.gcr.io" : "gcr",
    "eu.gcr.io" : "gcr",
    "gcr.io" : "gcr"
  }
}

[mattnworb]

When cf.GetCredentialsStore() is called with an empty string, it uses a fileStore struct as the credentials.Store implementation:

https://github.com/docker/cli/blob/v18.09.0/cli/config/configfile/file.go#L259-L266

fileStore.GetAll() just delegates to store.GetAuthConfigs():

https://github.com/docker/cli/blob/v18.09.0/cli/config/credentials/file_store.go#L49

which returns the AuthConfigs field from the configFile struct:

https://github.com/docker/cli/blob/v18.09.0/cli/config/configfile/file.go#L141

which is mapped to the auths field of the json config file:

https://github.com/docker/cli/blob/51668a30f26250ccfce31bcc13d9334eaafabe36/cli/config/configfile/file.go#L27-L28

In other words - seems like this bypasses credHelpers altogether.

[mattnworb]

I submitted a PR above that I think will solve the issue for pulls, but also wanted to mention that discovering useDockerCLI: true in https://skaffold.dev/docs/how-tos/builders/#dockerfile-locally-with-docker also fixes the problem for our use case.

[vitobotta]

HI, I am having this problem right now and have downloaded the latest binary at https://storage.googleapis.com/skaffold/builds/latest/skaffold-darwin-amd64. Is there a workaround? Thanks

[nkubala]

@vitobotta I think the fix for this went out in v0.36.0, you might have downloaded v0.35.0 right before we released. can you make sure you're on the latest version? if you are, can you provide a little more information about your issue?

[kwngo]

@nkubala Still getting this error in v0.36.0. It works fine with v0.29.0, so it must have been re-introduced at some point?

[mattnworb]

@kwngo do you see the problem with pushes, or pulls, or both? I am not sure if the PR I submitted above fixes it for pushes.