skaffold build w/ kaniko fails to push image to Docker Hub

Created on 28 Nov 2019  路  13Comments  路  Source: GoogleContainerTools/skaffold

Expected behavior

skaffold build w/ kaniko successfully pushes images to Docker Hub.

Actual behavior

skaffold build w/ kaniko fails to push image to Docker Hub.

skaffold build
Generating tags...
 - okepy/skaffold-example -> okepy/skaffold-example:v1.0.0-119-g2543e76d-dirty
Checking cache...
 - okepy/skaffold-example: Not found. Building
Creating docker config secret [docker-cfg]...
Building [okepy/skaffold-example]...
Storing build context at /var/folders/jn/6z4rmbv96wbgcpd7y4k1r7y00000gn/T/context-982e4adabe73af55eafdca0c5308aee2.tar.gz
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/okepy/skaffold-example:v1.0.0-119-g2543e76d-dirty": creating push check transport for index.docker.io failed: Get https://auth.docker.io/token?scope=repository%3Aokepy%2Fskaffold-example%3Apush%2Cpull&service=registry.docker.io: invoking docker-credential-desktop: exec: "docker-credential-desktop": executable file not found in $PATH; output: 
FATA[0014] build failed: build failed: building [okepy/skaffold-example]: kaniko build for [okepy/skaffold-example]: waiting for pod to complete: condition error: pod already in terminal phase: Failed

Information

  • Skaffold version: v1.0.1
  • Operating system: macOS Catalina 10.15.1
  • Contents of skaffold.yaml:
apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
    - image: okepy/skaffold-example
      kaniko:
        buildContext:
          localDir: {}
        cache: {}
  cluster:
    namespace: default
    dockerConfig:
      # specify path or secretName
      path: ~/.docker/config.json
      # secretName: docker-cfg
deploy:
  kubectl:
    manifests:
      - k8s-*

~/.docker/config.json

{
  "stackOrchestrator" : "swarm",
  "auths" : {
    "https://index.docker.io/v1/" : {
      "auth": "base64 decoded USER:PASSWORD"
    },
  },
  "credHelpers" : {
    "us.gcr.io" : "gcloud",
    "asia.gcr.io" : "gcloud",
    "staging-k8s.gcr.io" : "gcloud",
    "marketplace.gcr.io" : "gcloud",
    "gcr.io" : "gcloud",
    "eu.gcr.io" : "gcloud"
  },
  "credsStore" : "desktop",
  "experimental" : "disabled",
  "HttpHeaders" : {
    "User-Agent" : "Docker-Client/19.03.4 (darwin)"
  }
}

Steps to reproduce the behavior

  1. https://github.com/GoogleContainerTools/skaffold/tree/master/examples/kaniko-local
  2. edit skaffold.yaml
  3. skaffold build
arebuild kinquestion prioritawaiting-more-evidence
Source
picture oke-py
👍2

Most helpful comment

I finally make it works:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: super-random
build:
  artifacts:
  - image: saphoooo/super-random # dockerhub repo
    kaniko:
      cache:
        repo: saphoooo/super-random # dockerhub repo
  cluster:
    pullSecretName: regcred
    namespace: default
    dockerConfig:
      secretName: skaffold-cfg
deploy:
  kubectl:
    manifests:
    - super-deployment.yaml

I created the secret this way:

$ kubectl create secret generic skaffold-cfg \
    --from-file=config.json
$ kubectl label secret skaffold-cfg skaffold-kaniko=skaffold-kaniko

I hope it helps.

picture saphoooo on 31 Jan 2020
👍3

All 13 comments

@oke-py Its not documented in the docs, but looks like secretName is required.
Can you please specify the podSpec generated by skaffold for the kaniko pod.

tejal29 picture tejal29 on 2 Dec 2019

@oke-py friendly ping! Can you attach -v=debug output?

tejal29 picture tejal29 on 5 Dec 2019

build.cluster.dockerConfig.secretName

apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
    - image: okepy/skaffold-example
      kaniko:
        buildContext:
          localDir: {}
        cache: {}
  cluster:
    namespace: default
    dockerConfig:
      # path: ~/.docker/config.json
      secretName: docker-cfg
deploy:
  kubectl:
    manifests:
      - k8s-*

skaffold build -v=debug
INFO[0000] Skaffold &{Version:v1.0.1 ConfigVersion:skaffold/v1 GitVersion: GitCommit:934dd5ad304bef89cb3619b5b2ef53eb8cf04715 GitTreeState:clean BuildDate:2019-11-18T16:37:50Z GoVersion:go1.13.4 Compiler:gc Platform:darwin/amd64} 
DEBU[0000] validating yamltags of struct SkaffoldConfig 
DEBU[0000] validating yamltags of struct Metadata       
DEBU[0000] validating yamltags of struct Pipeline       
DEBU[0000] validating yamltags of struct BuildConfig    
DEBU[0000] validating yamltags of struct Artifact       
DEBU[0000] validating yamltags of struct ArtifactType   
DEBU[0000] validating yamltags of struct KanikoArtifact 
DEBU[0000] validating yamltags of struct KanikoBuildContext 
DEBU[0000] validating yamltags of struct LocalDir       
DEBU[0000] validating yamltags of struct KanikoCache    
DEBU[0000] validating yamltags of struct TagPolicy      
DEBU[0000] validating yamltags of struct GitTagger      
DEBU[0000] validating yamltags of struct BuildType      
DEBU[0000] validating yamltags of struct ClusterDetails 
DEBU[0000] validating yamltags of struct DockerConfig   
DEBU[0000] validating yamltags of struct DeployConfig   
DEBU[0000] validating yamltags of struct DeployType     
DEBU[0000] validating yamltags of struct KubectlDeploy  
DEBU[0000] validating yamltags of struct KubectlFlags   
INFO[0000] Using kubectl context: kind-kind             
DEBU[0000] Using builder: cluster                       
DEBU[0000] setting Docker user agent to skaffold-v1.0.1 
Generating tags...
 - okepy/skaffold-example -> DEBU[0000] Running command: [git describe --tags --always] 
DEBU[0000] Command output: [v1.0.0-119-g2543e76d9
]     
DEBU[0000] Running command: [git status . --porcelain]  
DEBU[0000] Command output: [ M examples/kaniko-local/skaffold.yaml
] 
okepy/skaffold-example:v1.0.0-119-g2543e76d9-dirty
INFO[0000] Tags generated in 259.970194ms               
Checking cache...
DEBU[0000] Found dependencies for dockerfile: [{main.go /go true}] 
 - okepy/skaffold-example: Not found. Building
INFO[0004] Cache check complete in 4.264391523s         
Creating docker config secret [docker-cfg]...
DEBU[0004] getting client config for kubeContext: ``    
DEBU[0004] No docker config specified. Checking for one in the cluster. 
Building [okepy/skaffold-example]...
DEBU[0004] Found dependencies for dockerfile: [{main.go /go true}] 
Storing build context at /var/folders/jn/6z4rmbv96wbgcpd7y4k1r7y00000gn/T/context-8752bdfecc4c3b654809c315f9d93c74.tar.gz
DEBU[0004] getting client config for kubeContext: ``    
DEBU[0004] getting client config for kubeContext: ``    
INFO[0004] Waiting for kaniko-nl4lz to be initialized   
DEBU[0014] Running command: [kubectl --context kind-kind exec -i kaniko-nl4lz -c kaniko-init-container -n default -- tar -xzf - -C /kaniko/buildcontext] 
DEBU[0015] Running command: [kubectl --context kind-kind exec kaniko-nl4lz -c kaniko-init-container -n default -- touch /tmp/complete] 
INFO[0016] Waiting for kaniko-nl4lz to be complete      
DEBU[0016] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0017] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0018] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0019] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0020] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0021] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0022] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0023] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0025] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0027] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0029] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0035] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-nl4lz" is waiting to start: PodInitializing 
DEBU[0000] Getting source context from dir:///kaniko/buildcontext 
DEBU[0000] Build context located at /kaniko/buildcontext 
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile 
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/okepy/skaffold-example:v1.0.0-119-g2543e76d9-dirty": creating push check transport for index.docker.io failed: Get https://auth.docker.io/token?scope=repository%3Aokepy%2Fskaffold-example%3Apush%2Cpull&service=registry.docker.io: invoking docker-credential-desktop: exec: "docker-credential-desktop": executable file not found in $PATH; output: 
FATA[0038] build failed: build failed: building [okepy/skaffold-example]: kaniko build for [okepy/skaffold-example]: waiting for pod to complete: condition error: pod already in terminal phase: Failed

build.cluster.dockerConfig.path

apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
    - image: okepy/skaffold-example
      kaniko:
        buildContext:
          localDir: {}
        cache: {}
  cluster:
    namespace: default
    dockerConfig:
      path: ~/.docker/config.json
      # secretName: docker-cfg
deploy:
  kubectl:
    manifests:
      - k8s-*

skaffold build -v=debug
INFO[0000] Skaffold &{Version:v1.0.1 ConfigVersion:skaffold/v1 GitVersion: GitCommit:934dd5ad304bef89cb3619b5b2ef53eb8cf04715 GitTreeState:clean BuildDate:2019-11-18T16:37:50Z GoVersion:go1.13.4 Compiler:gc Platform:darwin/amd64} 
DEBU[0000] validating yamltags of struct SkaffoldConfig 
DEBU[0000] validating yamltags of struct Metadata       
DEBU[0000] validating yamltags of struct Pipeline       
DEBU[0000] validating yamltags of struct BuildConfig    
DEBU[0000] validating yamltags of struct Artifact       
DEBU[0000] validating yamltags of struct ArtifactType   
DEBU[0000] validating yamltags of struct KanikoArtifact 
DEBU[0000] validating yamltags of struct KanikoBuildContext 
DEBU[0000] validating yamltags of struct LocalDir       
DEBU[0000] validating yamltags of struct KanikoCache    
DEBU[0000] validating yamltags of struct TagPolicy      
DEBU[0000] validating yamltags of struct GitTagger      
DEBU[0000] validating yamltags of struct BuildType      
DEBU[0000] validating yamltags of struct ClusterDetails 
DEBU[0000] validating yamltags of struct DockerConfig   
DEBU[0000] validating yamltags of struct DeployConfig   
DEBU[0000] validating yamltags of struct DeployType     
DEBU[0000] validating yamltags of struct KubectlDeploy  
DEBU[0000] validating yamltags of struct KubectlFlags   
INFO[0000] Using kubectl context: kind-kind             
DEBU[0000] Using builder: cluster                       
DEBU[0000] setting Docker user agent to skaffold-v1.0.1 
Generating tags...
 - okepy/skaffold-example -> DEBU[0000] Running command: [git describe --tags --always] 
DEBU[0000] Command output: [v1.0.0-119-g2543e76d9
]     
DEBU[0000] Running command: [git status . --porcelain]  
DEBU[0000] Command output: [ M examples/kaniko-local/skaffold.yaml
] 
okepy/skaffold-example:v1.0.0-119-g2543e76d9-dirty
INFO[0000] Tags generated in 253.696225ms               
Checking cache...
DEBU[0000] Found dependencies for dockerfile: [{main.go /go true}] 
 - okepy/skaffold-example: Not found. Building
INFO[0004] Cache check complete in 3.912092655s         
Creating docker config secret [docker-cfg]...
DEBU[0004] getting client config for kubeContext: ``    
FATA[0004] build failed: build failed: setting up docker config secret: creating docker config secret: secrets "docker-cfg" already exists: secrets "docker-cfg" already exists

after removing secret

skaffold build -v=debug
INFO[0000] Skaffold &{Version:v1.0.1 ConfigVersion:skaffold/v1 GitVersion: GitCommit:934dd5ad304bef89cb3619b5b2ef53eb8cf04715 GitTreeState:clean BuildDate:2019-11-18T16:37:50Z GoVersion:go1.13.4 Compiler:gc Platform:darwin/amd64} 
DEBU[0000] validating yamltags of struct SkaffoldConfig 
DEBU[0000] validating yamltags of struct Metadata       
DEBU[0000] validating yamltags of struct Pipeline       
DEBU[0000] validating yamltags of struct BuildConfig    
DEBU[0000] validating yamltags of struct Artifact       
DEBU[0000] validating yamltags of struct ArtifactType   
DEBU[0000] validating yamltags of struct KanikoArtifact 
DEBU[0000] validating yamltags of struct KanikoBuildContext 
DEBU[0000] validating yamltags of struct LocalDir       
DEBU[0000] validating yamltags of struct KanikoCache    
DEBU[0000] validating yamltags of struct TagPolicy      
DEBU[0000] validating yamltags of struct GitTagger      
DEBU[0000] validating yamltags of struct BuildType      
DEBU[0000] validating yamltags of struct ClusterDetails 
DEBU[0000] validating yamltags of struct DockerConfig   
DEBU[0000] validating yamltags of struct DeployConfig   
DEBU[0000] validating yamltags of struct DeployType     
DEBU[0000] validating yamltags of struct KubectlDeploy  
DEBU[0000] validating yamltags of struct KubectlFlags   
INFO[0000] Using kubectl context: kind-kind             
DEBU[0000] Using builder: cluster                       
DEBU[0000] setting Docker user agent to skaffold-v1.0.1 
Generating tags...
 - okepy/skaffold-example -> DEBU[0000] Running command: [git describe --tags --always] 
DEBU[0000] Command output: [v1.0.0-119-g2543e76d9
]     
DEBU[0000] Running command: [git status . --porcelain]  
DEBU[0000] Command output: [ M examples/kaniko-local/skaffold.yaml
] 
okepy/skaffold-example:v1.0.0-119-g2543e76d9-dirty
INFO[0000] Tags generated in 135.094818ms               
Checking cache...
DEBU[0000] Found dependencies for dockerfile: [{main.go /go true}] 
 - okepy/skaffold-example: Not found. Building
INFO[0004] Cache check complete in 3.874077023s         
Creating docker config secret [docker-cfg]...
DEBU[0004] getting client config for kubeContext: ``    
Building [okepy/skaffold-example]...
DEBU[0004] Found dependencies for dockerfile: [{main.go /go true}] 
Storing build context at /var/folders/jn/6z4rmbv96wbgcpd7y4k1r7y00000gn/T/context-adf097e5320bdc4f79c4131b53903cfc.tar.gz
DEBU[0004] getting client config for kubeContext: ``    
DEBU[0004] getting client config for kubeContext: ``    
INFO[0004] Waiting for kaniko-7mq4q to be initialized   
DEBU[0007] Running command: [kubectl --context kind-kind exec -i kaniko-7mq4q -c kaniko-init-container -n default -- tar -xzf - -C /kaniko/buildcontext] 
DEBU[0008] Running command: [kubectl --context kind-kind exec kaniko-7mq4q -c kaniko-init-container -n default -- touch /tmp/complete] 
INFO[0008] Waiting for kaniko-7mq4q to be complete      
DEBU[0008] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-7mq4q" is waiting to start: PodInitializing 
DEBU[0009] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-7mq4q" is waiting to start: PodInitializing 
DEBU[0000] Getting source context from dir:///kaniko/buildcontext 
DEBU[0000] Build context located at /kaniko/buildcontext 
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile 
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/okepy/skaffold-example:v1.0.0-119-g2543e76d9-dirty": creating push check transport for index.docker.io failed: Get https://auth.docker.io/token?scope=repository%3Aokepy%2Fskaffold-example%3Apush%2Cpull&service=registry.docker.io: invoking docker-credential-desktop: exec: "docker-credential-desktop": executable file not found in $PATH; output: 
FATA[0012] build failed: build failed: building [okepy/skaffold-example]: kaniko build for [okepy/skaffold-example]: waiting for pod to complete: condition error: pod already in terminal phase: Failed
oke-py picture oke-py on 5 Dec 2019

I ran through the same issue, but:

It's working with path:

  cluster:
    pullSecretName: regcred
    dockerConfig:
      path: ./config.json

The config.json file is in the same directory than skaffold.yaml

It's still not working with secretName, even if I followed what is documented in the code:

// DockerConfig contains information about the docker `config.json` to mount.
type DockerConfig struct {
    // Path is the path to the docker `config.json`.
    Path string `yaml:"path,omitempty"`

    // SecretName is the Kubernetes secret that contains the `config.json` Docker configuration.
    // Note that the expected secret type is not 'kubernetes.io/dockerconfigjson' but 'Opaque'.
    SecretName string `yaml:"secretName,omitempty"`
}

  cluster:
    pullSecretName: regcred
    dockerConfig:
      secretName: dockerreg

$ k get secret
NAME                  TYPE                                  DATA   AGE
dockerreg             Opaque                                1      4m58s
regcred               kubernetes.io/dockerconfigjson        1      50m
k describe secret dockerreg
Name:         dockerreg
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
config.json:  127 bytes
saphoooo picture saphoooo on 31 Jan 2020
👍1

I have also tried to create the secret this way:

kubectl create secret generic regdocker \
    --from-file=.dockerconfigjson=config.json

But it does work.

saphoooo picture saphoooo on 31 Jan 2020

I finally make it works:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: super-random
build:
  artifacts:
  - image: saphoooo/super-random # dockerhub repo
    kaniko:
      cache:
        repo: saphoooo/super-random # dockerhub repo
  cluster:
    pullSecretName: regcred
    namespace: default
    dockerConfig:
      secretName: skaffold-cfg
deploy:
  kubectl:
    manifests:
    - super-deployment.yaml

I created the secret this way:

$ kubectl create secret generic skaffold-cfg \
    --from-file=config.json
$ kubectl label secret skaffold-cfg skaffold-kaniko=skaffold-kaniko

I hope it helps.

saphoooo picture saphoooo on 31 Jan 2020
👍3

@oke-py can you try creating the secret the way @saphoooo suggests? if it works, we should probably update the documentation to include secret creation.

nkubala picture nkubala on 29 Feb 2020

@saphoooo Thanks and apologies for late reply.
The result is below. What's regcred?

$ kubectl create secret generic skaffold-cfg \
    --from-file=$HOME/.docker/config.json
$ kubectl label secret skaffold-cfg skaffold-kaniko=skaffold-kaniko

$ kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

$ skaffold build
Generating tags...
 - okepy/super-random -> okepy/super-random:v1.4.0-73-ga9c7b3a47-dirty
Checking cache...
 - okepy/super-random: Not found. Building
Creating kaniko secret [default/regcred]...
FATA[0004] build failed: setting up pull secret: checking for existing kaniko secret: secrets "regcred" not found
oke-py picture oke-py on 1 Mar 2020

Hi @oke-py,

You need two different secrets : one for the pull (if your image is private), and one for the push.

  • For the pull you have to create a registry credential (regcred in short), with type kubernetes.io/dockerconfigjson. You can follow the Kubernetes documentation for this
  • For the push, you have to put your credentials in a secret with type opaque.

This is the difference to keep in mind ; even if the value are the same, you can't use the same secret for both operation with Kaniko and Skaffold.

And I realized that the label is not needed.

saphoooo picture saphoooo on 1 Mar 2020

I got it. I found I created the secrets into a wrong namespace (kube-system).
I created them into default namespace.

Then,

$ skaffold build
Generating tags...
 - okepy/super-random -> okepy/super-random:v1.4.0-73-ga9c7b3a47-dirty
Checking cache...
 - okepy/super-random: Not found. Building
Creating kaniko secret [default/regcred]...
Creating docker config secret [skaffold-cfg]...
Building [okepy/super-random]...
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/okepy/super-random:v1.4.0-73-ga9c7b3a47-dirty": creating push check transport for index.docker.io failed: Get https://auth.docker.io/token?scope=repository%3Aokepy%2Fsuper-random%3Apush%2Cpull&service=registry.docker.io: invoking docker-credential-desktop: exec: "docker-credential-desktop": executable file not found in $PATH; output: 
FATA[0024] build failed: building [okepy/super-random]: waiting for pod to complete: condition error: pod already in terminal phase: Failed
oke-py picture oke-py on 1 Mar 2020

@oke-py 

// secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  namespace: ...
  name: registry-secret
stringData:
  .dockerconfigjson: |
    {"auths":{
      "registry.gitlab.com":{"auth":"..."}
    }}
type: kubernetes.io/dockerconfigjson

// skaffold.yaml
apiVersion: skaffold/v1
kind: Config
build:
  cluster:
    namespace: ...
    dockerConfig:
      path: ~/.docker/config.json
  artifacts:
    - image: registry.gitlab.com/...
      context: ...
      kaniko:
        cache: {}
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'apps/**/*.*'
            dest: .
deploy:
  kubeContext: ...
  kubectl:
    flags:
      global: [--namespace=...]
    manifests:
      - ./secrets.yaml
udanpe picture udanpe on 1 Mar 2020

The error is explicit, it is not a problem of secret but of authentication: credentials do not allow you to push to index.docker.io/okepy/super-random:v1.4.0-73-ga9c7b3a47- dirty

saphoooo picture saphoooo on 1 Mar 2020

I'm closing this issue as it hasn't seen activity in awhile, and if it does still exist, it doesn't seem to be getting any traction at the moment. If this issue appears in the most recent release of Skaffold, please feel free to add a follow-up comment and we will see about getting it prioritized appropriately.

If someone sees a similar issue to this one, please create a new issue, but do include a link to this issue if possible.

Thank you for sharing this issue with us!

tstromberg picture tstromberg on 21 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

nathanph picture nathanph  路  3Comments
abatilo picture abatilo  路  4Comments
garvincasimir picture garvincasimir  路  4Comments
achedeuzot picture achedeuzot  路  3Comments
yurchenkosv picture yurchenkosv  路  3Comments