Kaniko: Using `executor:latest` fails with `ERROR: Job failed (system failure): unable to upgrade connection: container not found ("build")`

[petervandenabeele]

Actual behavior
Building images on kubernetes GKE cluster (1.10.7-gke.6) with kaniko-project/executor:latest fails with ERROR: Job failed (system failure): unable to upgrade connection: container not found ("build")

While building images on kubernetes GKE cluster (1.10.7-gke.6) with kaniko-project/executor:debug passes.

Expected behavior
A clear and concise description of what you expected to happen.

To Reproduce
Steps to reproduce the behavior:

1. make a .gitlab-ci.yml file like this:

build-terraform:
   stage: build
   tags:
     - kubernetes-runner
   image:
     name: gcr.io/kaniko-project/executor:latest
     entrypoint: [""]
   variables:
     DOCKER_REPO: "eu.gcr.io/<PRIVATE REPO>"
     IMG: "terraform"
     TAG: "latest"
   before_script:
     - cd terraform
   script:
     - set -e
     - FULL_IMG="${DOCKER_REPO}/${IMG}:${TAG}"
     - echo ${DOCKER_REPOSITORY_READ_WRITE_AUTH_CONFIG} > /kaniko/.docker/config.json
     - /kaniko/executor --context ${CI_PROJECT_DIR} --dockerfile Dockerfile --destination ${FULL_IMG}

1. Trigger the pipeline in Gitlab. It passes when using the debug tag, but when changing to ...
   gcr.io/kaniko-project/executor:latest it fails with:

Running with gitlab-runner 11.4.0 (8af42251)
  on gitlab-runner-gitlab-runner-58d775d877-8n4tr f9f085f7
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image gcr.io/kaniko-project/executor:latest ...
Waiting for pod gitlab/runner-f9f085f7-project-245-concurrent-129mdq to be running, status is Pending
Waiting for pod gitlab/runner-f9f085f7-project-245-concurrent-129mdq to be running, status is Pending
Running on runner-f9f085f7-project-245-concurrent-129mdq via gitlab-runner-gitlab-runner-58d775d877-8n4tr...
Cloning repository...
Cloning into '/docker-images/common'...
Checking out c1e47ef3 as master...
Skipping Git submodules setup
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("build")

For reference, the passing variant (with :debug tag) yields:

Running with gitlab-runner 11.4.0 (8af42251)
  on gitlab-runner-gitlab-runner-58d775d877-8n4tr f9f085f7
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug ...
Waiting for pod gitlab/runner-f9f085f7-project-245-concurrent-1gm5lx to be running, status is Pending
Running on runner-f9f085f7-project-245-concurrent-1gm5lx via gitlab-runner-gitlab-runner-58d775d877-8n4tr...
Cloning repository...
Cloning into '/docker-images/common'...
Checking out 4d3ba76c as master...
Skipping Git submodules setup
$ cd terraform
$ set -e
$ FULL_IMG="${DOCKER_REPO}/${IMG}:${TAG}"
$ echo ${DOCKER_REPOSITORY_READ_WRITE_AUTH_CONFIG} > /kaniko/.docker/config.json
$ /kaniko/executor --context ${CI_PROJECT_DIR} --dockerfile Dockerfile --destination ${FULL_IMG} --build-arg KNOWN_HOSTS="${KNOWN_HOSTS}"
INFO[0000] Downloading base image hashicorp/terraform:light 
ERROR: logging before flag.Parse: E1105 12:50:03.884033      15 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
ERROR: logging before flag.Parse: E1105 12:50:03.886702      15 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
2018/11/05 12:50:03 No matching credentials were found, falling back on anonymous
INFO[0001] Extracting layer 0                           
INFO[0002] Extracting layer 1                           
INFO[0002] Extracting layer 2                           
INFO[0005] Taking snapshot of full filesystem...        
INFO[0007] ENV        PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" 
INFO[0007] ENTRYPOINT "/usr/bin/env"                    
2018/11/05 12:50:11 existing blob: sha256:604ce57c398210fd46e54f5a8acb5d4fd3433779df9cbd812c753ccaed957d3a
2018/11/05 12:50:11 existing blob: sha256:796e016008b88361ad45265a113716372b21122331462795ec097b3aede856ac
2018/11/05 12:50:11 existing blob: sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde
2018/11/05 12:50:13 pushed blob sha256:d1f96a0a8d24788312cd321ced0919641988fbd2a21bd6e2ba09dd507e8be269
2018/11/05 12:50:14 eu.gcr.io/<repo name>/terraform:latest: digest: sha256:d0ba2aad282a4e47ef48ccecddfa1c30a3777959fa8a7986d0f08c043100db99 size: 754

Additional Information

• Dockerfile

FROM       hashicorp/terraform:light

ENV        PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
ENTRYPOINT "/usr/bin/env"

Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)

No extra files needed.

• Kaniko Image (fully qualified with digest)

FAILING image (docker pull on a fresh GKE kubernetes node)

<user>@gke-<name>-default-pool-5e6e87ad-wx84 ~ $ docker pull gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
439102d2b862: Pull complete 
fbbf2931c27f: Pull complete 
6f5086a00faf: Pull complete 
8eae4cd380f1: Pull complete 
2fd86d2c668e: Pull complete 
0beb7d86796f: Pull complete 
84ee8fbb25a0: Pull complete 
Digest: sha256:6eb764a69b26888855213f9f09f8ac82910151ddcade4bb270c39363b1cbaaf9
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest

PASSING image (docker pull on a fresh GKE kubernetes node)

<user>@gke-<name>-default-pool-5e6e87ad-wx84 ~ $ docker pull gcr.io/kaniko-project/executor:debug
debug: Pulling from kaniko-project/executor
439102d2b862: Already exists 
fbbf2931c27f: Already exists 
6f5086a00faf: Already exists 
35cfe197cfda: Pull complete 
245bdcdc45f4: Pull complete 
f37c673a8fc7: Pull complete 
d7673b7f6af6: Pull complete 
Digest: sha256:6fa89fcad21915dad237cd62fc807976085ea9b70ced73865e4d0bb2ab91bfa1
Status: Downloaded newer image for gcr.io/kaniko-project/executor:debug

[priyawadhwa]

Hey @petervandenabeele, according to the Gitlab documentation, only the debug image will work in Gitlab because it requires that images have a shell.

The kaniko debug image is recommended (gcr.io/kaniko-project/executor:debug) because it has a shell, and a shell is required for an image to be used with GitLab CI/CD.

[petervandenabeele]

Thank you, that is clear. I must have read over it ...

I will close this issue. The strange thing was that in quite a lot of Googling for the exact Error message combined with Gitlab etc. , I did not find any hint to this trivial cause of the problem. So maybe this will help some next user hitting this.

[de1m]

I've the same error with debug image:

Running with gitlab-runner 10.3.0 (5cf5e19a)
  on runner-gitlab-runner-856c89dd74-szxpz (00dc1024)
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image gcr.io/kaniko-project/executor@sha256:52cd9aac9c73c217ce4b25bd55640d80f0bcece56b7b4783504a74277602c575 ...
Waiting for pod gitlab-managed-apps/runner-00dc1024-project-210-concurrent-02jrjv to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-00dc1024-project-210-concurrent-02jrjv to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-00dc1024-project-210-concurrent-02jrjv to be running, status is Pending
Running on runner-00dc1024-project-210-concurrent-02jrjv via runner-gitlab-runner-856c89dd74-szxpz...
Cloning repository...
Cloning into '/dkr/test-build'...
Checking out b53f3b5f as master...
Skipping Git submodules setup
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("build")

And my is .gitlab-ci.yml:

image: lachlanevenson/k8s-kubectl

services:
  - lachlanevenson/k8s-kubectl:latest

stages:
- buildk8s
- k8s-run-test

variables:
  CONTAINER_NAME: $CI_PROJECT_NAME

tags:
- k8s-int

test-connection:
  stage: k8s-run-test
  image: lachlanevenson/k8s-kubectl:latest
  script:
  - kubectl version
  - kubectl get secret

build_image_master:
  stage: buildk8s
  image: 
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
  - echo $CI_PROJECT_DIR
  - mkdir -p /kaniko/.docker
  - echo "{\"auths\":{\"$DOCKER_REG_URL\":{\"username\":\"$DOCKER_REG_USER\",\"password\":\"$DOCKER_REG_PASS\"}}}" > /kaniko/.docker/config.json
  - cat /kaniko/.docker/config.json
  - /kaniko/executor -v debug --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $DOCKER_REG_URL/$CI_PROJECT_NAME:latest

[arthurlogilab]

Am getting this too. @de1m have you found a way around this problem ?

[arashilmg]

As mentioned in previous posts you just have to make sure sh work(be in $PATH) in your docker container.
If you used the kaniko Dockerfile to create your docker image it is changing the PATH environement variable by default, make sure that PATH is correct in the docker image and it has the shell (sh)
@de1m @arthurlogilab