Skaffold: Skaffold crashes when building Docker image using Minikube Docker daemon with buildkit

[kasperisager]

When building (using skaffold dev) the following image with Buildkit enabled, Skaffold crashes:

FROM alpine
ENTRYPOINT /bin/sh

Expected behavior

The image is built successfully.

Actual behavior

Skaffold crashes with the following error:

FATA[0004] exiting dev mode because first run failed: build failed: building [foo]: Error parsing reference: "" is not a valid repository/tag: invalid reference format

Information

• Skaffold version: v0.24.0
• Operating system: macOS 10.14.3
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta6
kind: Config

build:
  local:
    useBuildkit: true
  artifacts:
    - image: foo
      context: .

[kasperisager]

$ docker version
Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.11.5
 Git commit:        6247962
 Built:             Mon Feb 11 18:42:20 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 06:40:51 2019
  OS/Arch:          linux/amd64
  Experimental:     true

[kasperisager]

This seems to happen consistently when the first line of a given Dockerfile is not FROM ... such as:

# syntax = docker/dockerfile:experimental
FROM ...

Or simply:

#
FROM ...

This also triggers the crash:

ARG foo
FROM ...

Do note that this is not the only thing that seems to trigger the crash. This specific Dockerfile, in addition to the one I originally posted, also did:

FROM golang:alpine AS build
COPY bar.go .
RUN go build -o /app/bar
ENTRYPOINT ["/app/bar"]

[dgageot]

@kasperisager I couldn't reproduce any of those with docker 18.09.2.
Have you tried to build those images directly with buildkit?

[kasperisager]

They build just fine when using docker build, the issue only appears when run through Skaffold.

[kasperisager]

Interestingly enough, the issue sometimes persists even after removing everything before the initial FROM .... I'm really out of my depth here 🙈

[kasperisager]

This may actually be related to my setup. I use a separate Docker machine to build images (created with --engine-opt experimental=true) and then deploy to Minikube. If I stop Minikube, but leave the Docker machine running, everything seems to work consistently when building images.

[dgageot]

Can't reproduce with your sample either...

[kasperisager]

That seems to be it:

$ minikube start
$ skaffold build
<crash>
$ minikube stop
$ skaffold build
<success!>

[kasperisager]

Wild guess: Could it be that Skaffold is trying to pull the image ID from the Minikube Docker env rather than the CLI Docker env, which is used when Buildkit is enabled?

[kasperisager]

To sum up: https://github.com/GoogleContainerTools/skaffold/issues/1749#issuecomment-470526097 was a red herring and so was Buildkit for that matter. The steps to reproduce seems to be:

1. Create and start a Minikube cluster
2. Create and start a Docker machine
3. Connect to the Docker machine (eval $(docker-machine env <name>))
4. Enable useBuildkit or useDockerCLI in Skaffold
5. Run skaffold build
6. The build fails

The reason why https://github.com/GoogleContainerTools/skaffold/issues/1749#issuecomment-470526097 was failing for me was that I had an otherwise clean repo and by modifying the Dockerfile caused the image tag to change from <sha> to <sha>-dirty. The Minikube context already had an image built with a tag of <sha> and the build therefore only failed once the tag changed to <sha>-dirty.

[cpoole]

I have the same issue running on linux and trying to use buildkit with skaffold

docker 18.09.3
minikube 0.35
skaffold 0.24

Like @kasperisager said it happens after the build has finished locally and then skaffold is about to do a helm install it throws the tag error FATA[0007] exiting dev mode because first run failed: build failed: building [redacted.my.repo.address/api-security]: Error parsing reference: "" is not a valid repository/tag: invalid reference format:

[kasperisager]

Did a little digging and the offending lines seem to be:

https://github.com/GoogleContainerTools/skaffold/blob/96ac37e30ee94b3f015a5eda14c7e7b1d99b0a34/pkg/skaffold/docker/client.go#L68-L70

This I could work around by changing the name of the default Minikube context to something like local. However, Skaffold would then attempt to push the image as it no longer thought the context was local.

I'm not really sure what the best course of action is to be honest, as it's of course also very convenient having Skaffold automatically pull in the Docker env of Minikube. Maybe make it conditional on there not already being an active Docker env?

[cpoole]

I'm pretty sure the issue comes from the combo of minikube and the local docker CLI.

In local docker CLI mode the image does not get built by the daemon in minikube and therefore does not exist in that daemon's context. And since the kubecontext is minikube it automatically does not push it to a registry

So there are two workarounds:

• add push:true to the config and push the image to a registry and have the daemon in minikube pull.
• fork minikube and enable experimental support in the daemon with 18.09. Execute a normal skaffold dev without the buildkit or CLI flags.

[nkubala]

@cpoole yep you're on it. I think the real fix is for skaffold to figure out if we're running against minikube, and if we are to pull in the minikube docker env and stick that onto the CLI calls we make. this shouldn't be too difficult of a change, if anyone in this thread is interested in sending a PR :)

[cpoole]

@nkubala I'm interested in submitting a PR. I looked over this the other week. But the issue I came to is that the docker daemon in 18.06 does not support buildkit unless the --experimental flag is enabled

I ended up forking minikube and installing 18.09 https://github.com/kubernetes/minikube/blob/7e6c68811654e0b2482b12fb6a700998e94f835d/deploy/iso/minikube-iso/package/docker-bin/docker-bin.mk#L7

but I started running into bizarre errors when using buildkit over remote daemon. I'll take another stab at this but it felt like a rabbit hole.

Does this sound right or am I chasing my tail here?

[nkubala]

mmm yeah these are two separate issues I think. even if you had 18.09 installed in minikube, it still wouldn't work with skaffold because of the way we're calling docker in buildkit mode. I think there's an issue with 18.09 in minikube which is why it's not installed, @balopat might have more info. as far as buildkit over a remote daemon goes, I have no idea :)

for now, if you want to fix the issue on the skaffold side, that should at least be pretty straightforward. I think it's probably fine to assume the docker daemon is configured to have buildkit enabled from skaffold's perspective.

[cpoole]

ok, I'll take a stab at fixing on the skaffold side. I'll integration test locally with my own minikube iso

[balopat]

@cpoole any news on this one? Yeah this does look a bit like a rabbit hole type of thing :)
docker in minikube will be upgraded: https://github.com/kubernetes/minikube/issues/3519
I have not tried yet buildkit on minikube - but based on my limited experimentation with buildkit it might not be that trivial to setup - any experience/pioneering can be valuable!

[meatherly]

I can confirm I get this issue on Linux using buildkit. If I delete m minikube instance it gets past the build but fails on deployment because there is no minikube instance but if I spin up minikube it breaks right after the build is finished.

@nkubala I was looking at the debug logs for skaffold and it looks like it pulls the docker env from minikube?

DEBU[0000] Running command: [minikube docker-env --shell none] 
WARN[0000] Could not get minikube docker env, falling back to local docker daemon: getting minikube env: Running [minikube docker-env --shell none]

[rugwirobaker]

I too can confirm the same behavior as @meatherly on Linux. My builds(building the skaffold examples without any change to the Dockerfiles) fail when Minikube is running like:

FATA[0057] build failed: building [guestbook-backend]: build artifact: docker build: Error response from daemon: client version 1.39 is too new. Maximum supported API version is 1.38

When I stopped minikube the build went through but as expected the deployment failed.

Then I tried to downgrade by running eval $(minikube docker-env | grep -v DOCKER_API_VERSION) as per this comment: https://github.com/kubernetes/minikube/issues/338#issuecomment-244580707 but still the problem persisted.

[rugwirobaker]

This thread seemed a little to quite so I opened this issue: #2187 and referenced it in the skaffold slack channel: https://kubernetes.slack.com/archives/CABQMSZA6/p1559010140012200. The issue was due to my minikube VM using an old ISO so minikube delete followed by minikube start which downloaded a new ISO solved it for me. @kdkeyser you should try it and see if you get the same results and close the issue. My current settings:
OS: ubuntu/amd64
minikube version: v1.1.0
skaffold version: v0.30.0

[rosscdh]

osx too
minikube version: v1.1.0
skaffold version: v0.30.0

[demisx]

Same problem for me. I've tried to delete minikube and completely reinstall it. No luck:

• OS: MacOs Mojave 10.14.5
• minikube: 1.3.1
• skaffold: (bleeding edge master f48e7dd)

[balopat]

I accidentally reproduced this with @haf's repro on a separate issue: https://github.com/haf/skaffold-dockerignore - change the skaffold yaml to useDockerCLI: true creates this issue on minikube context, works in non minikube/non-local.

[demisx]

I also have it set to true to be able to pass env vars as build args, I believe. This is my skaffold YAML if it helps:

apiVersion: skaffold/v1beta13
kind: Config
profiles:
  - name: dev
    activation:
      - command: dev
    deploy:
      kubectl:
        manifests:
          - api/app/k8s/dev/api.k8s-config.yaml
          - api/app/k8s/dev/db.k8s-config.yaml
          ...
          - web/k8s/dev/web.k8s-deployment.yaml
          - web/k8s/dev/web.k8s-service.yaml
    build:
      local:
        useDockerCLI: true
      artifacts:
        - image: dl-logstash
          sync:
            manual:
              - src: 'api/app/logstash/config/dev/pipeline/*'
                dest: pipeline
          docker:
            dockerfile: api/app/logstash/Dockerfile
            target: dev-env
            cacheFrom:
              - docker.elastic.co/logstash/logstash-oss:6.6.1
        - image: init-db
          docker:
            dockerfile: api/Dockerfile
            target: init-db
            buildArgs:
              DL_ORG_DEV_NAMESPACE: '{{.DL_ORG_DEV_NAMESPACE}}'
              DL_ORG_AWS_ACCOUNT_ID: '{{.DL_ORG_AWS_ACCOUNT_ID}}'
              DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID: '{{.DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID}}'
              DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY: '{{.DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY}}'
        # API service
        - image: dl-api.dev
          context: .
          sync:
            manual:
              - src: 'api/app/src/**/*.ts'
                dest: .
                strip: 'api'
              - src: 'api/app/src/**/*.json'
                dest: .
                strip: 'api'
          docker:
            dockerfile: api/Dockerfile
            target: dev-env
            buildArgs:
              DL_ORG_DEV_NAMESPACE: '{{.DL_ORG_DEV_NAMESPACE}}'
              DL_ORG_AWS_ACCOUNT_ID: '{{.DL_ORG_AWS_ACCOUNT_ID}}'
              DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID: '{{.DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID}}'
              DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY: '{{.DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY}}'
        - image: dl-api.test
          context: .
          sync:
            manual:
              - src: 'api/test/**/*'
                dest: .
                strip: 'api'
          docker:
            dockerfile: api/Dockerfile
            target: test
            buildArgs:
              DL_ORG_DEV_NAMESPACE: '{{.DL_ORG_DEV_NAMESPACE}}'
              DL_ORG_AWS_ACCOUNT_ID: '{{.DL_ORG_AWS_ACCOUNT_ID}}'
              DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID: '{{.DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID}}'
              DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY: '{{.DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY}}'
        - image: dl-web
          context: .
          sync:
            manual:
              - src: 'web/src/**/*'
                dest: .
                strip: 'web'
              - src: 'web/e2e/**/*'
                dest: .
                strip: 'web'
          docker:
            dockerfile: web/Dockerfile
            target: dev-env
      - resourceType: service
        resourceName: postgres
        port: 5432
      - resourceType: service
        resourceName: elasticsearch
        port: 9200

[haf]

@demisx You don't need to use the docker CLI for passing env vars: https://github.com/haf/skaffold-dockerignore/blob/master/skaffold.yaml#L6-L10 works for me using docker on desktop.

[woodcockjosh]

I had this issue when I specified an image tag in the image property. I removed :Tag_Name and it stopped crashing. The problem seems to be with the error message which is quite vague rather than a bug in the code.

[demisx]

@haf Oh, nice! Thanks man. I was told here it was necessary, but I guess no more. Maybe this was a requirement in an earlier version of skaffold.

UPDATE: The older v0.31 skaffold requires the local.useDockerCLI: true to work.

[demisx]

Also, I'd like to confirm that removing useDockerCLI: true made the error go away. It's working with minikube now.

[demisx]

Using build.local.useBuildkit: true will cause the same error too.

[dgageot]

Should be all good now

[dolsem]

I'm having the same problem on version 0.40.0 of Skaffold.

skaffold.yaml

apiVersion: skaffold/v1beta16
kind: Config

build:
  local:
    push: false
    useBuildkit: true
  artifacts:
    - image: nest
      sync:
        infer:
          - src/**/*.ts
          - .env.json
      custom:
        buildCommand: 'docker build -t nest -f Dockerfile.dev --ssh default .'
        dependencies:
          paths:
            - src
            - .env.json

deploy:
  kubectl:
    manifests:
      - k8s/nest-node-port.yaml
      - k8s/nest-deployment.yaml

Dockerfile.dev

# syntax=docker/dockerfile:experimental
FROM node:current-alpine

RUN apk add --no-cache \
  openssh-client git   \
  build-base           \
  bash python          \
  expect

RUN mkdir -m 0600 /root/.ssh/
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan gitlab.com >> /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts

WORKDIR /app

COPY package.json ./
COPY .env.json ./
COPY jest.* ./
COPY nodemon* ./
COPY tsconfig* ./
COPY scripts scripts
COPY src src

RUN --mount=type=ssh npm install

EXPOSE 3000
EXPOSE 3001

CMD ["npm", "run", "start:dev"]

[dgageot]

@dolsem It looks like you are using another builder, the custom builder. Could you please create another issue?

[dmarvasti]

This thread seemed a little to quite so I opened this issue: #2187 and referenced it in the skaffold slack channel: https://kubernetes.slack.com/archives/CABQMSZA6/p1559010140012200. The issue was due to my minikube VM using an old ISO so minikube delete followed by minikube start which downloaded a new ISO solved it for me. @kdkeyser you should try it and see if you get the same results and close the issue. My current settings:
OS: ubuntu/amd64
minikube version: v1.1.0
skaffold version: v0.30.0

Solved for me as well..