Skaffold: Local docker build fails when only the Docker CLI is available

[achedeuzot]

We’d like to use skaffold to build images in a setup where there is no Kubernetes cluster or docker socket available.

A docker daemon is available but only works through the docker CLI as there is no local docker socket. We expose a DOCKER_TLS_VERIFY=1 and DOCKER_HOST=tcp://host.example.com:1234 environment variables which allows the docker CLI and docker-compose to spawn containers.

When I run skaffold build, the image is built but the last step tries to fetch the image tag from the local docker socket ¹ which crashes with couldn't build "some-image": Error response from daemon: Client sent an HTTP request to an HTTPS server.

¹ see https://github.com/GoogleContainerTools/skaffold/blob/eb86b7f23abdd8d787dd61c8156a533ea9872279/pkg/skaffold/build/local/docker.go#L103

Expected behavior

skaffold should be shelling out to the CLI to get the image ID rather than trying to use the local daemon client so the skaffold build succeeds in the context where only the docker CLI is available.

Actual behavior

skaffold build crashes with couldn't build "some-image": Error response from daemon: Client sent an HTTP request to an HTTPS server.

Information

• Skaffold version: version v1.14.0
• Operating system: Linux
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta6
kind: Config
metadata:
  name: some-app
profiles:
  - name: "ci"
    build:
      local:
        useDockerCLI: true
      artifacts:
        - image: some-app-tests
          docker:
            target: "unit-tests"
        - image: some-app-release
          docker:
            target: "release"

Steps to reproduce the behavior

1. have a machine with no local docker socket available but a working docker CLI
2. skaffold build

[nkubala]

thanks for opening @achedeuzot, this does indeed look like a bug. we shouldn't be trying to reach out to the local docker via the CLI here.

[nkubala]

ok @achedeuzot I was finally able to reproduce this. there are a few things going on here so let me try and break it down:

first, there are a few reasons why a user might want to run their docker builds through the CLI vs the API client, but the main one I can think of is running buildkit builds (which aren't supported through the docker API), though if there are others I'd love for anyone to point them out. feature-wise, I don't think there's anything you're doing that requires you to use the CLI for builds.

when we create the API client in skaffold, we rely on docker's API itself to parse through all the various options and give us a client object we can use:

https://github.com/GoogleContainerTools/skaffold/blob/7e5ae4cbc742ac797913f76ec28f21b1540e1060/pkg/skaffold/docker/client.go#L91

here we use docker's client.FromEnv, which is supposed to parse the env vars set on the host and set the correct options - in your case, checking the DOCKER_TLS_VERIFY env var to set the scheme to https.

https://github.com/moby/moby/blob/9c15e82f19b0ad3c5fe8617a8ec2dddc6639f40a/client/options.go#L27-L43

but check out their code: all of that is contingent upon the DOCKER_CERT_PATH variable being set!

if dockerCertPath := os.Getenv("DOCKER_CERT_PATH"); dockerCertPath != "" {
 ... (set all the relevant params on the client object) ...
}

so it's as simple as that: if that env var isn't set, TLS won't be used for the client, and you'll hit the HTTP incompatibility error you're seeing.

I find this a bit silly because docker has its own default for the cert path (~/.docker) which totally bypasses the DOCKER_CERT_PATH env var, that works perfectly fine when shelling out to docker itself. it's only when you use the API client that this error surfaces 🤷

it's unclear to me whether or not using the API client when doing CLI builds in skaffold is actually "wrong": if the client is created properly, there's no reason we can't use the client to interact with the docker daemon, and the user shouldn't see any difference. right now the simplest answer is to simply leave this untouched, and document that you need to set DOCKER_CERT_PATH if you're also using DOCKER_TLS_VERIFY and doing CLI builds. I could be convinced that code changes in skaffold are the correct thing to do, but this is much more work :)

[nkubala]

going to close this issue as i'm not sure there's an actual fix to do in skaffold at the moment. if anyone has a differing opinion please comment and reopen!