Vscode-remote-release: Buildkit support at devcontainer.json level

[darkvertex]

This is technically a dupe of https://github.com/microsoft/vscode-remote-release/issues/87 but since the other issue was closed and further commenting disallowed, I could not continue there.

Buildkit is great and very useful and right now if my Dockerfile absolutely needs it to build, it's great that I can control the runArgs in the devcontainer.json but it won't set DOCKER_BUILDKIT=1 for me at build time, so I need to remember to export that env var prior to launching vscode or my dev container won't build. It's not very ergonomic.

I found this doc page that suggests adding env vars by modifying the runArgs but from what I can tell that's only good for passing env vars to docker run, not setting them for docker build:
https://code.visualstudio.com/docs/remote/containers-advanced#_adding-environment-variables

In the devcontainer.json, I would love either:

a) "use_buildkit": true or similar.
b) or something to set the env for the docker build call, maybe:

"buildEnv": {
  "DOCKER_BUILDKIT": "1"
}

Thoughts?

[chrmarti]

Does "runArgs": [ "-e", "DOCKER_BUILDKIT=1" ] work for your case? (https://github.com/microsoft/vscode-remote-release/issues/1201 will address having something like you suggest in b)).

[darkvertex]

Sorry, I made a typo repeatedly... Everywhere I wrote about docker run I meant docker build. Buildkit adds fancy features at build time; it does nothing for running the containers.

I've edited my original text; apologies for the confusion.

My request still stands though. I need an env to be set _before_ running docker build.

[chrmarti]

You can make buildkit the default: https://github.com/docker/buildx#setting-buildx-as-default-builder-in-docker-1903

Does that work for you?

[darkvertex]

That's a valid temporary workaround though I'm not a huge fan of having wrappers wrap builtin commands.

Sometimes I want the classic builder and it's nice to have the choice to decide when to use Buildkit or not inline without having to permanently edit my config by installing an alias.

[eedwards-sk]

I would like to use docker build secrets and cannot with the current devcontainer support for buildkit.

Forcing the entire system to use buildkit is not a solution or valid workaround -- the impact is too great to other projects.

[mrmachine]

You can make buildkit the default: https://github.com/docker/buildx#setting-buildx-as-default-builder-in-docker-1903

Does that work for you?

I believe this workaround does not work with:

    "dockerComposeFile": [
        "docker-compose.yml"
    ],

We need to set DOCKER_BUILDKIT=1 and COMPOSE_DOCKER_CLI_BUILD=1 env vars before VScode builds our image with compose.

[noedlm]

I'm running into this same issue. For now I just build the image then start the container in vscode. Having this feature would be nice for sure.

[chrmarti]

@mrmachine You can place a .env file with these environment variables in your project folder. Does that work?

[noedlm]

I got pulled to work on something else, I'll try it out as soon as I can and update whether it worked or not.

[agners]

Place DOCKER_BUILDKIT=1 in .env does work for me, the container gets built using BuildKit.

However, the build output looks a bit messy since BuildKit seems to assume a full terminal. Using --progress=plain in docker build does help but I did not found a way to pass extra arguments using devcontainer.json. There is also the environment variable BUILDKIT_PROGRESS=plain which does the same, and it seems to be working when using docker build on the command line. For some reason that environment variable does not work with .env, I assume it's related to how Visual Studio code starts the container build process.

[mrmachine]

Adding to .env file (read by compose) did not work for me, but adding to my ~/.zshenv file did work.

The output looks fine for me. I'm using zsh and zprezto on macOS.

UPDATE: Adding COMPOSE_DOCKER_CLI_BUILD=1 to .env is respected by docker-compose (CLI and vscode) because it is a Docker Compose var. But adding DOCKER_BUILDKIT=1 is ignored by docker-compose (CLI and vscode), presumably because it is a Docker var.

I don't know why adding it to my .zshenv file suddenly started working. I thought it was not working before. I had assumed that vscode would only inherit my env if started from CLI with code. But it seems ~/.zshenv is used even when vscode is launched directly from the macOS dock.

[agners]

@mrmachine you are indeed right .env also did not work for me. It seems I got confused during testing where I exactly added the variable already :-)

I assume you are using dockerComposeFile in .devcontainer/devcontainer.json then? I tested only with build.dockerfile so far. And indeed.

[CarlosDomingues]

Just adding to what others have said, my main use case for Remote - Containers is to share a ready-to-use, reproducible and isolated development environments with my team, so they can just git clone a project and start working right away.

Solutions that require manually configuring Docker or editing .env files (which often should not be checked) are cool workarounds, but greatly diminish the solution's value.

[cgorski-vso]

My team relies heavily on caching using cache_from in compositions to build containers on developer machines by using remote repositories as a cache. This requires BuildKit. Outside of VSCode, on the CLI, I can build a rather large composition rather quickly by first pulling the images and then running docker-compose using BuildKit.

Despite my best efforts (mostly around setting environment variables in every conceivable way I can think of, and installing the BuildKit CLI wrappers), I cannot get VSCode on Windows to use BuildKit. My compositions build, but the process is agonizingly slow. BuildKit reduces build times from 15 minutes to 2 minutes on some developer machines, so this is a big hit to productivity.

[cgorski-vso]

To follow up on my previous comment:

The current workaround is to create shell scripts to be run outside of VSCode that manually pull, build, and bring up the composition. I can then use "Open folder in container ..." in VSCode. This indeed allows me to build a large composition in dramatically reduced time.

However, as others have pointed out, this workaround defeats the purpose of some of the core principals around the docker and remote execution extensions. It requires manually running scripts outside of VSCode to control docker-compose. It also introduces the potential for errors into the build process due to human error, which I have already made several times (but despite that the workaround still beats waiting 15 minutes for a build).

[PavelSosin-320]

The recent Docker Desktop versions support buildkit as a CLI plugin(!) - Docker Desktop buildx. It is enabled by env variable.
Docker-compose is not perfect in the image build because its main purpose today is Docker stack deploy input. You can't expect too much from it.
But there are numerous OCI Container builders like Podman's buildah. Kubernetes' kaniko. Maybe, it is not so bad that image building is done outside VSCode as a Task?DockerHub is not the only possible repository.