Compose: Support buildkit?

[khs1994]

Please post on our forums: https://forums.docker.com for questions about using docker-compose.

Posts that are not a bug report or a feature/enhancement request will not be addressed on this issue tracker.

docker-compose build command support buildkit?

[shin-]

Not at the moment.

[ulgens]

@shin- Why?

[thaJeztah]

Being tracked here; https://github.com/docker/docker-py/issues/2230

[kenorb]

Is there any workaround for passing DOCKER_BUILDKIT as variable so buildkit is activated, or it's not supported either?

E.g.

docker-compose build --build-arg DOCKER_BUILDKIT=1 myservice

or:

DOCKER_BUILDKIT=1 docker-compose build myservice

I've tried and it doesn't work. My docker supports DOCKER_BUILDKIT.

[thaJeztah]

No, currently not possible through docker compose; the DOCKER_BUILDKIT=1 environment variable is handled by the docker cli to switch to a different API for the build; that API uses an "interactive session", where instead of sending the whole build-context, the daemon interacts with the client to only send the files that are needed.

This protocol is not (yet) supported by docker-py (the SDK that's used by Docker Compose to interact with the Docker API), so for that reason, it's not possible yet.

[batjko]

This protocol is not (yet) supported by docker-py (the SDK that's used by Docker Compose to interact with the Docker API), so for that reason, it's not possible yet.

Thank you for the clear explanation!

Does this mean compose is essentially waiting for docker-py to implement that capability and will then also support it?

[thaJeztah]

Likely the docker cli will be used as a helper to implement it; see https://github.com/docker/compose/pull/6584#issuecomment-502099098 for more details

[abdennour]

Any deadline to release this feature and make it GA ? I am switching back to docker build which is very bad.

[adriantorrie]

See PR: https://github.com/docker/compose/pull/6865

[adaviding]

@adriantorrie Does your PR cover everything that the build kit provides?

Basically I'm trying to use docker-compose to mimick docker build --ssh default my_image as described here: https://stackoverflow.com/a/55689772/2133451

I was guessing that something like this would work:

export COMPOSE_DOCKER_CLI_BUILD=1
export DOCKER_BUILDKIT=1
docker-compose build --ssh default my_image

Am I doing something wrong? Or is this basically not supported yet?

[thaJeztah]

Not all options are supported yet; we'll probably need to create a tracking issue for that.

The current implementation maintains the options that docker-compose build supported and that are supported by the cli/buildkit.

For other options, more discussion is needed, especially how t handle options that could apply to a build for a specific service, and therefore cannot / should not be set "globally"; see my comment on https://github.com/docker/compose/pull/6584#issuecomment-513210065

For options that make sense to set unconditionally on all service build's, those could be added as a flag, but for options that should be scoped per service/build, having then in the compose file would make more sense.

For example, with --ssh, you may not want to give _all_ services access to your ssh agent / credentials, and want to limit access to those builds that actually need them.

[gamesover]

@thaJeztah that's true. Probably docker-compose could consider to move --ssh into docker-compose.yml to support for a particular service. such as

web:
    build:
      context: .
      dockerfile: docker/Dockerfile.development
      ssh: true

Anyway, wish docker-compose to support --ssh soon :)

[maxwellsmart84]

To anyone else who found themselves here many times (like i did), I believe that when this is merged it will add the ability to use --ssh : #7046

[TheDiveO]

To anyone else who found themselves here many times (like i did), I believe that when this is merged it will add the ability to use --ssh : #7046

7046 isn't merged yet.

[gamesover]

7046 isn't merged yet.

You can fork and merge by yourself ;)

[abdennour]

It was released now. COMPOSE_DOCKER_CLI_BUILD=1 must be set with docker-compose CLI ,
like DOCKER_BUILDKIT=1 with docker CLI

[thaJeztah]

☝️ Actually, both have to be set if you want to use buildkit;

• set COMPOSE_DOCKER_CLI_BUILD=1 to use the docker CLI for building
• in addition, set DOCKER_BUILDKIT=1 to enable BuildKit

[guysoft]

@abdennour What version was it released in, is there a way to confirm its working?

[zhammer]

@guysoft from this blog post seems like 1.25.1 https://www.docker.com/blog/faster-builds-in-compose-thanks-to-buildkit-support/