Docker-py: Support BuildKit

[chris-crone]

Currently the Python SDK performs builds using the older build infrastructure. It would be good to support BuildKit so that users of the SDK can benefit from its various advantages.

[chris-crone]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the _client_ side) and that service is distinctly non-trivial to implement.

[chris-crone]

Request for build secret support: https://github.com/docker/docker-py/issues/2174

[haizaar]

Request for build secret support: #2230

Did you mean to link to my original ticket? :)

[chris-crone]

Fixed, thanks @haizaar

[AkihiroSuda]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

In addition to fssync, there are auth, content, secrets, and sshforward client-side services.

As @mipearson commented in https://github.com/moby/buildkit/issues/685 , docker-py should just invoke docker build CLI.

[tonistiigi]

My recommendation is to create a helper binary that compose can call as a quick practical solution. Note that fssync is optional for BuildKit, the Docker API can still receive tarballs fine and build them with BuildKit. It's less optimal but you still get the other benefits. But if you want secrets/ssh support then you do need either a helper or reimplement the grpc provider.

[AkihiroSuda]

Do we need a helper binary other than existing docker CLI?

[tonistiigi]

There are some things that might be interesting for compose that can be done in Go library and not in CLI atm. Like doing multiple requests in parallel with a shared session.

[mipearson]

Is there a possible future where most docker-compose things are done via a helper binary rather than via docker-py?

I seem to be having a very bad run of hitting docker-py specific issues lately :(

[tedivm]

Is this something people are working on? It sounds like a direction hasn't really even been picked yet.

[pmcanseco]

I have chosen to work around the lack of buildkit support by just calling the docker-cli in my projects. It works pretty well for me.

os.environ["DOCKER_BUILDKIT"] = 1
docker_build_command = 'docker build {} {}'
                       .format('--tag {} --target {} --rm=true --no-cache={} '
                               .format(tag, target, True),
                               path)
raw_build_result = subprocess.check_output(docker_build_command, 
                                           shell=True, stderr=subprocess.STDOUT,
                                           timeout=240)

[haizaar]

I have chosen to work around the lack of buildkit support by just calling the docker-cli in my projects.

Same here.

[tedivm]

I think that's what we're all doing, but it means that docker-compose is basically a nonentity at this point, and frankly docker-compose is much easier to use when setting up test environments. It's really unfortunate that such a useful tool is no longer usable due to this issue, and that there doesn't seem to be any timeline for resolving it (at this point I'm assuming docker compose will only get buildkit functionality when it becomes the default option).

[AkihiroSuda]

There is an exec-based PR for docker-compose: https://github.com/docker/compose/pull/6584

[adriantorrie]

There is an exec-based PR for docker-compose: docker/compose#6584

That PR has now been closed, and a new one opened: https://github.com/docker/compose/pull/6865

[abdennour]

Any deadline to release this feature and make it GA ? I am switching back to docker build which is very bad.

[FernandoMiguel]

for all the folks here
have you seen https://github.com/docker/buildx ?
I successfully replaced docker-compose build with it
here's an example https://github.com/FernandoMiguel/BuildKit/blob/556808da7543cdb46f73853876c087628b6221dc/.github/workflows/build.yml#L137

[hrw]

Any update?

[AkihiroSuda]

https://github.com/docker/compose/pull/6865 got merged

I suggest closing this issue

[hrw]

How docker/compose is connected to docker/docker-py?

[AkihiroSuda]

docker/compose is the most widely known consumer of docker/docker-py.
Other python projects should follow docker/compose (i.e. execute docker build command)

[hrw]

Then what the point of whole 'docker-py' if all projects have to switch to use Docker cli?

Your comment suggest "abandon all hope, forget about docker-py, use cli directly'.

[chris-crone]

Hi @hrw,

We'd gladly accept PRs for this and would help where possible. Note that this is tricky because of the issue mentioned above.

For Compose, the decision was taken to use the Docker CLI for build as it's unlikely someone will have Compose but not the CLI.

[hrw]

@chris-crone outside of my Python knowledge I suspect ;(

[FernandoMiguel]

not trying to be repetitive here, but this all already works out of the box with buildkit.
just use buildx bake

[gabrieldemarmiesse]

Hi all, I made a Python client for Docker that sits on top of the Docker client binary (the one written in go). It took me several months of work. It notably has support for Docker buildx (build and bake at the moment). Actually the default call docker.build(...) uses buildkit/buildx underneath.

It's currently only available for my sponsors, but It'll be open source with an MIT licence May 1st, 2021 🙂

https://gabrieldemarmiesse.github.io/python-on-whales/

[gabrieldemarmiesse]

Hi all, in the end, making Python-on-whales pay-to-use wasn't a success. So I've open-sourced it.

It uses buildx (so with buildkit as backend) by default to build images.

It's free and on Pypi now. Have fun 😃

$ pip install python-on-whales
$ python
>>> from python_on_whales import docker

>>> my_image = docker.build("./")
[+] Building 1.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                             0.1s
 => => transferring dockerfile: 32B                                                                              0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3.7                                                    0.0s
 => [make-sources 1/7] FROM docker.io/library/python:3.7                                                         0.0s
 => [internal] load build context                                                                                0.1s
 => => transferring context: 111.37kB                                                                            0.1s
 => CACHED [make-sources 2/7] COPY requirements.txt /                                                            0.0s
 => CACHED [make-sources 3/7] RUN pip install keras-autodoc -r /requirements.txt                                 0.0s
 => CACHED [make-sources 4/7] WORKDIR /python-on-whales                                                          0.0s
 => CACHED [make-sources 5/7] COPY . .                                                                           0.0s
 => CACHED [make-sources 6/7] RUN pip install -e .                                                               0.0s
 => CACHED [make-sources 7/7] RUN cd docs && python autogen.py                                                   0.0s
 => exporting to image                                                                                           1.2s
 => => exporting layers                                                                                          1.2s
 => => writing image sha256:967755340bad79c6b222b9470729a4d95513cab51ea881c8625980034f7c7678                     0.0s

>>> output = docker.run(my_image, ["echo", "Hello world!"])

>>> print(output)
Hello world!

There is support for nearly all buildx commands and flags.

docker.buildx.build(..., cache_from="my_registry/project:cache")
````
for example for remote layer caching. There are secrets, multi-platform images 

docker.build(...., platforms=["linux/amd64", "linux/arm64"])
```
bake, making builders, ssh, pushing, etc...

Link to the documentation for docker.buildx.build(...)
Link to the github repo

I hope you like It!