buildx in an CI environment

[bjoernhaeuser]

Hello!

First of all I want to thank everyone who contribute(s|d) this project. It looks really interesting.

I tried to use buildx in our CI environment, to utilise the cache-to/cache-from semantic to speed up our build. After trying out various scenarios, the problem always was which builder to use when, without causing conflicts. My assumption is that each builder can only do one parallel build.

Is there a way to balance builders automatically or some external tool which could do that? I understood that the k8s driver is already doing something like this? Though there is no way that the docker-container driver is doing something similar?

Thank you!

[tonistiigi]

My assumption is that each builder can only do one parallel build.

No, you should not create builders per build request. Every builder instance is designed to take in parallel requests. Builder instances just provide isolation between environments and backend selection.

Eg. this is CI for one of BuildKit's internal projects that is using buildx https://github.com/tonistiigi/fsutil/blob/master/.github/workflows/ci.yml

[bjoernhaeuser]

Hi!

Thank you for your quick answer.

When trying to use one builder per jenkins worder node, I was running into this problem:

#1 [internal] booting buildkit

#1 pulling image moby/buildkit:buildx-stable-1

#1 pulling image moby/buildkit:buildx-stable-1 1.9s done

#1 creating container buildx_buildkit_a6900bc54-4395-40f8-8c15-bf7d9663bfa40 done

#1 ERROR: Error response from daemon: Conflict. The container name "/buildx_buildkit_a6900bc54-4395-40f8-8c15-bf7d9663bfa40" is already in use by container "62af991b46d9a6cbca117a9b893dbdadcd6a8fbe6d3c6b65dda71085e6dc3293". You have to remove (or rename) that container to be able to reuse that name.

------

[tonistiigi]

Looks like your previous runs left state behind that you should clean up. Also, please try latest buildx, previous version had trouble seeding randomness so you could possibly get the same container names.

[bjoernhaeuser]

Hi!

I was able to rerun our tests with 0.4.2, but still hitting the same error. It only happens if I am starting several builds at the same time.

I think I am hitting a race condition in the bootstrap method of the docker-driver driver: https://github.com/docker/buildx/blob/v0.4.2/driver/docker-container/driver.go#L36-L42

On the first run the container inspect returns would return there is no docker container and thus would create it. Though if another build job now created the container, the create would fail with exactly the error/log I am seeing:

#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.9s done
#1 creating container buildx_buildkit_default-builder0 done
#1 ERROR: Error response from daemon: Conflict. The container name "/buildx_buildkit_default-builder0" is already in use by container "9c4f79daf2ea65c8e483537d1ec88e1fe22be210339c444e64e87bb6bc4f8704". You have to remove (or rename) that container to be able to reuse that name.
------
 > [internal] booting buildkit:
------
Error response from daemon: Conflict. The container name "/buildx_buildkit_default-builder0" is already in use by container "9c4f79daf2ea65c8e483537d1ec88e1fe22be210339c444e64e87bb6bc4f8704". You have to remove (or rename) that container to be able to reuse that name.
script returned exit code 1

Is this observation correct?

[teddy-dubal]

Hi @bjoernhaeuser , i've got the same issue , did you found a solution ?

[nigelgbanks]

The workaround I have is to start a single build that doesn't do anything to get the build container to start, afterwards I can run concurrent builds.