Buildkit: always display image hashes

[TrentonAdams]

It's tough to debug docker building when I can't just get into the previously successful intermediate build image and run the next command manually...

docker run -it --rm hash_id bash
# execute the next RUN line here manually.

I would therefore argue that image hashes should always display, just like they do in the current docker.

[AkihiroSuda]

There is no image hash for an intermediate cache because it is not exported to the docker image store.

maybe we should have buildctl build --debugger image=busybox that invokes an interactive busybox shell image with bind-mount for the failed build rootfs on failure.

We can also consider having buildctl build --output type=image,name=foo,output-on-failure=true but it might require some changes on the solver design.

[TrentonAdams]

There is no image hash for an intermediate cache because it is not exported to the docker image store.

Intermediate build images are indeed cached in the local docker image store, I use them all the time to find out what happened at a particular step. Every single docker line creates a permanent cached image, and can be ran using docker run --rm --it hash sh to diagnose build issues at that step, or rather the step prior to when the error occurred. If this was not the case, builds would take a long time every time.

I will think about your other comments when I get back to work. But, one thought is to drop to a shell at a particular step, like build --debugstep=7. But, in either case, having the cached image hash would allow one to jump into any step and see what went wrong.

[TrentonAdams]

Oh, and printing that small hash on the screen every time improves "usability", in that people don't have to know some special parameter to debug build images. If it were the left most item, it would also then be in the same spot every time, the same size every time (resulting in a more columnar output), and therefore improve readability of the output as well.

[niklasbuechner]

I am also trying to debug a docker build and expected all intermediary builds to be exported as an image, since that was the previous behaviour.

@AkihiroSuda Is there currently a way to look into the filesystem after the last successful build step?

[TrentonAdams]

@niklasbuechner I don't expect build kit is changing the behaviour of the building of intermediate cached images. I expect it's just not displaying the hash ids. What I don't know is if there's a way of listing them after the fact, for workaround purposes.

[dko-slapdash]

Yeah, setting DOCKER_BUILDKIT=1 disimproves debug experience a lot. Without DOCKER_BUILDKIT, it was possible to run a container with a layer right before the failed step, and with DOCKER_BUILDKIT it's not possible anymore since it just doesn't show image cached IDs.

This looks like a clear regression...

[dko-slapdash]

@AkihiroSuda, could you please advise us, is there anything changes regarding debugging of failed builds while DOCKER_BUILDKIT=1?

How do you debug such cases in your own practice?

I.e. if the build fails, I'd like to somehow spawn the container with the layer before the failed step and bash there. Previously it was possible with docker run -it --rm hash_id bash, but now DOCKER_BUILDKIT=1 doesn't show the layer hash...

[AkihiroSuda]

My own (bad) practice is RUN sleep infinity and nsenter

[TrentonAdams]

@dko-slapdash I do not think you can debug them. It's a serious failing of the build kit feature.

[dko-slapdash]

My own (bad) practice is RUN sleep infinity and

@AkihiroSuda is there some alternative approach or there is none? If not, how much is it painful in practice, are you satisfied with it in general or not quite? (Just curious.)

[AkihiroSuda]

Not satisfied, but can't come up with other approach that works with existing releases

[silviot]

My own (bad) practice is RUN sleep infinity and nsenter

This simple pointer saved my day. I knew about nsenter but forgot about it. I recommend to try this technique to anyone stuck that came here for help. True, it's not sexy. But it works.

[nottrobin]

I don't understand this technique - I added RUN sleep infinity, but how do I use nsenter to find the running image?

[AkihiroSuda]

sudo nsenter -a -t $PID_OF_SLEEP sh

[nottrobin]

And how do I get the $PID_OF_SLEEP?

[AkihiroSuda]

ps auxw

[nottrobin]

Oh I see, just find it in the standard running processes. Sounds good. Thanks.

[Leletir]

Is there any way to export already built layers to the output specified for debugging ?

[AkihiroSuda]

Not yet, contribution is wanted.

[TrentonAdams]

@AkihiroSuda Hmmm, I wonder how tough it is to build on my MacOS. Do I need to build all of docker, or just buildkit, to try out making a contribution?

[AkihiroSuda]

Hmmm, I wonder how tough it is to build on my MacOS.

Should be quite easy if you have Docker for Mac.
make images will create moby/buildkit:local image on Docker.
See https://github.com/moby/buildkit#containerizing-buildkit for the usage of Dockerized BuildKit.
The buildctl client can be compiled on Mac by just running go build ./cmd/buildctl.

Do I need to build all of docker, or just buildkit, to try out making a contribution?

Just BuildKit.

[TrentonAdams]

Sweet, thanks for the quickstart. I don't know go, but I'm just going to go! Not promising anything. 😛

[maximveksler]

My bad practice is notepad debugging, I comment out from the dockerfile everything after the failing step. Then I build. Then I apply steps manually and add them as RUN commands to the docker file.

[TrentonAdams]

My bad practice is notepad debugging, I comment out from the dockerfile everything after the failing step. Then I build. Then I apply steps manually and add them as RUN commands to the docker file.

Ouch, that sounds very painful for certain docker images that take a long time. :(

[javabrett]

Could we get a somewhat official answer on which of the following are true, in the context of building with BuildKit:

There is no image hash for an intermediate cache because it is not exported to the docker image store.

Intermediate build images are indeed cached in the local docker image store, I use them all the time to find out what happened at a particular step.

• It is currently impossible to a) produce a hash of an intermediate image in a BuildKit build and/or b) reference/run that intermediate image even if you held the hash.

[tonistiigi]

Could we get a somewhat official answer on which of the following are true, in the context of building with BuildKit:

"There is no image hash for an intermediate cache because it is not exported to the docker image store." is correct. Never doubt @AkihiroSuda

Closing in favor of #1472

[TrentonAdams]

@tonistiigi That's not true, intermediate builds are cached, and available to "docker run". I've used them countless times to figure out what's wrong. In fact, docker wouldn't even work if they weren't, because docker re-uses them with every build. If they were not cached, docker would have to rebuild your image repeatedly, every time you run docker build

[AkihiroSuda]

@TrentonAdams That's non-BuildKit mode

[TrentonAdams]

lol, I was just going to ask that question, you beat me to it.

[TrentonAdams]

So, how does docker cache the image then, cause if it didn't, you'd have to rebuild every time. Is that a special cache of buildkit then?

[trinitronx]

Perhaps this Issue title "_Always display image hashes_" is a bit misguided. Regardless of whether intermediate filesystem layer content hashes are output, there is an implicit usability feature here that many Docker users have stumbled upon. Due to not displaying (or computing) intermediate hashes, BuildKit builds are harder to debug, whereas normal Docker layer IDs provided a helpful feature for developers to use that was helpful to debug a failed build.

So, how does docker cache the image then, cause if it didn't, you'd have to rebuild every time. Is that a special cache of buildkit then?

Good question! I'm not 100% sure, but given that buildkit/solver design docs state:

The implementation of the solver is quite complicated, mostly because it is supposed to be performant with snapshot-based storage layer and distribution model using layer tarballs. It is expected that calculating the content based checksum of snapshots between every operation or after every command execution is too slow for common use cases and needs to be postponed to when it is likely to have a meaningful impact. Ideally, the user shouldn't realize that these optimizations are taking place and just get intuitive caching. It is also hoped that if some implementations can provide better cache capabilities, the solver would take advantage of that without requiring significant modification.

It seems like it tries to avoid computing checksums too often (which speeds up builds). However, this could also be the reason they say that intermediate layers are not stored in the usual Docker image store.

Conceivably intermediate layers are stored _somewhere_, and could be exported (given that buildkit/solver says they are exportable as tarballs). It's obviously able to cache layers internally somewhere right now, otherwise it would be impossible to store and recall those layers to speed up later builds. The question then is: How can the user's debugging experience be made easier from a usability standpoint, given the current design & goals to speed up builds overall.

The "user story" way to state this feature might be:

As a developer, I need a way to gain a debug shell inside a failed Docker build layer, so that I can introspect the target OS filesystem state and fix my build faster.

[tonistiigi]

As a developer, I need a way to gain a debug shell inside a failed Docker build layer, so that I can introspect the target OS filesystem state and fix my build faster.

See this linked issue