Cli: [epic] documentation on BuildKit garbage-collect policies is missing

[thaJeztah]

Pull request https://github.com/moby/moby/pull/37846 and https://github.com/docker/cli/pull/1327 added support for configuring BuildKit's garbage-collection policies in the daemon configuration and when using docker builder prune.

Both of these options are currently undocumented, as well as (bash) completion scripts missing.
We should add proper documentation for this feature, accompanied by some examples.

Related PR's and issues:

• https://github.com/docker/cli/pull/1327 build: add options to builder prune
• https://github.com/moby/moby/pull/37846 builder: use buildkit's GC for build cache
  
  
  
  • https://github.com/moby/moby/issues/39601 Example Buildkit GC configuration results in daemon error
  
  
  • https://github.com/moby/moby/pull/39979 daemon/config: fix filter type in BuildKit GC config
  
  

Things to resolve:

• [ ] update daemon configuration documentation (in dockerd reference) with (syntax) examples
• [ ] given that the dockerd configuration has grown to become a bit unwieldy, we may want to add a dedicated section / page elsewhere in the documentation to describe this feature in more depth (could be in the documentation repository: https://github.com/docker/docker.github.io, but also should be referenced from the builder.md file
• [ ] Add a "builder_prune.md" file in the documentation directory, containing an "examples" section for the docker builder prune reference docs to
  
  
  
  • include the --filter, --keep-storage, -f / --force, and -a / --all options that were added in https://github.com/docker/cli/pull/1327)
  
  
  • show some basic example uses
  
  
  • add a link to the "in-depth" documentation about build cache garbage collection
  
  
• [ ] update the (bash) completion scripts in this repository to include the --filter, --keep-storage, -f / --force, and -a / --all options that were added in https://github.com/docker/cli/pull/1327)
• [ ] backport changes to the 19.03 branch, given that features are already were included in that release

[thaJeztah]

@tiborvass @tonistiigi could you either work on this, or provide input if someone wants to work on this?

[thaJeztah]

/cc @chris-crone @nebuk89 (for planning, if needed), @usha-mandya (FYI)

[adeniyistephen]

@tiborvass @tonistiigi could you either work on this, or provide input if someone wants to work on this?

Hi, i would love to work on this @tiborvass @tonistiigi as input been provided for it? @thaJeztah

[thaJeztah]

ping @tiborvass @tonistiigi could one of you help providing the details?

[adeniyistephen]

Hi, @thaJeztah I haven't received any details yet from @tiborvass and @tonistiigi, if you don't mind, where is the best method of communication is?

[thaJeztah]

I tried asking them on our internal Slack (but timezones are sometimes a pain); if you're on the Docker Community slack, you could try the #buildkit channel

[adeniyistephen]

Ok, I'm sorry but can I private ping them for better communication or just throw it open on the channel? Thanks.

[thaJeztah]

Feel free to post it in the channel there!

[tiborvass]

Thank you @adeniyistephen for offering your help on this!

Let me give you a brain dump of the situation.

First, when the BuildKit garbage collector is enabled it runs periodically in the background and follows an ordered list of policies. Each policy is a prune operation so the order matters. But pruning can also be done manually with docker builder prune and when testing it, it's probably easier to run things manually first.

Therefore, what needs to be done here is:

• documenting and testing builder prune operations with various inputs that can be also used as examples in docs
• documenting and testing BuildKit GC
  
  
  
  • how GC works (GC policy = prune operation)
  
  
  • GC config in daemon config
  
  
  • default policies
    
    
    
    
    
    • which depends on a configurable DefaultKeepStorage (the easiest way to configure GC without learning all the details of policies)
    
    
    
    • default for DefaultKeepStorage is approximately 10% of free disk space: https://github.com/moby/moby/blob/fb7883e980f6a4b5c5d40f673dd539499c328885/builder/builder-next/worker/gc_unix.go#L15
    
    
    
  
  

Most people should simply enable GC and set the desired DefaultKeepStorage in the GC config.

Below I'll try to describe the detailed part of how pruning works.

The prune operation deletes build cache records and the main logic is in the pruneOnce() function at https://github.com/moby/buildkit/blob/d168063ee5b73510b07d80e8eef1dfd86adcb853/cache/manager.go#L606
Note that pruneOnce() calls a recursive prune() method.

Build cache records can have the following attributes:

• id: ID of cache record
• parent: ID of parent of cache record
• type: type (see below)
• description: a metadata string
• inuse: if true, it cannot be reclaimed (aka deleted)
• shared: if true, the same content is also referenced by an image in docker images -a. By default record will not be removed as it is impossible to know whether the user also intended to remove the referencing image (be it untagged or not).
• private: not shared

The type attribute can be one of the following:

• regular: intermediate snapshots
• source.local: cache of local context enabling fast incremental uploads
• exec.cachemount: cache type for RUN --mount=type=cache,...
• frontend: cache for BuildKit frontend images (# syntax = user/somefrontendimage)
• internal: cache for helper tools internal to BuildKit

These build cache records can be inspected with docker system df -v (but shows more than just build cache records) or with buildx via buildx du --verbose with a bit different output.

The prune operation's main inputs are:

• filter: list of AND-ed filters each containing one of the cache record attributes above
• all: if true, will delete cache records that are either marked as shared, as internal or as a frontend
• keepDuration: duration between now and the cutoff time after which cache records should be kept
• keepBytes: threshold of cache disk usage, above which the prune operation starts deleting cache records until the threshold is no longer exceeded. Below that threshold nothing happens. If 0, then all cache records matching the other criteria are deleted.

The builder prune/buildx prune CLI maps these inputs a bit differently than BuildKit: keepBytes is --keep-storage, all is --all, and filter is a set of --filter flags but, in addition to the cache attributes detailed above, it can also contain an until key which maps to keepDuration. Implementation of this mapping is in toBuildkitPruneInfo().

Hope this helps.

[thaJeztah]

I see there's some discussion in https://github.com/moby/buildkit/issues/1385 as well that may be relevant

[adeniyistephen]

@tiborvass @thaJeztah Thanks, working on it. will ping you if i need any help.