buildah to use cache, layers for bash builds as well

[zvonkok]

Description
Buildah can use cache/layers for building with buildah bud (#767). It would be beneficial if buildah could use a cache/layers for bash builds as well. I am using bash builds extensively as there are more convenient/useful for all of my use-cases.

Dan mentioned that I could use commits in-between and base off the next step of them.
Nice idea but I think this will clutter up the script and the signal-to-noise ratio would be rather high.

BTW using bash scripts to build the images is/was a great idea, this way I can use everything, really everything as a tool that I can install on my Linux and I am not tied to a DSL that I cannot extend or is limited.

Describe the results you received:
Executing the buildah script a second time, buildah executes each step again.

Describe the results you expected:
Skip steps that did not change compared to the last invocation of the script.

Output of rpm -q buildah or apt list buildah:

buildah-1.4-3.gitc8ed967.el7.x86_64

Output of buildah version:

Version:         1.5-dev
Go Version:      go1.10.2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-alpha1
Git Commit:      c8ed967
Built:           Tue Nov  6 14:01:57 2018
OS/Arch:         linux/amd64

[zvonkok]

/cc @jeremyeder

[TomasTomecek]

This is an awesome suggestion! I wrote a tool which utilizes buildah for building images (and ansible as a frontend) and I had to solve the same issue: caching layers.

It would be awesome if buildah had an API for caching. On the other hand, it will be really tricky because you would need to tell buildah up front what the script is and how are you changing the filesystem and then buildah would need to figure out if there is a matching entry in the cache.

[rhatdan]

@umohnani8 What do you think? Would this even be possible.
Should we keep all of the buildah commands, we could record the fact that we have already executed them. But anything that modified the content outside of the script we would fail.

We might be able to figure out that

buildah from fedora
buildah run
buildah config
buildah run
buildah mount
yum --installroot We would definitely fail here.
buildah config.

Also how would we know that two buildah from fedora calls were related?

[arthurbarr]

I guess that any of those buildah run commands could have a different effect if the bash script has (say) copied some different files into the mounted filesystems. I guess that docker build gets round this by having a fixed "context" directory, where all the source is located. They can then do a hash of that source, or look at time stamps (not sure which they do). So I wonder if buildah could have something which helped users hash and compare particular source files/directories, and skip the step if it hasn't changed. For example:

buildah --skip-if-unchanged=./src:./*.go run blah

[rhatdan]

@umohnani8 Any update on this issue?

[rhatdan]

@QiWang19 Can you look into this?

[gcs278]

I'm very interested in this concept, great idea

[rhatdan]

@zvonkok and @gcs278 PRs would make this move along faster...

[cben]

unbaked thought: could a design like redo be a good fit?

A core idea of redo different from make is that dependencies of a build step are not _declared_ in up front; they are _recorded_ by each target's build script (which could be in any language!) as it does it work, by shelling out to redo-ifchange dependency ... and/or redo-ifcreate dependency .... (And there is an implicit ifchange on the build script itself.)
The system is recursive. When a "do" script executes redo-ifchange, the given dependencies are themselves processed. (So a first run always runs all steps, later runs are partailly cached.)

There are multiple implementations of redo, recording dependencies in different ways; each puts its own redo-ifchange / redo-ifcreate into PATH before running build scripts.

• It seems redo is still focused on producing _files_, with names, so may not fit here as-is.

• Not sure buildah wants recursion.
  Dockerfile caching is based on anonymous intermediates in a script-like sequence.
  Conceptually it's a chain: [[[_step 1_] <--dependency-- _step 2_] <--dependency-- _step 3_] ...
  but most of the time, writing it as a single flat script is more fun.

But maybe a contract like "I'm doing whatever I want with buildah mount, but I'm gonna run build ifchange ... to tell you what I depended on" is a good idea?

So if we want a flat script, how would a script with arbitrary bash steps skip those steps? Maybe a 2-directional contract, with a buildah command reporting info on cache hit/miss?

BTW, is it technically possible for buildah to "fast forward" the same container from current state to a an image found in cache? Or would we need container=$(buildah ...) after every step?

[rhatdan]

@cben interested in opening a PR for this?

[dsonck92]

I was also looking into this cache idea and actually spent some time playing about with redo (which I like enough to port my golang based projects to use it to generate things, indeed much more natural than make).

That said, I think for buildah there are 2 approaches:

• Actually suggest redo and write a tutorial on some examples of how you would go about it
• Implement some sort of state tracking like redo.

Using Redo

The short time I used redo I understand the following:

• You have a file you want to build, for which you create a .do file
• The do file explains both how to build it and when to rebuild it
• redo does expect that this file is going to exist and wishes you to fill it by writing to $3

With this information, you can split a buildah script to different discrete steps that should be built as a unit. Considering it's impossible build halfway unless history was kept or it was committed, it's imperative that a build script will end with a commit. So a possibility:

NAME=stage-name # could be random but keep in mind the lingering images
OUTPUT=$3
CONTAINER=$(buildah from fedora)
# do buildah steps
buildah commit $CONTAINER $NAME
echo $NAME > $OUTPUT

This, in itself, is a do file, though it could also be written generically:

• default.buildah.sh.do:
  bash # Call the script and tell it to output its result to "$3" ./"$1$2" "$3"
  and the accompanying build instructions:
• initial-stage.buildah.sh:
  bash NAME=initial-stage CONTAINER=$(buildah from fedora) # do buildah steps buildah commit $CONTAINER $NAME if [ $# -gt 0 ] then echo $NAME > $1 fi
• second-stage.buildah.sh:
  bash NAME=second-stage redo-ifchange initial-stage buildah from $(cat initial-stage) # do further buildah steps buildah commit $CONTAINER $NAME echo $NAME > $1
  A stage leaves behind a file containing the output it made, in buildah terms an image it can reuse. A cat of this output specifies a valid from name.

Built in support

Instead of relying on an external tool (although a tool that has many implementations and is not difficult to obtain), buildah could also attempt to do this kind of tracking itself. For which I propose the following idea:

• To speed up builds, buildah can be instructed to use a cache file that tracks information on build steps.
  
  
  
  • This cache file can be specified by the user.
  
  
  • The cache file records for each instruction what its previous instruction was
  
  
  • The cache file is also accompanied by pointer files
  
  
  • Pointer files track which instructions have been done from a cache file
  
  
  • Each working container has its own pointer file
  
  
  • Some instructions cannot be cached and terminate a recipe
    
    Most notably any instruction that cannot tell what happened like mount
  
  
  • --add-history and commit additionally mention their container name
  
  

The process of how it works:

1. When a cache file has been set, when buildah from is executed, it will create a pointer file pointing to the first instruction matching the from. This first instruction is considered a "saved point"
2. If this first instruction matches the cache file, it will not do anything. If it doesn't match it will perform its action and write this instruction to the cachefile
3. When a next buildah invocation is made, it will seek a next instruction pointing to the current one
   
   
   
   • If there is an instruction that matches, it will do nothing
   
   
   • If there is not a matching instruction:
   
   
   • it will go back in the cachefile till a "saved point"
   
   
   • It will load up this point and replay the cachefile till the point diverged
   
   
   • it writes down the new arguments and performs its own action
   
   
4. If --add-history is used, the generated image will be mentioned in the cache file, this is another "saved point"
5. This process loops from 3 till there is a cache incompatible instruction:
   
   
   
   • At this point, the cache pointer is removed to indicate the container is in a non-cached stage.
   
   
   • If any further commands are executed, since the pointer does not exist (which would have been created by from) it will directly apply it without any cache information
   
   

So as an example:

export CACHEFILE=.mybuild
buildah from fedora # saves its details (or hash of these details) and fedora. Also creates .mybuild.fedora-working-container
buildah run $CONTAINER -- yum install nginx # saves its details and advances the pointer
buildah copy $CONTAINER www /var/www # saves its details (including hash of the content it copied) and advances pointer
buildah commit end-image # saves its details and also end-image

Now the same thing is executed again:

export CACHEFILE=.mybuild
buildah from fedora # recognizes the start of .mybuild
buildah run $CONTAINER -- yum install nginx # recognizes the command hasn't changed and moves pointer
buildah copy $CONTAINER www/ /var/www # if www hasn't changed, recognizes and moves pointer
buildah commit second-image # recognizes end-image and recommits it as second-image

It must be noted, if www changes, it will rebuild all the instructions starting with from fedora. In order to prevent this:

export CACHEFILE=.mybuild
buildah from fedora
buildah run --add-history $CONTAINER -- yum install nginx # now becomes a saved point
buildah copy $CONTAINER www/ /var/www # now if this changes, it can continue from after yum install nginx
buildah commit third-image

With this system, even if instructions diverge, this creates multiple alternative paths that can still resolve to a saved image.

[dsonck92]

Note that the redo script could possibly be improved by extracting the first buildah from instruction written in it and applying some logic to identify whether it's a local build step or a global one. Wait, I guess it's possible to detect the specific cat

[dsonck92]

It might also be noted that for the buildah native method, there might be options that also cause 5 to happen. I have not spent much time with buildah to know which situations would cancel a cache run. But I think this would give the closest experience of the cache given the limitations of not tracking history.

If the user wants to speed up its rebuild, they should also apply strategic uses of --add-history before anything that might cause diverging points so it doesn't have to backtrack too much steps in order to get to a known state.

[dsonck92]

Lastly, if this addition to buildah is something that is wanted @rhatdan I could attempt a PR for it. It might take a while though.

[rhatdan]

Sounds good, I don't know when we can get someone else to look at it.

[dsonck92]

I will then try to familiarize myself with the buildah code and see if my idea is somewhat feasible. I'm not sure when I can work on it but as I'm migrating away from docker and dockerfiles (appreciating the daemonless, rootless abilities of buildah/podman and the ability to use system tools during image steps of buildah and mount dirs (e.g. for extracting test results)), some form of buildah cache would be beneficial. While builds in general seem to be faster, nothing beats a "well I can skip this 10 minute build process as no code changed". I guess it's a bit of a bad example as that could also be fixed by keeping a dirty build directory around and others fixed by a cache mount, but for sake of repeatability, the cache can help without resorting to mounts and dirty build dirs that have their share of potential issues.

I predict it will take me some time and I made some assumptions that I might run against when trying to implement, but I'll keep this issue for any news I have on that topic.

[rhatdan]

@dsonck92 Did you ever get a chance to work on this?

[dsonck92]

Not yet as I'm pretty occupied with other things but I'm considering picking this up in December.

[kwshi]

I've tried playing around a little with the redo idea, and one major drawback to that approach is that, because it relies on buildah commit, its disk usage scales quite poorly. Consider, as a rough example, a build that consists of steps A, B, C, each of which adds 1GB of stuff to the container. Using a built-in cache:

• Step A creates a container of size 1GB, total cache use is 1GB;
• Step B creates a container of size 2GB, but since it's layered on top of A, total cache use is also 2GB;
• Step C creates a container of size 3GB, total cache is 3GB.

Meanwhile, using a redo/make-like mechanism to reuse previously committed steps:

• Step A creates a container of size 1GB and commits it, total size of image cache is 1GB;
• Step B instantiates a new container from image A and creates a container of size 2GB, but AFAIK the buildah image cache is _not_ layered, so each individual image is as large as its full contents, so total image cache is 1GB (A) + 2GB (B) = 3GB;
• Step C instantiates a container from image B, creates a container of size 3GB, total image cache use is (1 + 2 + 3) GB = 6GB.

My belief that buildah's image cache does not reuse layers arises from some experimentation I did myself with this approach (where I found that simply cloning an image, e.g. buildah commit "$(buildah from <image>)" still boosted my disk usage (as reported by df -h) by the size of <image>, despite no changes/layers being added). I don't fully understand how containers/image storage really works or is supposed to work, so maybe I'm doing something wrong here, but as far as I can tell, this means the redo approach is really not scalable, as disk usage scales "quadratically" with the layers, even if there is a performance time-save.

My understanding is that Docker's cache does not suffer this issue because its cache _does_ store things in a layered-manner, unlike buildah commit. I may be wrong about this, though.

[jsirex]

I recommend you to not over-enginering here. Ask yourself why do you really need a cache or layer?
All yours' volatile part is actually what you want to always build as a single level.
And all your cached/layered part is always something you want to use as base image.

[dsonck92]

Well, meanwhile I changed my personal build system quite differently. I'm utilizing gitlabs (and probably other CI's have similar features) ability to run build steps in a container directly. This gave the ability to extract the different stages and reuse those later on, making the buildah step essentially a "from, add, commit". This lowered the complexity of builds considerably and essentially runs the build steps through podman on kubernetes, which adds some parallelism. Too bad it doesn't (yet) have a generic container runner and is tied to either kubernetes or docker.

Now I know that this essentially evades the problem but I find it relatively elegant. I did discuss this at work, which basically shot it down with "but now it's not a single dockerfile that you can just execute" (or buildah dependent shell script), but then again, you can put the stages in a shellscript and execute those after eachother in the gitlab file and separate buildah steps. Though that will limit its usability somewhat. Complex matter