Buildah: Allow adding files to image with deterministic creation time

[hendrikhalkow]

Hello, we should consider building a feature into buildah that allows adding files to an image with a deterministic time stamp. If we had this, we can run the same code with the same input data twice that produces the exact same image digest. It could work like this:

buildah add --creation-time="1970-00-00 00:00:00" "${container}" "${file}"

or with the mount option:

buildah mount --creation-time="1970-00-00 00:00:00" "${image}"
dnf install --installroot  ...

or via commit that doesn't only omit timestamps, but also change all creation time stamps:

buildah commit --omit-timestamps --creation-time="1970-00-00 00:00:00" ...

As a fourth option, we this feature can be kept in an external tool like libfaketime.

The rationale behind this is that I want to build truly reproducible images that create the same image digest from the same Git commit hash in every case. Does anybody else have a need for that feature and if so, which of the four options would you prefer? Or do you have better ideas?

[rhatdan]

@nalind WDYT? I am fine with this.

[rhatdan]

Might be better to set this in buildah from and buildah bud.

buildah from --epoch-time ...
buildah bud --epoch-time

@hendrikhalkow Are you interested in working on this?

[hendrikhalkow]

Hi @rhatdan I'd love to, but I am pretty new to Go programming, so it would require quite some guidance. Maybe we achieve better results when somebody else implements it. As of now, I can help with concept work.

I like the idea setting it in buildah from or buildah bud. So let me rephrase how you think this would work:

Running the following script twice would create exactly the same image with identical hashed, assuming the installation sources have not changed.

ctr="$( buildah from --epoch-time '1970-01-01 00:00:00Z' scratch )"
mnt="$( buildah mount "${ctr}" )"
dnf install --installroot="${mnt}" ...

# remove some files that make our files non-reproduceable
pushd "${mnt}"
rm -rf ./etc/machine-id ./lib/.build-id ...
popd

# what would the creation time of these files be?
ls -la "${mnt}"

# If we have have `from --epoch-time, do we still need `commit --omit-timestamps`?
buildah commit --omit-timestamps "${ctr}" image

At which point would the actual magic happen? When the files are written into the mounted directory or when the container is committed into an image?

[rhatdan]

I talked to @nalind About this a while ago, but we did not go forward on it.
@ashley-cui Could you take a look at this. Talk to @nalind about it.

[rhatdan]

       --omit-timestamp bool-value

       Set  the  create timestamp to epoch 0 to allow for deterministic builds (defaults to false).  By default, the created timestamp is changed and written into the image manifest with every commit, causing the image's sha256 hash
       to be different even if the sources are exactly the same otherwise.  When --omit-timestamp is set to true, the created timestamp is always set to the epoch and therefore not changed, allowing the image's sha256 to remain  the
       same.

We have this feature.

[hendrikhalkow]

Hi @ashley-cui, @rhatdan, --omit-timestamp is about the image itself, but not the file within the image. The problem I am trying to solve is to create a unique mapping from the Git commit of my source code to the resulting image for which I need to create the exact same image with every build run for the same commit.

Imagine during the build, I run dnf install whatever. Or I have an external build process that is able to produce exactly the same artifact with every run, but with different creation time stamp. The simplest case is this:

#!/bin/sh
ctr="$( buildah from registry.access.redhat.com/ubi8/ubi-minimal )"
mnt="$( buildah mount "${ctr}" )"
# Simulate our artifact
touch "${mnt}/a"
buildah commit --omit-timestamp "${ctr}" "${1}"

Run this script twice with like buildah unshare ./build.sh i0 and buildah unshare ./build.sh i1.
Now how do I ensure that the resulting image is still the same? The resulting images differ with every run although the content of the files are exactly equal. The reason is just that the timestamp is different.

I worked around this with a conditional push in my build pipeline and I compare my images like this:

mnt0="$( buildah mount "${ctr0}" )"
mnt1="$( buildah mount "${ctr1}" )"
diff -r "${mnt0}" "${mnt1}"

I also tried to solve it with the faketime command which worked on Fedora but not on RHEL because the package isn't available there and I wanted to stay with official packages.

So I thought it might be a great idea when buildah can do this, e.g. by passing the desired epoch time for the files into buildah commit or by extending the --omit-timestamp parameter to the files.

[rhatdan]

I believe we can do this now, reopen if I am mistaken.

[hendrikhalkow]

So how do we do this?

[rhatdan]

Oops it looks like it is only in buildah-commit

man buildah commit
...

       --omit-timestamp bool-value

       Set  the create timestamp to epoch 0 to allow for deterministic builds (defaults to false).  By default, the created timestamp is changed and writ‐
       ten into the image manifest with every commit, causing the image's sha256 hash to be different even if the sources are exactly the same  otherwise.
       When  --omit-timestamp  is  set to true, the created timestamp is always set to the epoch and therefore not changed, allowing the image's sha256 to
       remain the same.

I will add support to buildah bud.

[rhatdan]

After we have support for this in buildah bud, the next step would be to get support into the storage driver to record all times as epoch 0. Then all files in the layers created could be created as epoch 0.

[rhatdan]

This one has been fixed.