Kaniko: Question regarding completeness of Dockerfile tests

[synergiator]

Hi, Kaniko team,

this is not a bug!

I just have a question regarding completeness of Dockerfile integration tests.

How complete are these tests and how do you know?
https://github.com/GoogleContainerTools/kaniko/tree/master/integration/dockerfiles

Something like ACID test?

http://acid3.acidtests.org/

[cvgw]

Hi,

First, it would be helpful to know what you mean by completeness. It could have a variety of different meanings IMO

Does kaniko test all of possible execution paths in the code? No
Does kaniko test all possible behaviors of docker build? No (probably not a goal of the project)
Does kaniko test all possible Dockerfile combinations? No (also probably not a goal of the project)

At least since I've started working on the project we've added new cases to the integration tests as they have come up in bugs or we added new features.

[synergiator]

@cvgw thank you! from the Docker/moby people, I've found there is great repository by @jessfraz with really great number of Dockerfile. Possible, altogether they exploit many possible and meaningful Dockerfile combinations.

https://github.com/jessfraz/dockerfiles

[cvgw]

One thing to consider is that the integration tests already take a significant amount of time to execute. That is already a burden on the developer experience. Significantly increasing the number of tests run on each commit or PR may not be a good idea. Perhaps there could be a large suite of tests run on release though

[synergiator]

yes something like that - before coming to that though, IMHO already a single run could provide some insights (of course for those Dockerfiles that also link intact third-party resources)

[cvgw]

That's true, a single run might reveal some obvious areas for improvement.

Another thing to note is that we don't expect Kaniko to ever be 100% feature complete with respect to docker build. Doing a test run of those Dockerfiles would also imply interpreting the results to identify cases where we do want parity with docker build

[cvgw]

cc @samos123 @tejal29 I think this could be a good idea

[tejal29]

@cvgw I agree a single run can provide us a lot of insights and we can look in to this.
However, i would always prioritize issues filed on github over disparities found from these tests.

I would also say, we do one before every milestone release like v1.0.0 etc instead of every release.
We could do one run now and measure this as baseline compatibility score.
The next one will be before our next major milestone release.

@1605200517, Kaniko has an easy way to set up a git buildcontext or a GCS buildcontext.
Do you want modify our integration_test.go and give it a try?
Sample Test here https://github.com/GoogleContainerTools/kaniko/blob/master/integration/integration_test.go#L185

[synergiator]

@tejal29 love the container diff feature! did not know about that. However, shouldn't we say image-diff?

I can't estimate the required effort for somebody experienced with Golang and the Docker development ecosystem and I do not yet see the solution you might have in mind.

Let me assess a little bit what we are going to do here. Please feel free to rewrite it to what seems best to you.

Goal

For specific Kaniko and Docker daemon versions tuple (K,D), we want to have some score S computed by a function using a reference test data set D. S-> f(K,D,S) If there are no differences i.e. S=0 that's a perfect optimum. How exactly we compute the score we can consider later.

Design

So for each test sample (Dockerfile project) s in the test set S we want to have the following computation:

• Can we build the sample both with Kaniko and Docker daemon?
• Are resulting containers (binary) identical?
• Additionally, the test system has to be easily respawned for given (D,K).

Solution A

Use the easy way to setup a buildcontext starting from the integration_test.go and collect data somehow .. (please share your further thoughts if possible)

Solution B

(this is what I was thinking of initially, do not know how much different is that from A)

• Setup a CI pipeline automation build environment at GitLab using Python client toolkit for GitLab API python-gitlab.
• Implement a master which iterates through the data set. For each sample, it generates a worker pipeline and aggregates results (can be also run in parallel).
• Each worker pipeline runs a parallel build for Docker daemon as baseline reference and Kaniko and reports results to the master pipeline.
• Optionally, the worker pipeline also computes container diff data, using a modification of integration_test.go towards a linter-like tool or other tooling like container-diff or maybe just export images to files and apply md5sum?

===========
What do you think?

[tejal29]

This all sounds great @1605200517.
We also have @samos123 interested in working on this.

I would leave it upto you two who wants to implement this.

Thanks
Tejal

[synergiator]

@samos123 further thoughts?

[cvgw]

@1605200517 for fun I threw together this PR https://github.com/GoogleContainerTools/kaniko/pull/1042 where I copied over all of the dockerfiles from jessfraz/dockerfiles which don't require any context to build.

I was unable to run the suite on my local machine as I ran out of disk space (I'm assuming to build all the images). Our CI env Kokoro also failed for the same reason. TravisCI also failed due to the length of the build (max 50 minutes for free tier).

So, it appears there are some technical challenges to be solved in order to run such a large test suite. It isn't a top priority for the team as we have some more urgent items at the moment, but we will keep poking at this and you are more than welcome to try it out as well.