Concourse: Write tests for making sure concourse was properly shipped to all destinations

Created on 16 Jun 2020 · 3Comments · Source: concourse/concourse

We want something that will make sure that all the publish jobs ran properly and that the correct outputs were published to the destinations. The destinations that we expect concourse to be shipped to is:

[x] the concourse github repository through a github release
[x] dockerhub through the concourse/concourse image
[x] bosh release to bosh.io

These tests should be written generically enough that I would be able to point it to something else and it would be able to run against them (ex. a mock github repo or a mock dockerhub account). This is because our goal is to be able to run this test against a mock of all the destinations and have this run before shipping concourse so that we can make sure that everything will be shipped correctly before actually shipping a release. https://github.com/concourse/concourse/issues/5774

Some things that we might want to test are:

the correct binaries are shipped (darwin, linux, etc binary of concourse or fly binaries)
the correct version of the binaries
when running the binary and passing a --version flag, the version outputted is correct

Source

clarafu

Most helpful comment

Added a release-validate pipeline https://github.com/concourse/ci/commit/3ee7bddfff764aab03f23e3fcd9e285f06c4c6ca
The pipeline is running at https://ci.concourse-ci.org/teams/main/pipelines/release-validate/jobs/validate/builds/9

One thing to note: the build will trigger by each individual release. In our case since https://ci.concourse-ci.org/teams/main/pipelines/release-6.3.x?group=publish runs the releasing jobs in parallel, there will be a window that not all release verions are in sync, and thus will generate some failure builds in the validate job. If we pipe the validate job after the publish group we could then have pass condition to make sure to run the validate job only after all releasing jobs have finished.