Tools: Build container for releases in GitHub actions

[ggabernet]

We got the issue with @drpatelh and @MaxUlysse when running aws tests that the container tag was already bumped to the tagged release version, but it had not been built yet. It's not that easy to tag the container for the AWS tests, as we do for GitHub actions CI tests.

This could be solved by adding a GitHub actions workflow that builds the container before a release, already with the release tag. nf-core bump-version should also bump this version too then.

So the TODOs would be:

• Adding a new GitHub actions workflow to build a release container on a PR to master
• Adding the tag of the container in that workflow to bump-version so it is updated when bumping versions

[MaxUlysse]

I guess we can even think of a GitHub action like they have on bioconda, with a do release 2.0 which bump version and build the container

[apeltzer]

There is even actions for that available, e.g. for building docker containers and uploading them with a shared token to DockerHub and/or keeping them in the Github registry...?

should be super easy to add: https://github.com/marketplace/actions/build-tag-publish-docker

[apeltzer]

Ok maybe one question before, I don't really get the full problem maybe:

Is it more or less the same problem that we have with other tests also being triggered automatically and then using the dev container on release instead of the master container as its typically not built completely when these tests are triggered?

We could circumvent this by enforcing the tests on master to only run after a container has been built and uploaded successfully to either DockerHub or Github Docker Registry ...? That would then require this to be updated and could be done relatively easily?

Not sure I'm getting the issue mentioned above ....

[ewels]

Side idea - if we’re adding this then we could also skip the build steps on dev if environment.yml hasn’t changed (should do it anyway on release / push to master etc)

[ggabernet]

@apeltzer, yes it is the same problem. For the CI tests, it is possible to just tag the container with docker tag, but that is not possible for the AWS tests (that I know of).

If we want to do tests _before_ a release, but after pushing to master (and therefore the container tag is already the release tag), this cannot be achieved. We would need to build the container with the release tag, before publishing a release.

And as @ewels mentioned, this way we can also skip the build steps, etc. and have a bit more control on when to trigger a container build.

[apeltzer]

Sounds all great and good - wasn't clear to me why the behavior is different on AWS, but that makes sense now :-)

[ggabernet]

yes, I mean there might be still a way of making it work, but building the container with a GitHub actions sounds like it would be useful for other purposes too, like having the release container available right away when releasing.

[ewels]

The only downside of this is that the release-tagged container will be pushed multiple times, as there are several commits between bumping the version number and actually pushing the release. But although confusing and kind of _technically_ not great for reproducibility etc (it's not the _exact_ same image in the AWS test as it is after the release), I'm not sure that there are any real practical downsides of this.

[ggabernet]

From our slack discussion, so it does not get lost, a temptative release workflow would look like:

1. Run AWS tests (+ wait for correct completion)
2. Bump versions + build tagged container
3. Make release
4. Run AWS tests again on released pipeline + publish results

The disadvantage is that we run the AWS tests twice. The advantage is that we don't run into troubles of versions being bumped and the container not existing yet for the first AWS test.

[MaxUlysse]

The only downside of this is that the release-tagged container will be pushed multiple times, as there are several commits between bumping the version number and actually pushing the release. But although confusing and kind of _technically_ not great for reproducibility etc (it's not the _exact_ same image in the AWS test as it is after the release), I'm not sure that there are any real practical downsides of this.

I don't see why we can't use the pre-built release-tagged container as the actual container for the release (just need to disable automatic built on docker).

[ewels]

I suppose we could.. Currently each docker image build is linked to a GitHub commit, but I guess if we disable the automatic build then they are essentially stand-alone. So yeah, we could just skip the build on release.

A bit tricky to automate though.. For example, it's not uncommon to have several merges to master before a release. But maybe that's ok.

[ggabernet]

It would be cool to have a release bot that we can call to do the release, so we still have manual control and a final check on when to release

[ewels]

I'm not sure how that would help?

[ggabernet]

Maybe that we don't trigger the container build with the release tag with every merge to master, but only when we call this bot specifically asking for it. And we don't release until calling the bot.

But I never set up a bot like this before, so I would need to check what is possible and what not, I was just thinking about the bioconda bot.

[MaxUlysse]

I suppose we could.. Currently each docker image build is linked to a GitHub commit, but I guess if we disable the automatic build then they are essentially stand-alone. So yeah, we could just skip the build on release.

A bit tricky to automate though.. For example, it's not uncommon to have several merges to master before a release. But maybe that's ok.

I think that as long as we don't change the conda environment, the container should be good

[ewels]

Made a draft pull-request in https://github.com/nf-core/rnaseq/pull/423 for a GitHub Actions workflow that builds a docker container.

Feedback / discussion needed! Please see my PR description there.

[ewels]

Done in https://github.com/nf-core/tools/pull/616