Taskwarrior: Discussion: Surveying Continuous Integration providers

[8-bit-fox]

Mentioned as part of #2338 , @pbeckingham mentioned that the project has lost its testing infrastructure.

The project is now looking for a new infrastructure setup. Requirements are apparently

• multi-platform testing, even among linuxes
• Automated builds of PRs
• Free of charge

This issue aims at collecting options to revive the continuous testing for taskwarrior.

Collected options:

• [ ] [SuSE OBS](https://build.opensuse.org/)
  
  
  
  • Linux based only
  
  
  • Packaging mechanism included
  
  
  • No dedicated testing pipelines (must be part of the building/packaging)
  
  
• [ ] Github Actions
  
  
  
  • Only three hosted-VMs are available (Win10, MacOS, Ubuntu) in various versions
  
  
  • Self-hosted VMs likely produce additional cost and maintenance effort
  
  
  • Docker container actions allow for running arbitrary Docker images, which are built during execution time.
  
  
• [ ] [TeamCity](https://www.jetbrains.com/teamcity/)
  
  
  
  • Free of charge for Open Source project
  
  
  • Project needs to meet JBs criteria (OSI + regular commits within last three months)
  
  
  • Closed-source vendor
  
  
• [ ] [AWS CodeBuild](https://aws.amazon.com/en/codebuild/)
  
  
  
  • Usage based pricing (per minute payment)
  
  
  • 100 free minutes included on the smallest node (2 vCPU + 3 Gi RAM)
  
  
• [ ] [Azure Devops Services](https://azure.microsoft.com/en-us/services/devops/)
• [ ] [Gitlab](https://about.gitlab.com/solutions/github/)
  
  
  
  • Can be integrated with Github for free for OpenSource projects
  
  
  • Offer only limited?
    
    > So, for the next year we are making the GitLab CI/CD for GitHub feature a part of our GitLab.com Free tier. `
  
  
• [ ] [Travis CI](https://travis-ci.org/)
  
  
  
  • Ubuntu or Windows as Host-System
  
  
  • Docker available only for linux
  
  
  • Already in use
  
  
• [ ] Cirrus CI

[kwoot]

May I suggest SuSE OBS? I am no OBS expert but I think it offers what is needed. Even building on non-SuSE distros and IBM mainframes :-).

[8-bit-fox]

I have not used it before, but https://github.com/keepassxreboot/keepassxc is using TeamCity. Jetbrains offers free licenses to OpenSource projects meeting their criteria. If taskwarrior is applicable or not is yet to be decided.

[8-bit-fox]

I'd gather the options as check boxes in the initial post. Would you agree, @kwoot ?

[kwoot]

I wonder if TeamCity is maybe overkill for a project of this size? Also it makes you as an open sourc e project very dependent on a closed source supplier (full disclosure, I did pay Jetbrains for a commercial license for their awesome pycharm product). I wonder what the added value of TeamCity is compared to say Github or Gitlab? Maybe I am missing something?
Since I do not claim to know everything I am much in favor of checkboxes, or polls, or some other way to gather proposals and after that quickly zoom in on the best solution to use. I strongly believe that a 100% solution is not a realistic goal, but 80% should work and allow us to finish this thought proces in a timely manner. "#include 2ct.std"

[8-bit-fox]

I wonder what the added value of TeamCity is compared to say Github or Gitlab?

Github Actions are great, but they offer three types of runners:

1. MacOS
2. Windows
3. Ubuntu

If @pbeckingham would not have mentioned that multiple Linux distros are a hard requirement, I would have opted for GH Actions: Easy to setup and easy to use.

Of course, one can add self-hosted runners to GH, but that means those need to be hosted somewhere else (private machine, private cloud, ...). Additionally, GH does not advice using them for public projects as you need a fair amount of jailing to do in order to prevent malicious codes to break out from the VM.

[8-bit-fox]

I wonder if TeamCity is maybe overkill for a project of this size? Also it makes you as an open source project very dependent on a closed source supplier.

I agree and disagree :wink: By using Github, the project already depends on a closed source supplier (although, to be fair, more and more parts of GH are getting open-sourced, AFAICS). Iff the integration with any vendor is lean (a few additional YAML files and runner configurations), then I believe that the specific vendor can be chosen regardless of their license models. If however, one would need to restructure the project just to fit the needs of the vendor, I would not recommend choosing this vendor.

To be honest, I don't believe in "free" offers. Either the service is offered for free to facilitate integration in other areas (see, e.g., GH) or its used to bring acquaint developers with the services in other areas (TeamCity, AWS CodeBuild). So I'd just choose the cheapest and leanest option available - taking into account its expected lifetime and scalability.

I strongly believe that a 100% solution is not a realistic goal, but 80% should work and allow us to finish this thought proces in a timely manner.
:+1:

[kwoot]

@8-bit-fox: thanks for the nice overview of build alternatives!

So GH and GL have runners (MacOS, Windows and Linux), and SuSE OBS has a multitude of distros that can do the building and packaging. And then there is Docker. If run locally you can build and test any distro you like. First create a run script with a desent directory structure and then create a first Dockerfile for an example distro for the test process. After that adding another Linux distro is easy. So all Linux distros can be done using Docker.

I see challenges with the other OS's. One solution is to have a maintainer per OS but that also makes it vulnerable. One solution is to document the test and build process for those so that for example a FreeBSD nitwit like me could simply follow the steps and do the test. Another option is to have a VM running with that OS somewhere.

So, do we want some central infra to be used by this project? Or do we want to add scripts to the repo to build and test as much as possible locally?Maybe those alternatives need to be added to the overview above: cloud-infra and local build. What do you guys think?

(I am now checking of I can somehow get a nice server on the net to be used by this project.)

Regarding "free" offers: I agree, there is no such thing as a free lunch. That is why I prefer the free offers (free as in freedom) when that is an option as well as a practical solution. (Not every piece of free software is also practical in its use.).

[lauft]

I wonder a bit why Travis and Cirrus CI are not listed above. Timewarrior has its current CI on Travis (and apart from the Gentoo-Image the maintenance effort was low - https://github.com/GothenburgBitFactory/timewarrior/issues/380) and I am currently trying to add Cirrus CI with their FreeBSD-VMs to libshared and timewarrior. Are there any insights I missed here?

[pbeckingham]

Thanks @lauft, I think Travis would be a good choice, but I have not look at it for a long while.

[pbeckingham]

If @pbeckingham would not have mentioned that multiple Linux distros are a hard requirement, I would have opted for GH Actions: Easy to setup and easy to use.

We have learned the hard way that we need to test on as many distros as possible, caused by the different default tool chains. There is always some #ifdef astray.

[kwoot]

I think the fact that Travis and Cirrus have not been mentioned is a simple oversight. If there is something in use already that people are happy about it seems logical to continue with that.

[8-bit-fox]

The list above is far from complete. I just did not invest more time around CET noon :wink: I'll add them and look for more (and more details) regarding the different options.

[8-bit-fox]

Just in order to see how far one gets with Github Actions, I defined an TaskWarrior "Build and test" action and used it in a local fork of taskwarrior.

At the cost of having x-different docker actions, this allows to stay within the Github sphere.

By provisioning specific base docker images (i.e. the OS + the few libraries), this could likely be reduced to one actions where the

FROM centos:7

is parametrized by an argument.

ARG OS
ARG OS_VERSION
FROM $OS:$OS_VERSION

[lauft]

@8-bit-fox You mean you could use only one Dockerfile for all? 🤔

What about the specific package managers which have to be used to setup the environment? Also at Timewarrior we had some trouble in setting the locale and timezones correctly which was also required system-specific tweaks.
I would not mind to migrate to GH - as long as the Docker images run reliably and produce useful test results. So I am quite interested in your insights 👍🏻

[8-bit-fox]

I'll investigate further. Until now, it was just a quick shot to have some kind of mvp.

[heiderich]

Concerning Travis CI keep their new billing model in mind.

An additional option might be CircleCI. I used their docker executor in the past. They also support Linux, Mac and Windows VMs.

[lauft]

@heiderich As far as I see it, it currently only concerns macOS. For this there is (still) Circle CI and Github, so we can work around that. No matter what we choose, I think there will always be the need not to bet on a single horse... 🤔

[tbabej]

Not sure if I am missing something here, but assuming we are not going to run out of credits to use Travis CI, the testing infrastructure I setup there on a New Years Eve couple of years back should still work.

For Taskwarrior see PR #2094. The testing framework was then basically mirrored into Timewarrior in PR GothenburgBitFactory/timewarrior#202 so any lessons learned there should relatively cleanly port back to Taskwarrior.

TL;DR: We currently have CI service, we just need to fix it. Migration is on the table, but only because it's likely Travis CI will disapear in the medium term, but it's not an urgent need.

[tbabej]

I opened #2344 and #2345 as a followups here.

[8-bit-fox]

@8-bit-fox You mean you could use only one Dockerfile for all? thinking

I have to withdraw my previous statement: Docker Actions allow to define runtime parameters (e.g., environment variables). Parametrization at _build_ time is not allowed. I hoped that this would be a no-brainer, as it would allow to inject the actual base image.

What about the specific package managers which have to be used to setup the environment? Also at Timewarrior we had some trouble in setting the locale and timezones correctly which was also required system-specific tweaks.
I would not mind to migrate to GH - as long as the Docker images run reliably and produce useful test results. So I am quite interested in your insights 👍🏻

However, I finalized a small mvp using ubuntu, arch and centos as example. It contains three parts:

1. the workflow definition
2. the Dockerfiles
3. the packages

In my adaption, the all images can be used with the same workflow, as the OS specific setup has already been done during image build time. Also, image builds are not part of the pipeline any more, thus providing environments for reproducible builds. None of this is GH specific, but it helped in defining one parametric workflow. It could be ported to Travis CI as well.

@lauft : From the quick glance I had at the Timewarrior Test images they look rather similar if not identical to the Taskwarrior Test images. IMO it would be a good practice to separate the environment setup (= Container build) and the project's build and test pipeline. The containers could be hosted as packages at GBF. The pipeline definition would stay in each project, be it a definition for a self-hosted Jenkins, a Github Action or a Travis Pipeline.

---

@tbabej good to know. How's the credit system working at TravisCI? GH Actions + GH Container Registry are for free for public repos.

[heiderich]

@lauft I agree that we do not need to restrict ourselves to one CI. I have seem other projects using several vendors to test on different operating systems.

I currently have no idea how many credits taskwarrior will need. I just recently saw another open source project migrating away from Travis, because their free credits did not suffice at all. So I just wanted to raise some awareness for this issue. This was not meant to be a strict argument against Travis at all.

[tbabej]

Thanks to efforts from @mrossinek and others we now have a passing CI pipeline, which has been happily running on Travis CI for one more month up until now.

However as of yesterday, the Travis CI was becoming throttled on docker pulls, and since they are planning to migrate all OSS projects to their paid hosting service anyway (as @heiderich hints at above), I migrated the entire CI workflow to Github Actions today.

Thanks everyone for participating and ideating the way forward!