Jrnl: Should we switch away from Travis CI?

[wren]

Travis is becoming less and less stable. The last few releases have all given the maintainers troubles that all ended up being Travis problems (for example v2.4.4 wouldn't even pip install for a seemingly random subset of the tests). There's also been some PRs that error out inexplicably, and the only fix is to clear the Travis build cache (which most contributors can't do, so they are stuck).

Also, Travis' support has also been less than stellar. I've gotten responses that basically amount to "we can't help you" even when the problem occurs long before any jrnl code is ever run, and seems clearly related to the pipeline itself.

Normally, I'd try to work with a vendor more to fix problems like this, but it's been getting steadily worse. And since they had some big layoffs in early 2019, I don't have high hopes that they'll be fixing the problems any time soon. I've also seen similar frustrations among other open-source projects that have started switching away.

This discussion is to weight the pros and cons of moving our CI pipeline elsewhere, and if we move, then where to? So, if you have any experience with some vendors, please weigh in and let us know what you think.

Here are the requirements we're looking for:

• Supports Linux, MacOS, and Windows
• Supports Python 3.7+
• Can manually trigger pipelines
• Runs on managed servers
• Has free or low-cost options for open source projects

These are the vendors I've worked with:

• Travis
  
  
  
  • Pros: Already our current system. Sorta mostly works. No effort to migrate.
  
  
  • Cons: Months of internal neglect have lead to an increasingly unstable platform with inadequate support.
  
  

• Github Actions

  • Pros: Has lots of momentum right now. Good support for official steps. Less moving parts if pipeline is same place source is stored.
  • Cons: Migration effort. Heavy reliance on user-submitted steps (with little support or QA). Putting all of our open-source eggs in one Microsoft basket (doesn't always go well, historically speaking). Can't manually trigger a pipeline (useful for maintenance and releases).

• CircleCI

  • Pros: Been around a while in internet terms. Good track record for open-source. Many large clients to keep the lights on. Very fast (uses warm VMs to run docker containers). Good out-of-the-box caching options.
  • Cons: Migration effort. Unclear if all 3 OSes we need are included in open-source plan.

[MinchinWeb]

Solely in terms of Windows support:

• Travis is experimental. Their set up is a little odd because it seems to almost be bash running on PowerShell. It's frustrating because it doesn't quite respond like like either would on their own. (I guess the theory is that this way you don't have to re-write any shell scripts...). No official Python project support.
• Github Actions seems to support Windows as a first class citizen from day one. I haven used it.
• Circle CI has Windows support. The current image has Python 3.7.3 installed, although other version should be install-able through Chocolately.
• AppVeyor is the grand-daddy of Windows CI (I think originally they were Windows-only). It looks like now supports Linux and Mac as well. In days gone by, it wasn't uncommon to see a project using Travis for Linux tests, and AppVeyor for Windows tests.

My two bits:

• if Travis is breaking, then it's worthwhile to look for alternatives.
• GitHub Actions does seem to be the new hot thing, but it seems to work decently once set up. It does tie to project closer to Github, but I have no personal issues with that (I like GitHub as a platform). Creating custom steps (if needed) is a bit of a learning curve, but not insurmountable.

[wren]

AppVeyor does look pretty solid. Looking through their open source plan, though, it's only 1 concurrent job. I'm worried that would slow our tests down quite a bit. Do you think that would happen?

[wren]

I think circle is the best fit for us in order to properly support all the platforms we currently support, and will also make it easier to add more as we go. We'll start adding circle alongside travis, and run them concurrently for a while before switching travis off.

Thanks for your insight, @MinchinWeb!