Hypothesis: Move our CI to Azure Pipelines

[Zac-HD]

https://azure.microsoft.com/en-us/services/devops/pipelines/ - TLDR for open source: 10 parallel jobs, identical to $40/month jobs so probably fast, and supports Windows, Linux, and OSX.

I'd be interested in replacing Appveyor's serialised jobs with identical-but-parallel jobs, and we'd have enough space capacity to replace CircleCI too and still be faster than the status quo.

Tactically, this is probably one PR to add the new CI, wait until it's been working stably for a few weeks, and then a second PR to turn off the old CI. This also makes reverting the experiment really cheap, because we can just disable the new one instead if we don't want to stick with it.

[vtbassmatt]

Hi there, I'm a PM on Azure Pipelines. Let me know if there's anything I can do to help out here.

[Zac-HD]

1854 ported our main build stages from Travis to run on Azure pipelines, and it looks like it's going pretty well so far! As a checklist for future work:

• [x] Our CI scripts rely on the travis_event_type envvar being present. They shouldn't.
• [x] Migrate OSX jobs to Pipelines and drop CircleCI.
• [x] Run cover tests on Pipelines under Windows
• [x] Run remaining Windows tests on Pipelines.
• [x] Drop Appveyor.
• [ ] Move deployment from Travis to Pipelines. Here's the secrets docs.
• [ ] Move pip-compile pushes for pyup PRs from Travis to Pipelines.
• [x] Make check-whole-repo-tests non-concurrent (i.e. part of the normal prechecks) once Pipelines supports build caching.
• [x] Investigate Pipeline's story for auto-cancelling redundant builds (cc @vtbassmatt?) - important for efficient use of machines when we can fail early or have several pushes to a PR in quick succession. In particular we may want more stages again once caching is a thing and jobs are faster.
• [x] Eventually stop running on other CI platforms.

Anything else we should do or consider in moving over?

[vtbassmatt]

New commits appearing on a PR will cancel running pipelines for that PR. We don't support fail-fast behavior (e.g. where one leg of a matrix fails, cancel the pipeline) today, but it's on my wishlist for Q2.

Also, if concurrency becomes an issue moving over your Windows and macOS builds, I can make that problem go away.

[Zac-HD]

New commits appearing on a PR will cancel running pipelines for that PR. We don't support fail-fast behavior (e.g. where one leg of a matrix fails, cancel the pipeline) today, but it's on my wishlist for Q2.

Awesome!

Also, if concurrency becomes an issue moving over your Windows and macOS builds, I can make that problem go away.

:heart_eyes:

We don't currently have a concurrency problem, but increased contention is literally the only reason not to move those builds IMO. We've been living with Travis because there wasn't a better alternative, but for CI quicker is always better - especially with e.g. PyCon US sprints coming up.

[vtbassmatt]

Feel free to start moving your Win/Mac pipelines over. I'll get concurrency sorted today.

Will you be at PyCon? I'm going (my first time!) and would love to have you stop by the Pipelines booth.

[Zac-HD]

:tada:

I am - it'll be my first PyCon US too, and I'd love to catch up in person! As well as sprints, I'll be presenting a poster, giving a talk, and teaching a tutorial. I really wanted to justify the trip from Australia, but didn't anticipate all my proposals being successful - fortunately I can draw on the talks and workshops I've done at Pycons AU, APAC, and London :sweat_smile:

[vtbassmatt]

... I think your trip is justified! I'll try to catch at least one of those editions of your talk, and for sure we should catch up in person.

[Zac-HD]

We have everything but deployment running on Pipelines now (:tada:), so I've created a more focussed issue 2043 for that last push. Thanks to everyone who was involved in this!