Spinalcordtoolbox: Moving away from travis.org

[jcohenadad]

Context

Travis .org EOF as of Jan 1 2021. We need to migrate to another CI. Options include:

• travis.com
• Github actions

Internal related discussions:

• https://github.com/neuropoly/spinalcordtoolbox/pull/2739#issuecomment-640671269

External links/refs:

• transitioning from travis.org -> travis.com: - https://docs.travis-ci.com/user/migrate/open-source-repository-migration
• Other projects reporting slowdowns on travis org domain, speed better on new .com domain: https://twitter.com/dwijnand/status/1321873017299939329

[kousu]

We migrated this repo today to travis-ci.com following the instructions.

CI is now at https://travis-ci.com/github/neuropoly/spinalcordtoolbox/, and the .org link is now archived.

If you are a member of the team, you should have access to https://travis-ci.com/github/neuropoly/spinalcordtoolbox/settings and to the "Restart Build" link in each build's menu. If you don't yet see those, then to get it:

1. Go to https://travis-ci.com/
2. Click "Log in"
3. Choose "Log in with Github"
4. Accept Github's OAuth prompt.
5. Go to https://travis-ci.com/account/repositories to see what permissions have been granted so far.
6. On that same page, click Review and add your authorized organizations and make sure that Neuropoly is checked.

7. On that same page Click "Manage repositories on Github"; for simplicitly, I granted Travis full access to my account:

   
   but you can also limit your choice to e.g. only this project.

8. Log out of Travis-CI.com and log back in via Github again; this is necessary to trigger a second OAuth prompt authorizing the newly authorized repos.
   
9. On https://travis-ci.com/account/repositories again, check what repositories and organizations Travis has access to. if you search for "spinalcordtoolbox" it should come up. I also _believe_ it should list Neuropoly on the left, whether or not you are a formal neuropoly member.
   

I'm not 100% clear on this process. It might be that now that the repo is migrated everyone in it automatically is too. Or maybe you only need to log-in the travis-ci.com to hit that second OAuth prompt. I'd appreciate if you could give some feedback, with screenshots, of your experience in this thread. That will make migrating the other 20 (50?) repos we have scattered across Github smoother over the next month.

[Drulex]

Hello on my side I simply had to sign-in using my Github account and could see the 2 repositories on the left:

I did not grant travis access to any of my personal repositories.

[lrouhier]

Hi thanks for the well-detailed plan, here is my feedback:

• It works, I can access the Travis setting for spinalcordtoolbox.
• I didn't get a second Oauth prompt but Travis asked for access to the organization the first time (I am guessing that is the reason)
• To find the spinalcordtoolbox repo I need to select the neuropoly organization in the https://travis-ci.com/account/repositories page (see in these screenshots).
  
  I did give authorization to my personnal repo, I didn't check if it works before giving this authorization.
  otherwise, everything looks good!

[kousu]

My 2 cents about the choice of server:

Actions:

• Github Actions is new and shiny; it has a large collection of CI subroutines which is very helpful e.g. for uploading to PyPI.
• Github Actions, seemingly, is very fast. They've invested a lot in compute power, probably to entice people over.
• Github Actions is weird (it's a weird mixture of docker, javascript, and shell, and optional other languages) and generally still unstable (e.g. a CVE one month ago
• Github Actions doesn't cover old macOS builds.

Travis:

• Travis is old and has weird corners too, but we're familiar with them.

It's possible to use multiple CIs in tandem. We can use Actions for publishing to PyPI and Travis for testing, e.g. skimage has Circle, Actions, Azure and Appveyor, and they used to have Travis. I worry that would be overwhelming for our small team to manage, though.

[lrouhier]

I have a less strong opinion about which CI to use now that testing only takes ~25 minutes again! Thanks for the move!

[lrouhier]

I do think we need to modify something in the repo configuration because there are two CI for each PR (one is travis.org). see #2990 for example

[joshuacwnewton]

My experience with the above steps:

• It works, I can access the Travis setting for spinalcordtoolbox.
• I didn't get a second Oauth prompt.
• I don't appear as a member of the organization (looks the same as @Drulex's screenshot in https://github.com/neuropoly/spinalcordtoolbox/issues/3057#issuecomment-733205140), despite giving Travis access. Not sure how big a deal this is, considering I can still access spinalcordtoolbox settings OK, but it's a bit weird.

[joshuacwnewton]

Re: @kousu's https://github.com/neuropoly/spinalcordtoolbox/issues/3057#issuecomment-733211296

Github Actions doesn't cover old macOS builds.

To clarify, "old macOS builds" would mean losing 10.13 and 10.14 if we were to transition completely. GitHub Actions provides only 10.15, as well as (surprisingly) Big Sur (11.0).

I think I share @lrouhier's thoughts in https://github.com/neuropoly/spinalcordtoolbox/issues/3057#issuecomment-733220555: now that we're back to normal, I'm not sure there's a pressing need to do anything until we face a new need (e.g. PyPI). But, definitely something to keep in mind for the future, I think.

[joshuacwnewton]

I do think we need to modify something in the repo configuration because there are two CI for each PR (one is travis.org). see #2990 for example

According to Travis' documentation, I believe the Travis.org repository was automatically deactivated. (See here: https://travis-ci.org/github/neuropoly/spinalcordtoolbox) So, it shouldn't receive any new GitHub events. The failing Travis.org build in the screenshot seems to be one that ran yesterday (i.e. before the migration), so I imagine on new PRs it shouldn't show up?

[Drulex]

Also Github actions can use self-hosted runners which is really cool if you have infrastructure.

When I was experimenting with a pip-only install I had already done the work to port the CI to github actions for all current platforms (with separate jobs for batch_processing). We can reuse all of that: see https://github.com/Drulex/spinalcordtoolbox/actions.

Nowadays I have a lot of CPU power available, perhaps I should start renting some self-hosted runner instances :money_mouth_face:

[notZaki]

For those that can see 'NeuroPoly' in the organization list, e.g. https://github.com/neuropoly/spinalcordtoolbox/issues/3057#issuecomment-733206894, does the Plan tab mention anything about credits?

[jcohenadad]

ha! thank you for pointing it out @notZaki , i guess we will be switching to GH actions after all 😉

currently we don't have any plan though... so we must be in limbo 🙌

[joshuacwnewton]

ha! thank you for pointing it out @notZaki , i guess we will be switching to GH actions after all wink

Yikes. This is a very recent change, too. The blog post Travis CI's new pricing plan threw a wrench in my open source works does a good job at summarizing what's going on here.

Those who, like me, finished migrating to travis-ci.com in the past couple of weeks to get out of that backlog hell are now realizing that we are going to have to beg for extra build minutes after our 1000 'trial plan' build minutes run out:

GitHub Actions does indeed seem like the way forward.

Ever since the January 2019 buyout, the writing's been on the wall—especially after so much of the existing engineering staff was laid off. I had already started building newer projects using GitHub Actions for CI instead of Travis CI.

I had planned on slowly migrating over the course of a few years, since many of my older projects still work, and are used, but are not in active development.

But the actions Travis CI took this month—without any prior notice—forced my hand

[zougloub]

Since the cloud is just someone else's computer, let's not forget to keep an indirection script which allows testing on run on any kind of runner, with minimal configuration.
Since I don't have a heat pump, I'll be glad to host runners on my infrastructure for the winter time.

[kousu]

I'm in full agreement with @zougloub . That's part of what I had in mind when I moved the bulk of the CI steps out of .travis.yml and into ci.sh.

[jcohenadad]

To anticipate cost for our macOs builds:

$15 will buy you 25 000 credits (1 minute of mac build time costs 50 credits)

so, our macOs builds take about 40min for catalina (15min for other macOss), we have:

• ~5 builds/day from PR --> 40 * 5 * 50 = 10000
• one CRON build which covers 3 different macOs --> (40 + 15 + 15) * 50 = 3500
  This gives 13500 credits/day = 8$/day = 240$/month
  Which, only for SCT, is a bit ridiculous...

[Drulex]

As of right now the Travis-CI integration is not working. Last build dates from November 27th and the repository is shown as "not active" on Travis. I created a support ticket to have Travis investigate..

[joshuacwnewton]

As of right now the Travis-CI integration is not working.

This is also happening with AxonDeepSeg, see https://github.com/neuropoly/axondeepseg/pulls

Like us, they had gotten it working briefly (see https://github.com/neuropoly/axondeepseg/pull/407) before running into the same "Expected — Waiting for status to be reported" message on PRs.

Tagging @mathieuboudreau.

[Drulex]

We solved the issue affecting the spinalcordtoolbox repository this morning. This is the reply we got from Travis support:

Hi Alexandru,

Thank you for following up!

It seems we can't detect any permissions or members for your organization. To sort this out, I'd recommend checking if the neuropoly Organization has configured third-party application restrictions that might be limiting the access of Travis to these repositories and organization in general. To test this, I'd recommend going to:

https://github.com/organizations/neuropoly/settings/oauth_application_policy

And checking that Travis CI for Private Projects is approved, in case the access to third-party apps has been restricted. The page should look like [see attached]. However, this page will only be available for admins/owners of the organization.

It might also be interesting to check what appears at https://github.com/settings/connections/applications/88c5b97de2dbfc50f3ac. This is the direct page of the Travis CI for Private Projects integration that is also available from the profile at https://travis-ci.com/profile.

We are looking forward to hearing back from you!

Regards,

The problem was indeed a permission setting at the organization level (which doesn't seem to be the case for axondeepseg since it is already enabled on Travis whereas spinalcordtoolbox was not). @jcohenadad gave Travis access to sct via this setting:

Note that after enabling the repo I had to force push a PR to get rid of the "Expected — Waiting for status to be reported" message on PRs. It picked up the change and started building as expected.

[joshuacwnewton]

I'm closing this issue, as we have finished moving away from Travis.org, and @Drulex has created a new issue with discussion focused more on setting up GitHub Actions. (#3071)