Pillow: The new pricing model for Travis CI

Created on 2 Nov 2020  ·  19Comments  ·  Source: python-pillow/Pillow

Travis CI announced a new pricing model today:

It's not entirely clear how this affects open source, but https://github.com/zopefoundation/meta/issues/38#issuecomment-720638731 looks like the best interpretation:

  • You get a free trial of 10k credits to begin (equates to 1,000 Linux minutes; macOS costs more)
  • When the trial runs out, they want you to pick a paid plan
  • They will offer free credits for OSS "reviewed and allocated on a case by case basis", email support to apply
  • Once these run out, you can email support again and ask for more (does this mean not being able to build during the review process?)

It looks like we should move more stuff from Travis CI to GitHub Actions.

cc @python-pillow/pillow-team

Pillow

This repo, and the most active.

image

  • Let's ditch lint, it's on GHA
  • The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there
  • 3.6-3.9 (AMD64) builds: let's remove these, we have them on GHA. Is there anything unique to these that aren't included in the equivalent GHA jobs?

pillow-wheels

The biggest build, less active. Essential for releases.

image

With great timing, @nulano (thank you!) had already done the work and opened this PR to move all the jobs to GHA, except for Arm64 which need to stay there: https://github.com/python-pillow/pillow-wheels/pull/169.

Let's get that reviewed and merged.

docker-images

A big build, less active.

image

Let's shift it all over to GHA. We'll need to set credentials as secrets to upload to Docker Hub. Or alternatively upload to GitHub Packages.

Also

Least active, might as well move these over too, should be easy enough.

Checklist

Infrastructure
Source
picture hugovk

Most helpful comment

The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there

See dbhi/qus. It was created and it is used precisely for migrating ARM, PPC64 and S390X jobs to GHA.

picture umarcor on 3 Nov 2020
👍2

All 19 comments

Yeah I think @wiredfool likes GitHub Actions … I'm game.

aclark4life picture aclark4life on 2 Nov 2020

The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there

See dbhi/qus. It was created and it is used precisely for migrating ARM, PPC64 and S390X jobs to GHA.

umarcor picture umarcor on 3 Nov 2020
👍2

The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there

See dbhi/qus. It was created and it is used precisely for migrating ARM, PPC64 and S390X jobs to GHA.

Thanks for that, it works quite well. However, the performance is unsurprisingly an order of magnitude slower; arm64 takes ~30m, ppc64le ~55m, s390x ~45m.

Since most of that time is spent installing dependencies, I think it would be reasonable to migrate these jobs to https://github.com/python-pillow/docker-images (since they run in a Docker container anyway). The build and test steps alone take ~10m, which is much more reasonable.

I'm not sure if it is feasible to migrate the arm64 wheel builds to GHA as they compile all dependencies from scratch instead of downloading from Ubuntu, so I wouldn't be surprised to see those take multiple hours.

nulano picture nulano on 4 Nov 2020

You get a free trial of 10k credits to begin (equates to 1,000 Linux minutes; macOS costs more)

To quantify this, here is the usage for the past month (6 Oct to 4 Nov):
https://travis-ci.com/github/python-pillow?tab=insights: 38 builds, 1,580 minutes
https://travis-ci.org/github/python-pillow?tab=insights: 208 builds, 12,503 minutes

That is ~140,000 credits per month, assuming it is all Linux (it is more if accounting for macOS builds).

Looking at just the arm64 wheels on Travis, one build takes ~1h54m total, so 10,000 credits is enough for just 8 builds!

They do at least mention the possibility of a renewable amount:

How many credits (build minutes) you’d like to request (should your run out of credits again you can repeat the process to request more or discuss a renewable amount)

nulano picture nulano on 4 Nov 2020

Thanks for that, it works quite well. However, the performance is unsurprisingly an order of magnitude slower; arm64 takes ~30m, ppc64le ~55m, s390x ~45m.

Correct. qemu-user is 5-10 times slower than native execution or KVM. At the same time, in qemu-user some features (lower level signals, etc.) might not be complete, compared to qemu-system. Hence, combining qemu-user and containers might not fit all the use cases. However, when it does, it comes quite handy.

Since most of that time is spent installing dependencies, I think it would be reasonable to migrate these jobs to python-pillow/docker-images (since they run in a Docker container anyway). The build and test steps alone take ~10m, which is much more reasonable.

Agree. Installing dependencies should be decoupled from building and testing a project. Images with dependencies can be updated weekly or monthly, while the project is expected to be tested more frequently.

umarcor picture umarcor on 4 Nov 2020

I've created https://github.com/python-pillow/docker-images/pull/94 to migrate the docker-images job.

nulano picture nulano on 4 Nov 2020
👍1

Two weeks after the Travis CI change I'm now out of credits, their UI both says they will be replenished and that they won't be replenished. I've emailed them asking for credits, but nothing yet. So it'd be great to move ahead with this!

We now have PRs for all five repos:

Reviews/merge especially welcome on https://github.com/python-pillow/Pillow/pull/5029.

And @wiredfool please could you add DOCKER_USERNAME and DOCKER_PASSWORD to https://github.com/python-pillow/docker-images/settings/secrets/actions?

Thanks!

hugovk picture hugovk on 16 Nov 2020

I've added credentials to the docker-images repo. Any existing credentials aren't going to work anymore, so the automated build on travis is going to fail going forward, if it ever has credits to run again.

wiredfool picture wiredfool on 16 Nov 2020
🚀1

Thanks!

There's no .travis.yml at docker-images now, so there's nothing for it to build. For good measure, I've also deleted the cron at https://travis-ci.com/github/python-pillow/docker-images/settings and flipped off the "Build pushed branches" and "Build pushed pull requests" toggles.

hugovk picture hugovk on 16 Nov 2020

I just got a sort-of notification (blue dot on my profile icon) on GitHub about "GitHub Container Registry" beta: long link to docs

Looks like it is a free alternative to Docker Hub which recently introduced new usage limits of 200 container image requests per 6 hours and is replacing GitHub Packages Docker.

It seems to be easier to set up than before (maybe even easier than Docker Hub), so while I don't think this is needed right now, it might be worth looking into at some point in the future if the limit becomes an issue.

For good measure, I've also deleted the cron at https://travis-ci.com/github/python-pillow/docker-images/settings and flipped off the "Build pushed branches" and "Build pushed pull requests" toggles.

I didn't realize there was a cron job as it was not specified in .travis.yml, but it should be easy to set up on GHA as well: https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#scheduled-events

nulano picture nulano on 17 Nov 2020

Yeah, the GitHub registry may be useful in the future, but we don't need it right now as GitHub has been working with Docker Hub so GHA isn't affected by the rate limits: https://github.com/actions/virtual-environments/issues/1445#issuecomment-713861495

That's not the case with Travis CI, where the limits do apply: https://travis-ci.community/t/docker-hub-you-have-reached-your-pull-rate-limit/10517. So it's good we've moved our Docker stuff off of Travis!

Yup, we had cron enabled for the three main repos, you can only set it via the web UI for Travis. The settings:

  • Branch: master
  • Interval: weekly (not daily or monthly)
  • Options: Do not run if there has been a build in the last 24h (not "Always run")

It would be good to add cron to these three GHA once all merged. GHA is good that you can actually define POSIX cron syntax. We should set them to run at different times/days, would be nice to try and pick "quiet" times, but we're of course a global team so that might not exist :)

I'll merge https://github.com/python-pillow/Pillow/pull/5029 tomorrow if there's no further feedback.

hugovk picture hugovk on 17 Nov 2020

It would be good to add cron to these three GHA once all merged. GHA is good that you can actually define POSIX cron syntax. We should set them to run at different times/days, would be nice to try and pick "quiet" times, but we're of course a global team so that might not exist :)

Hint: you can also set 'dispatch' event keys, in order to programmatically sequence workflows in the same repo or across repos. For example:

Note: the default GITHUB_TOKEN cannot trigger multiple subsequent workflows. That's a limitation for preventing unexperienced users creating infinite loops. However, you can achieve it if you use a regular token instead.

umarcor picture umarcor on 17 Nov 2020

I see the credits have now begun counting down at https://travis-ci.com/organizations/python-pillow/plan:

Credits

3200 available credits (purchase date: December 02, 2020)
You have used 6800 of 10000 credits
3200 available

image

OSS only credits

0 available credits (next replenish date: December 02, 2020)
You have used 0 of your 0 monthly credits
0 available

image

I'll email Travis CI support requesting OSS credits for the python-pillow organisation. In the meantime, to save precious credits I've turned off the remaining crons at https://travis-ci.com/github/python-pillow/Pillow/settings and https://travis-ci.com/github/python-pillow/pillow-wheels/settings.

Let's also use [CI skip] when we don't want to test on Travis CI. ([CI skip] doesn't work on GHA.)

For reference, for my own account, after being out of credits and unable to test anything on Travis for 16 days, they've added 50k whilst they're waiting (11 days and counting) for "managerial approval":

https://twitter.com/hugovk/status/1334054612672901121

So I won't be too surprised if we're out of credits for the January release, which would prevent building aarch64 wheels (the rest are done by GHA). It might be that we can build those on a personal account, or add them to the release later on. Let's see how it goes.

hugovk picture hugovk on 2 Dec 2020

Sent:

Hello,

Please may I open a new support ticket to request OSS credits for the open-source Pillow project? I'm one of the maintainers and I believe it meets your OSS requirements.

Pillow is a graphics library for Python, with some 17 million downloads per month. The project is 10 years old and in active development, forked from an earlier 25-year-old library called PIL.

We depend upon Travis CI for both CI (to test changes) and for CD (to build binary packages called "wheels" for releases).

I would be grateful if you could ensure we have credits before 2 January 2021, when our next quarterly release is due out and need Travis to build binary wheels.

Thank you!

Received:

Hello,

Thank you for reaching out to Travis-CI Support. This is an automated message to notify you that we have received your ticket (23740) and one of our support agent will get back to you to assist with this matter. In case you are seeing a bug and haven't shared this information already, please reply to this email by sharing the steps to reproduce along with expected and actual results and your system (environment) information to supplement the troubleshoot process.

Thank you for your patience while one of the support agent gets in touch with you!

Cheers,
Your Friends @Travis-CI

hugovk picture hugovk on 2 Dec 2020

Seven days later, no reply beyond an autoreply, we're down 950 credits, from 3200 to 2250:

image

At this rate, we'll be out in 2.5 weeks, roughly Christmas day 🎅 🎄 🎁

I'm not getting my hopes up, someone else was told "credit allocation for OSS projects is on hold as per directives from management".

hugovk picture hugovk on 9 Dec 2020

@hugovk Only loosely following but I'd say we don't need Travis anymore, and by that I mean I'm not waiting around for them to give us credits. I'd rather find an alternative, even if we have to pay a Travis competitor. Can you tell me if we've completely moved to GitHub Actions or if we need to do more work and once we move there are we done? Or do we need to look for additional, possibly paid alternatives? Thanks

aclark4life picture aclark4life on 9 Dec 2020

We're in quite a good situation now.

We've moved pretty much everything we can and everything essential over to GitHub Actions.

We've kept testing of three CPU architectures on Travis for this repo, because they're not available elsewhere. These are quite new to Travis anyway. We could probably ditch two and just keep the big-endian one, that's the most useful.

Similarly, we've moved all wheel building to GHA except some (aarch64) not available elsewhere. These are additions this summer, so probably not the biggest loss if we need to skip them.

All the other repos are migrated 👍

I'm not too worried if these things suddenly stop working and have pretty much given up on Travis.

There may be alternatives to fill the gaps, it'll need further investigation (e.g. https://github.com/python-pillow/Pillow/issues/5028#issuecomment-721428190) to find sonething feasible, and see what other projects do.

hugovk picture hugovk on 9 Dec 2020
👍1

In https://github.com/python-pillow/pillow-wheels/pull/172#issuecomment-743269682 @nulano suggested temporarily switching off Travis to conserve the last remaining credits. That's a very good idea, as they're going to run out anyway soon.

At https://travis-ci.com/github/python-pillow/Pillow/settings and https://travis-ci.com/github/python-pillow/pillow-wheels/settings I've flipped these left two toggles off:

image

Let's remember to turn the pillow-wheels one back on for the release, and when/if we ever get credits.

hugovk picture hugovk on 11 Dec 2020

We've kept testing of three CPU architectures on Travis for this repo, because they're not available elsewhere. These are quite new to Travis anyway. We could probably ditch two and just keep the big-endian one, that's the most useful.
(...)
There may be alternatives to fill the gaps, it'll need further investigation (e.g. https://github.com/python-pillow/Pillow/issues/5028#issuecomment-721428190) to find sonething feasible, and see what other projects do.

I've created #5088 and https://github.com/python-pillow/docker-images/pull/98 to move the three non-x86 builds in this repo to GHA Docker tests.

I haven't investigated whether that approach is reasonable for the Arm64 wheel builds yet, but I'm guessing not.

nulano picture nulano on 11 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

etc0de picture etc0de  ·  4Comments
damianmoore picture damianmoore  ·  4Comments
Larivact picture Larivact  ·  4Comments
HansHirse picture HansHirse  ·  3Comments
steph-ben picture steph-ben  ·  4Comments