Staged-recipes: Travis CI builds not working?

[endrebak]

Seems like a lot of the last PRs have been mysteriously cancelled and that new ones aren't being built: https://travis-ci.org/conda-forge/staged-recipes/pull_requests

[ghost]

All OSX builds are currently stopped.

[CJ-Wright]

relevent https://www.traviscistatus.com/

[djsutherland]

So, my proposal on best practices on staged-recipes for situations like this. @conda-forge/staged-recipes, do you agree?

• noarch packages can definitely be merged normally.
• Pure-Python but not noarch packages (because of a version constraint or whatever):
  
  
  
  • If there's nothing OS-specific at all, IMO can definitely be merged if it passes other platforms (though of course Mac builds won't be available until Travis recovers).
  
  
  • If there's some OS-specific stuff, IMO things can still be merged if you're careful, and especially if you test them locally on a Mac.
  
  
• Compiled packages that should support Mac, I see a few options:
  
  
  
  1. Don't merge anything until Travis resolves itself.
  
  
  2. Merge with skip: true  # [osx], and send a PR to the feedstock enabling Mac builds. Merge that once Travis is working again.
  
  
  3. Merge without skipping Mac, looking carefully at things to make sure you think they'll work and also with someone having tested it locally. It's possible things will work locally but not on the CI, but in that case the first Mac build fails, and we resolve it on the feedstock.
  
  

(i) is not ideal, especially when Travis is bad for a long time (as it's been this week). (ii) and (iii) have essentially the same end-user behavior (Mac builds aren't available until they're working). (iii) is more likely to work faster once Travis is up again and reduces the load on Travis (because there'll only be one build, instead of a PR plus one when it's merged), but also has added risk that a broken Mac build will be published (if it for some reason passes the tests but we would have noticed from the logs that it's broken). I'd probably go with (iii), but maybe people prefer (ii), or maybe there's some other option?

[ghost]

The Travis-CI OSX situation is worsening over time and may eventually become unavailable entirely. Travis-CI is aware of the situation but they simply lack the budget to resolve it. It is therefore unreasonable to delay any PR because the OSX build has not finished.

First, I suggest that OSX builds be disabled entirely in staged-recipes because there is no reason to build them if the OSX result won't make a difference. Second, I suggest modifying the bot to automatically submit an OSX PR once the recipe has been added. It's not a pretty solution but it is simply acknowledging reality.

[ghost]

Note that the linter and re-renderer should also be modified to skip OSX by default in staged-recipes.

[djsutherland]

@xoviat I think there are a few Mac-only packages where that wouldn't work, but that's not a terrible idea. I don't entirely understand how the Travis queue works, but it also might speed up feedstock builds substantially to eliminate a lot of usually-irrelevant staged-recipes builds.

[CJ-Wright]

@xoviat what do you mean that the linter should also skip OSX? The linter doesn't run on Travis' infrastructure and doesn't have any dedicated OSX linting (to the best of my knowledge).

[ghost]

When someone submits a PR to staged-recipes that includes Windows or Linux, the linter should include the following message:

"Please add skip: true # [osx] to the recipe."

[jakirkham]

Sorry not following your suggestions to always skip macOS, @xoviat. If it amounts to skip all macOS builds entirely, that will be a -1 from me. Though I suspect you are referring to something else that I'm still not picking up on.

[djsutherland]

I think @xoviat's suggestion is:

• Never do anything with MacOS on staged-recipes. Always merge (non-noarch) packages with skip: true  #[osx] in them.
• Have some automated way to, once a feedstock is merged, open a PR removing the skip statement and re-rendering.

[ghost]

What I mean is that submissions to this repository should skip OSX by default. Once the "-feedstock" repository is created, the bot should submit a PR to enable OSX (simply removing the skip line).

[ghost]

Have you considered testing OSX on circle CI? They seem to offer it as an option and I think it's worth considering.

[djsutherland]

CircleCI's free Mac builds for open source projects only give 500 minutes of build per month: https://circleci.com/pricing/#build-os-x. Though they do say "If you are building a bigger open-source project and need more resources, let us know how we can help you!" – so maybe someone should get in touch with them and see if there are options.

[jakirkham]

It's true it is a pay for service though I think I agree we might need to figure out some funding (joining NumFOCUS will make it easier to collect donations) and pay for macOS support. The other option is MacStadium, which is what Travis CI uses. Though we might have to do more legwork to configure that to build. Unfortunately Travis CI doesn't provide a similar option. Though maybe we should ask them again. I'm sure they would like the revenue.

Generally core has also been interested in having some sort of "conda-forge@home" type solution where users give us build time on their machines. There are some challenges to that approach. Some of which are addressed by conda-concourse-ci. The rest is automatic configuration, which normally is tricky with anything, but definitely worth the effort for those that have time and interest.

[jakirkham]

Also we have had discussions about culling builds on CIs at staged-recipes (based on the content of skip in the recipe). Some discussion within issue ( https://github.com/conda-forge/conda-forge-webservices/issues/79 ). It might be time to resurrect this as there are many noarch builds at staged-recipes yet no way to restrict just to CircleCI yet.

[CJ-Wright]

If we went with the always skip osx in staged why do we need to explicitly skip OSX. Can't we just eliminate travis from staged all together? Then when the feedstock renders we can have it build OSX as normal (or send it a signal if need be).

[jakirkham]

I don't think it is a good idea to disable Travis CI on staged-recipes outright.

[ghost]

@CJ-Wright That's probably a simpler option.

Note that a major problem with the Travis-CI and CircleCI offering is that they run untrusted code on a VM. Performance can be significantly improved if you're willing to relax the trust requirements (meaning that a malicious feedstock can affect another feedstock, but that's not really a concern here).

[djsutherland]

@CJ-Wright The skip would be so that the new feedstock doesn't automatically try to build a completely-untested Mac build. I guess that's my option (ii) vs (iii) above, but I'd be worried about having that be a global default.

[pkgw]

@jakirkham I am pretty sure that it's not possible to do a conda-forge@home that can guarantee that a rogue client can't inject malicious binary code. That's just not possible unless you have complete control of the build machine. (Relatedly, I hope Travis and friends have well-funded security teams! The are probably the targets of sophisticated government-backed attacks.)

[djsutherland]

Reproducible builds could help with possible conda-forge@home concerns, though it's definitely beyond our scope to require all packages we build to be reproducible. :)

There are already plenty of potential security flaws in the current conda-forge setup, though.

[ghost]

For clarity, what I mean is that if you were to purchase a dedicated OSX instance on MacStadium, you could simply reuse the same OS image to build the recipes, which would remove the startup time completely. You also would not need to patch the recent processor vulnerabilities.

[jakirkham]

We can also customize the image. For instance, see all the time we waste cleaning up Homebrew and installing Miniconda on Travis CI (something like 2m30s). Should add that if you enable caching on Travis CI it ends up taking a whopping 7-8mins (suffered this recently) just to get to the point where you run your own code.

[jakirkham]

Would encourage everyone to take a look at issue ( https://github.com/conda-forge/conda-forge.github.io/issues/306 ). Many similar points have come up before.

Forgot that Travis CI did offer us more concurrency though it might not matter if the queue is still very backlogged. MacStadium is a much better deal.

[djsutherland]

Note that the MacStadium plan you listed there is the highest-end option: the cheapest plan is $50/month, for one i5 Mac Mini. There's more infrastructure someone needs to code up and less parallelism, but that's already ~43,000 build minutes per month, compared to CircleCI's $449/month plan that gives you 25,000.

[pkgw]

Getting away from the macOS-specific problems, as I recall, the staged-recipes housekeeping activities (creating repos for approved feedstocks, etc.) also run on Travis, which seems to have the biggest reliability problems. Would it be possible and/or helpful to transition those to Circle CI instead?

[jakirkham]

Indeed. I think the comment mentioned it as well. There are definitely a variety of options. Essentially we are paying for the CI build infrastructure and image updates with the other CIs. Though I think we want to ultimately control the latter if we want to make any statements about compatibility. One definitely saves a lot of money if they are willing to do some upkeep for this stuff. We are definitely in a better place now to take this on than when we initially discussed moving off CIs.

[jakirkham]

Would it be possible and/or helpful to transition those to Circle CI instead?

Right. Raised issue ( https://github.com/conda-forge/staged-recipes/issues/916 ) on this topic some time ago. As noted in the final comment, we have managed to get these builds to run on Travis CI's sudo-free Linux images, which Travis CI started running on a separate queue from the macOS builds. So I don't think this matters any more. That said, if you are seeing issues, would be good to know. Can always reopen that issue.

[ghost]

Would it be possible and/or helpful to transition those to Circle CI instead?

As @jakirkham said, container-based Travis-CI reliability is not a problem. The problem is only with the OSX builds.

[jakirkham]

Back to your original questions @dougalsutherland, my recommendation would be some form of ii. Feel free to present the option of adding skip: true # [osx] to recipes for users eager to get things in. Would not require users to add skip: true # [osx] as it should be their choice. However I expect much like noarch: python, people will be very excited to speed up getting their recipes in and packages out there. So encouragement is likely unnecessary. Have already seen some users do exactly this without suggestion.

Would add that with noarch packages, feel free to cancel non-CircleCI builds (including AppVeyor). Ultimately we want a webservice to do this for us, but in the interim that is an ok thing to do. Resources are limited. So they certainly shouldn't be wasted. Also feel free to merge noarch packages that have CircleCI, linter, and reviewer comments addressed.

With all the gray areas, ultimately I trust your judgement. Though certainly feel free to ask if unsure.

[djsutherland]

So, to summarize, and please correct me if I'm wrong somewhere:

• For now, we should:
  
  
  
  • More-or-less follow https://github.com/conda-forge/staged-recipes/issues/4803#issuecomment-358414278 with option (ii).
  
  
  • Manually cancel Travis builds (and, with less priority, AppVeyor builds) for noarch packages / those that skip that platform
  
  
  • Make sure any package that can be noarch is:
  
  
  • Packages that e.g. would work on Windows except that a dependency isn't available are fine (https://github.com/conda-forge/staged-recipes/issues/4787)
  
  
  • Once conda-forge is using conda-build 3, we can start doing noarch: python packages that depend on e.g. python >=3; this apparently doesn't work with conda-build 2 (https://github.com/conda-forge/conda-forge.github.io/issues/486#issuecomment-345301145)
  
  
• Medium term, we should consider resurrecting https://github.com/conda-forge/conda-forge-webservices/issues/79 – with the rise in noarch this could drastically reduce our Travis load from staged-recipes
  
  
  
  • Can't really write a webservice to cancel builds because of https://github.com/travis-ci/travis-ci/issues/6838
  
  
• Longer term, conda-forge should consider options for more resources, as discussed in https://github.com/conda-forge/conda-forge.github.io/issues/306
  
  
  
  • Figure out NumFOCUS stuff and some source of donations so we can pay for things, and then one of:
  
  
  • Pay Travis for more concurrency
  
  
  • Talk to CircleCI about options for paying for more than 500 build-minutes per month
  
  
  • Buy capacity from MacStadium and run Jenkins/etc on it
  
  
  • Get conda-forge@home working (based on conda-concourse-ci, but with more work to be done)
  
  

[tkelman]

Are there any Mac Travis builds running anywhere in the conda-forge organization right now, or is everything triggered in the past 5-6 days completely stuck? At https://www.traviscistatus.com/incidents/p1g70p3qrq4z they say

Please restart any builds that are currently "hanging". We will cancel any remaining stalled jobs later today.

[ghost]

Note: I may be wrong, but Travis-CI appears to have disabled conda-forge OSX builds entirely to reduce load on their system. Have any OSX packages been uploaded to anaconda.com within the last week?

[ghost]

A quick look says no, which means that we might as well drop the pretense that conda-forge supports OSX at all, because it no longer does.

[CJ-Wright]

@xoviat I think this went in https://travis-ci.org/conda-forge/cmd2-feedstock

[isuruf]

From what I guess, every time 2 jobs (not 2 builds) are run in conda-forge, we go to the back of the queue which is around 3000 jobs at the moment. So, we have to wait until 3000 of those jobs to build before we get 2 more jobs.

[tkelman]

Does anyone affiliated with conda-forge have access to Travis CI's slack channel? I used to be on there but was removed without any warning or having done anything recent other than lurk. If not, someone can contact their support channel and ask why throughput seems so low. Having this repo and all feedstocks in the same organization makes the problem much worse, and recovery times much longer after any issues on Travis' side.

[ghost]

Throughput is lower because the patches to address recently-discovered processor vulnerabilities disabled speculatively executing certain kernel calls, which reduced build performance.

Having this repo and all feedstocks in the same organization makes the problem much worse, and recovery times much longer after any issues on Travis' side.

Travis-CI considers this to be a feature: there are only so many OSX machines to go around, and if they can throttle the OSX builds from any particular organization, their queue is reduced.

[jakirkham]

So I think the original question was answered and the plan laid out. Going to close this out.

[jakirkham]

Also further discussion about acquiring macOS workers should be directed to issue ( https://github.com/conda-forge/conda-forge.github.io/issues/306 ), which is actively being discussed. Your feedback and help gathering information about options and current usage would be very useful. Thanks all.