Conda-forge.github.io: CI support for long build times

Created on 18 Nov 2019 · 4Comments · Source: conda-forge/conda-forge.github.io

There's several packages in conda-forge that exceed the CI time limits (e.g. 6h for azure pipelines), and some of them are of central importance to the ecosystem (gcc, qt, etc.).

These timeouts make building the packages extremely cumbersome, but this is unfortunately common enough that a dedicated proposal is being worked on to at least have a process for these cases.

I wanted to open this issue to initiate a discussion with the azure people (where most of the conda-forge-CI is run anyway), to see if there's a way to increase these limits for selected packages, e.g. to 24h? It's not like repeatedly running failing 6h-jobs reduces the total load compared to much fewer successful runs that might take 7-10h or so. Not sure if this has been discussed elsewhere or privately already, but I couldn't find anything, and so having a public discussion can't hurt IMO.

I remember seeing @vtbassmatt helping a lot of (python) packages migrate their CI to azure pipelines, so maybe you can help there, at least by involving/CCing the right people on the azure side? Even if the affected packages would have to go through an individual approval process, removing this timeout on some key feedstocks would be a big win.

Source

h-vetinari

👍1

Most helpful comment

@ocefpaf: Do you mind copying your comments above to #902 and closing this one just for "housekeeping's" sake?

Done.

@vtbassmatt, please post any answers to #902 rather than here. :)

h-vetinari on 18 Nov 2019

👍2

All 4 comments

👋 Thanks for pinging me. Let me talk to some folks on my side. I think the answer is going to come in the form of a new feature we're launching soon - but I want to double check if I'm forgetting something that's already available.

Not sure if this helps but: the 6 hour limitation is per job, not per pipeline. Can you divide the work into multiple jobs that each stay under the limit? I know this comes with some extra complexity, like needing to upload/download artifacts in several places, so it's not a trivial fix.