staged-recipes blocked up

[jakirkham]

It looks like staged-recipes has been blocked up for the past 3 hrs. Please hold on merges. Let's investigate what is wrong.

cc @ocefpaf @msarahan

[jakirkham]

Looks like someone tried restarting Travis CI. Normally that might be a workable solution, but the fact that this has been consistently failing for many merges suggest that it is not that simple.

[jakirkham]

Failed during rfc3986, which is funny as rfc3986 is already rendered properly.

[isuruf]

Error seems to be because sphinx-bootstrap-theme folder name and meta.yaml name are different.

[jakirkham]

Nice catch.

[ocefpaf]

Error seems to be because sphinx-bootstrap-theme folder name and meta.yaml name are different.

Yep. I have a PR to fix that.

[jakirkham]

Would have figured Travis CI would have run by now. 😕 We don't know why, correct?

[jakirkham]

@pelson, I think it is worth investigating moving the feedstock generation process to CircleCI. These long waits with Travis CI are becoming a regular occurrence. 😒

[RSully]

looking at https://www.traviscistatus.com/ it appears they may only support 164 concurrent OSX builds at once - at least in the last 24h they don't peak above that. Since our builds have OSX ahead of them they share the same queue. Maybe that can be separated somehow.

edit: just saw https://github.com/travis-ci/travis-ci/issues/6211

[jakirkham]

I think you are correct about the limit (was also looking at that earlier). Also, it is true that our build queue combines OS X and Linux container builds. So, this is possibly blocked by a long list of OS X builds. This is why I'm thinking it is worthwhile to use CircleCI for feedstock conversion. We don't tend to have the same problems with that queue also that queue is pretty fast. Though we could investigate a totally different CI service like Wercker or similar for just maintenance stuff so that it is not blocked by any of our routinely used CI queues.

[jakirkham]

Well, the build finally ran, but we had a different issue. Going to try restarting.

GitHub API Rate Limit Info:

---------------------------

Currently used 4973 out of 5000.

Will reset in 0:57:51.125304.

CI Summary for conda-forge/cmd2-feedstock (can take ~30s):

Traceback (most recent call last):

  File "/Users/travis/miniconda/bin/conda-smithy", line 9, in <module>

    load_entry_point('conda-smithy==0.10.4', 'console_scripts', 'conda-smithy')()

  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 228, in main

    args.subcommand_func(args)

  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 156, in __call__

    ci_register.add_project_to_travis(owner, repo)

  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/ci_register.py", line 147, in add_project_to_travis

    response.raise_for_status()

  File "/Users/travis/miniconda/lib/python3.5/site-packages/requests/models.py", line 840, in raise_for_status

    raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.travis-ci.org/auth/github

Traceback (most recent call last):

  File ".CI/create_feedstocks.py", line 181, in <module>

    subprocess.check_call(['conda', 'smithy', 'register-ci', '--feedstock_directory', feedstock_dir] + owner_info)

  File "/Users/travis/miniconda/lib/python3.5/subprocess.py", line 584, in check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['conda', 'smithy', 'register-ci', '--feedstock_directory', '/var/folders/gw/_2jq29095y7b__wtby9dg_5h0000gn/T/tmpnprqki6x__feedstocks/cmd2-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 1

[isuruf]

Maybe something to do with, https://github.com/travis-ci/travis-ci/issues/5649 ?

[jakirkham]

@pelson, thoughts on how to fix this? It has really ground things to a halt.

[jakirkham]

Maybe, nice find, @isuruf.

[jakirkham]

FYI it appears Travis CI has stopped all Mac builds. ( https://www.traviscistatus.com/incidents/rqb906yypnk0 )

[jakirkham]

I'd like to try removing the recipes that have already been feedstocked successfully. It would be nice to narrow down the problem and leaving these in here is making things a mess. Is there any opposition to this? See PR ( https://github.com/conda-forge/staged-recipes/pull/901 ), which has waited basically a day for any feedback. If I don't hear back soon, I'm going to assume that means there is no opposition.

[jakirkham]

Went ahead and put PR ( https://github.com/conda-forge/staged-recipes/pull/901 ). It will be awhile before we see an attempt generate the remaining feedstocks. Still this should already be a huge win for people working on PRs here to have less clutter.

Sorry it is still such a mess. 😞

[jakirkham]

As general FYI, we still have sphinx-bootstrap-theme and tzlocal awaiting feedstocking. They will still affect master and CI builds for PRs. If they are really obnoxious, please remove them from your PRs. Sorry for the inconvenience.

[jakirkham]

FYI Travis CI is canceling old builds. IMHO we should let them do this and hold off restarting for a bit. Restarting staged-recipes should be at the start of our list so we can clean this up. Maybe we can slowly explore restarting PRs after that. I'm a little concerned by the potentially large fraction of Travis CI builds we are using ATM.

[jjhelmus]

Out of curiosity is there another CI service that offers a OS X platform? I don't want to leave Travis CI but if there are other options it would be nice to at least know about them.

[jakirkham]

I believe CircleCI offers this, but I don't think it is free. I looked a long time ago and there might have been one or two other viable candidates.

[jakirkham]

This was one of the other viable candidates ( https://hosted-ci.com ).

[jjhelmus]

I suppose we could also look at a dedicated host from Macstadium or a similar service but then we would need to design, maintain and deploy the CI software ourselves which does not sounds fun to me.

[ocefpaf]

but then we would need to design, maintain and deploy the CI software ourselves which does not sounds fun to me.

Ping @kwilcox :wink:

[jakirkham]

🎉 🎉 🎉 It's cleared! 😂 🎉 🎉 🎉

[jakirkham]

Feel free to keep discussing. I have closed as the original issue is resolved.

[kwilcox]

I've given this lots of thought over the past few months and talked with @ocefpaf a few times about it.

Unless all of the cloud-based CI services grant us in-kind parallel builds forever or conda-forge receives a sizable grant... our current CI situation is unlikely to get better. It won't be able to scale as conda and conda-forge usage grows. We need to explore other solutions. Lots of hard work has gone into creating a great ecosystem of community supported packages and I'm not trying to diminish from that... we need to build upon the lessons learned from the past year.

We should entertain the idea of migrating to GitLab and GitLabCI. It would be a ton of work to port everything to a new system... are the benefits worth it? Is autonomy worth it? I think so.

[RSully]

I agree with the sentiment, but there is no question that the community and ubiquity of GitHub is one of the many factors that works in this project's favor. Moving to GitLab would be a step away from that. If we could use GitLab CI with GitHub that may be ideal, but I don't believe such integration is possible. Perhaps another self-hosted CI solution that does integrate.

It looks like Circle CI does offer free macOS builds for open-source projects (here, scroll to the bottom):

We also offer the Seed plan for OS X open-source projects. Contact us at [email protected] for access. If you are building a bigger open-source project and need more resources, let us know how we can help you!

I'm inclined to see how Travis handles scaling. Right now they're back to 164 builds. Maybe worth reaching out and seeing if this is something they have the capacity to address, or if this is what we should expect going forward.

[kwilcox]

If we could use GitLab CI with GitHub that may be ideal, but I don't believe such integration is possible

mirroring github repos to gitlab isn't hard

I'm not advocating a switch away from GitHub. I'm advocating for a discussion on our absolute reliance on third party CI providers and solutions to issues like this that can really bring things to a grinding hault.

• What is an acceptable time period for a staged-recipe build to complete?
• What is an acceptable time period for a feedstock build to complete?
• Should new (staged) and existing (feedstock) builds have the same priority?
• Are some packages more important than others?

[jakirkham]

I've also thought about moving to GitLab, @kwilcox. I have debated suggesting it, but wasn't sure how it would be taken. It does put us on a totally open platform and let's us self-host everything, which is a big deal.

However, there are trade-offs. The libraries for working with the API are pretty immature IMHO. So, that will make much of the automation challenging. We have to explore hosting solutions AWS, GCE, etc. IMHO we also need to adjust our mindset from handling IaaS in a community friendly way to making sure stuff is operational and having dedicated people focusing on it.

There is this point that @RSully raises about being accessible to GitHub. Though I don't think using GitLab is so bad. They let one login from GitHub with their GitHub ID and it is always possible to add mirrors of things on GitHub.

Honestly, none of these proposals is less challenging than the other per se. Supporting a mix of CIs is challenging. Moving to a totally different Git hosting service is challenging. Going our own on AWS, GCE or similar is challenging.

However, we do have to plan for this as we are coming up on scaling issues again. To be blunt the head-in-the-sand strategy is a definite loser. We got around this last time by disabling lots of AppVeyor stuff when Windows builds were skipped. Unfortunately doing the same with Travis CI and CircleCI cannot be expected to provide the same yield as those are very rarely skipped.

[jakirkham]

Maybe something to do with, https://github.com/travis-ci/travis-ci/issues/5649 ?

So, we did have a User-Agent, @isuruf. It just wasn't Travis/1.0. This PR ( https://github.com/conda-forge/conda-smithy/pull/204 ) would change the User-Agent. Though I don't know why this one is better as there are no docs to backup what it should be.

[pelson]

Rather than throwing the baby out with the bath water, maybe we should be looking at adding our own workers to Anaconda Build. Has anybody got any practical experience with doing this? FWIW docs are available at https://docs.continuum.io/anaconda-cloud/build#BuildQueues.

[jakirkham]

FWIW the grass isn't greener on the other side. ( http://status.circleci.com/incidents/4nl5cbz2sf7f )