Staged-recipes: Source download issue

[jakirkham]

Not sure if this spreads beyond CircleCI, but am seeing it in several places. It would be good if everone can consolidate efforts here to understand what is going on.

cc @ericdill @ocefpaf @frol @msarahan

[ericdill]

I am currently trying to replicate this problem locally with the condaforge/linux-anvil docker container (nice name btw!)

[ocefpaf]

I can reproduce that only when invoking the build with conda-build-all using conda-build 1.20.2.
Previous version of conda-build and a direct build call seems fine.

[jakirkham]

Seems to effect AppVeyor too. Though maybe not all versions as Python 3.4 64-bit seems unaffected.

[ocefpaf]

Seems to effect AppVeyor too. Though maybe not all versions as Python 3.4 64-bit seems unaffected.

Because that one is downgrade to an older conda-build :grimacing:

[jakirkham]

Right, thanks for the reminder.

Should add that rolling back did fix CircleCI. Also, Travis CI is currently unaffected, but that this is probably also due to a downgrade there.

[ericdill]

I have a similar report as @ocefpaf

• Successful invocation of conda-build-all with conda-build-all=0.11.0 and conda-build=1.20.1
• Successful invocation of conda-build with conda-build=1.20.2
• Failed invocation of conda-build-all with conda-build-all=0.11.0 and conda-build=1.20.2

[msarahan]

So this is pointing to an issue between conda build and conda build all?

I did not test that specifically, but this highlights a critical test hole.

Thanks all for your efforts to identify the issue.

On Fri, May 13, 2016, 14:31 Eric Dill [email protected] wrote:

I have a similar report as @ocefpaf https://github.com/ocefpaf

• Successful invocation of conda-build-all with conda-build-all=0.11.0
  and conda-build=1.20.1
• Successful invocation of conda-build with conda-build=1.20.2
• Failed invocation of conda-build-all with conda-build-all=0.11.0 and
  conda-build=1.20.2

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/conda-forge/staged-recipes/issues/613#issuecomment-219138449

[ericdill]

So this is pointing to an issue between conda build and conda build all?

Seems that way. I've been entirely unsuccessful at reproducing this problem locally. If someone wants to give me admin access to circle-ci I can ssh in and debug in the environment where I know the problem exists. Otherwise, I am going to have to tap out, sorry @msarahan Good luck!

[ericdill]

I lied, this is pretty straightforward. conda_build.build:build was overhauled in conda-build 1.20.2 and has a new kwarg need_source_download, which defaults to False. conda-build-all calls this function here and is unaware of this new API, so the source never gets downloaded. See scitools/conda-build-all#33 for a potential fix.

[msarahan]

Good find. Whoops. We need to agree on some common functionality and agree on an API that will be stable. It is dangerous for conda build all to be directly using conda build's internals this way.

[jakirkham]

Yeah, had a feeling there was going to be friction at some point between these tools.

IMHO the rendering API that @msarahan has nicely exposed should be that contact point as it is designed in part to be exactly that. We should discuss if this is sufficient for the use cases served by conda-build-all and what changes (if any) need to be made on either side of the conda-build/conda-build-all equation in order to make this work.

[msarahan]

For now, I will fix conda-build so that conda-build-all will just work here (change default, or something). However, we should hammer this API out sooner rather than later.

[pelson]

Good find. Whoops. We need to agree on some common functionality and agree on an API that will be stable. It is dangerous for conda build all to be directly using conda build's internals this way.

I agree completely. The sooner there is an API (that _isn't_ the CLI) the better.

For now, I will fix conda-build so that conda-build-all will just work here (change default, or something). However, we should hammer this API out sooner rather than later.

I think we should fix conda-build-all, not conda-build. Let's move forwards, not put conda-build in a straightjacket.

[pelson]

So, I'm going to update the metadata for conda-build-all and pin appropriately. Then, I will cut a new release of conda-build-all with @ericdill's awesome fix. Once complete, there should be no need for a patch version of conda-build (though setting requires_source to True does sound sensible for a build step 😉 ).

I'll update here once complete.

[pelson]

OK:

• [x] update metadata for conda-build-all v0.11 (https://github.com/conda-forge/conda-build-all-feedstock/pull/8)
• [x] release conda-build-all v0.12 (https://github.com/SciTools/conda-build-all/releases/tag/v0.12.0) - thanks @ericdill!
• [x] update feedstock for v0.12 (https://github.com/conda-forge/conda-build-all-feedstock/pull/9)

If we see this again, take a look at the conda-build and conda-build-all versions - if they are both at the latest versions there should be no problem.

[jakirkham]

As we seem to have fixed this, going to go ahead and close this out. Thanks everyone for helping figure out the issue and getting it corrected. 🍰

[jakirkham]

@jjhelmus, here is the issue for source download failing in response to your question. This happens with all source downloads not just Python source.

It turned out this was caused by conda-build-all because of an API change in conda-build. See @ericdill's summary above for cases when this occurs. Though this should be fixed for the latest conda-build and conda-build-all.