Conda-build: Allow to specify channels in a recipe

[asmeurer]

like

channels:
  - asmeurer

Not to be confused with specifying channels on a package, which is https://github.com/conda/conda/issues/988.

[asmeurer]

The problem with this is that the channels would only be added at build time, so it's only useful if you need them for your build dependencies. For the run dependencies, your users will still need to add those channels manually.

[SlaterByte]

+1
I'd like to be able to send my whole package to a build machine and not have to set special channels. Most of my conda builds are agnostic scripts, except when it comes to channels that are required for build.

[yohannLelievre]

@asmeurer
Hi, I encounter exactly the same issue while I try to specify in my meta.yaml a dependency toward the perl package in @dan-blanchard channel. Indeed, I'm no more able to load the perl package from the default channel...
What is the current state of this issue? Is there a way to get around it?

[yohannLelievre]

I finally found a way to get around this situation.

When I build my package I use the following command :
conda build myPackage -c theDependentChannel

And when I install my package in a virtual environment :
conda install myPackage -c myChannel -c theDependentChannel

It's far from being clean... But it works.

[alstuder]

@yohannLelievre
Does not work for me. Whatever is found in default channel takes precedence. Is your dependent package in the theDependentChannel also available in the default channel?

[yohannLelievre]

@alstuder
I've made some tests with the following command:
conda build myPackage -c theDependentChannel1 -c theDependentChannel2
And, during the packaging, the precedence of my channels are respected for a package hosted in both channels. Here, the dependent packages from "myPackage" are first linked from "theDependentChannel1", if possible, and then from "theDependentChannel2".

When I executed the command below:
conda build myPackage -c theDependentChannel2 -c theDependentChannel1
I observed the expected change of precedence toward the channel "theDependentChannel2".

When I executed this command:
conda build myPackage -c conda-forge
With as dependent package "readline" (version 6.2), I linked readline from conda-forge.

And when I finally executed this command:
conda build myPackage -c defaults -c conda-forge
"readline" was linked from the defaults channel.

So maybe your issue is coming from your conda or conda-build version?
My conda version is the 4.3.15 and my conda-build version is the 2.1.8.

[martin-sicho]

Any news on this issue? It would be really nice to be able to specify the channels in the meta.yml and not having to use the command line options. And not just for the reason @SlaterByte mentioned. It would also be nice to be able to do the same for runtime dependencies. Maybe not enforce specific channels, but at least notify the user that this package might not work with some packages they already installed.

I understand it might be complicated to implement, but it would definitely be a nice feature.

And a side note:

I think this issue actually goes deeper since there are many channels on Anaconda Cloud that provide the same version of the same package, but they are not really equivalent because sometimes people don't package the software properly or not in its entirety for whatever reason. So it can happen that a package might not work due to rather cryptic reasons because it is linked against an incomplete package during runtime, for example.

[CicoZhang]

Is there any solution to it now? Could be a nice feature.

[krinsman]

Here to jump on the bandwagon and state that it might be a nice feature.

On the other hand, it might introduce unnecessary restrictions, e.g. a conda build recipe says it requires XYZ package from channel ABC, but in actuality the XYZ package from channel DEF would work just as well to satisfy that dependency.

Yet again, packages from different channels which should work well together often don't actually work as well together as would be expected (e.g. unexpected clashes between packages in the standard Anaconda channel and packages in Conda-Forge).

At the very least, it might be helpful to be able to indicate in a recipe somehow that a dependency is required which isn't found (at the time of the creation of the recipe) in the standard channel. But that might not necessarily be the same thing as the proposed feature.

[msarahan]

Conda now supports channel as part of a package spec. This is in theory possible to support in conda-build. Unfortunately, the team at anaconda is overbooked with other work, and as this feature does not directly benefit anaconda (the company), it is especially low priority for us. This feature will not happen without a community contribution.

[anton-malakhov]

We are interested in this feature. E.g. we might want to build using tools from defaults channel but we don't want any of these packages for the test environment.

[linnabrown]

I really need this feature.

[jakirkham]

Just to provide an example of what this would look like with the package spec syntax @msarahan mentioned...

# meta.yaml
package:
  name: test_recipe_needing_channel
  version: 1.0

requirements:
  build:
    - python
    # this is only available at conda_build_test, and is a test of whether that channel is available.
    - conda-forge::conda_build_test_requirement

Modified this test and reused it for this example.

[jakirkham]

Conda now supports channel as part of a package spec. This is in theory possible to support in conda-build. Unfortunately, the team at anaconda is overbooked with other work, and as this feature does not directly benefit anaconda (the company), it is especially low priority for us. This feature will not happen without a community contribution.

@msarahan, completely understand this. Would you be able to provide the steps needed and some guidance about where to look for relevant pieces? I think we will need some of this knowledge transfer for a community contributor to get started 🙂

[msarahan]

This is a rabbit hole. Adding it for one spec is obvious, and probably easy. However, where should dependencies of that come from? If they're only available on the specified channel, that's fine, but what if they're available on both the specified channel and the global channels? What is the priority? How do you treat that priority separately from other packages that should follow the global priority order? I don't think that can be defined in an always-right way.

The places you have to add this are wherever conda-build is parsing the meta.yaml into MatchSpecs. Here is a good place to start: https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L1203

[jakirkham]

Is it any less of a rabbit hole than doing something like this?

conda create -n env -c ch_a -c ch_b foo ch_c::bar

Ok so is it just an issue of parsing? Or is there an additional step of passing this on to Conda to solve?