Staged-recipes: Getting your run dependencies right

[mcg1969]

Hey folks,

Something I said in the comments of the merged RIOS PR:

I'm not convinced you should relax your gdal 2.0.2 spec (or perhaps it should become a >= spec). The TravisCI output suggests that rios isn't working with gdal 2.0.0. You need to prevent people from installing this package alongside 2.0.0, then.

I wanted to make sure that this doesn't get lost in an already closed/merged thread, because it doesn't just apply to RIOS, it applies across the entire conda-forge ecosystem.

It is important that a package's run requirements have sufficient version requirements to ensure proper operation. In particular: _it is not sufficient to assume that a user will have, or obtain, the latest version of a given dependency._ If you _know_ your package is not compatible with older versions of a dependency, you _must_ reflect that in the run dependencies.

For example: suppose I have a functioning environment built containing gdal 2.0.0 from defaults; then I type conda install rios. If rios simply specifies gdal as a run-time dependency, with no version specification, then it will happily install it _without upgrading gdal_. But rios will be broken according to this. This means that an unqualified runtime dependency on gdal _is incorrect_. On the other hand, if rios includes gdal >2.0 as its runtime dependency, it will cause conda to upgrade past the failure.

On the other hand, you want your dependencies to be as relaxed as possible, otherwise you invite a litany of unsatisfiable specs in complex environments. So, for example, the current rios recipe specifies gdal 2.0.2. In my view, this ought to be relaxed to gdal >2.0 or gdal >=2.0.2, unless you have good reason to believe that subsequent versions of gdal will break rios.

To be fair, the problems here are made worse by channel collision issues: it could be that conda-forge's version of gdal 2.0* works just fine, but the ones in default do not. We're working on that, as you know. But for the foreseeable future, people are going to be mixing and matching packages from defaults and conda-forge, so one should still take care to lock down dependencies to avoid breakage.

[mcg1969]

@jakirkham @ocefpaf @pelson, I'm a bit concerned by some comments in the rios PR and this one as well. I think there may be some misconceptions about how conda's dependency solver is supposed to work. If so, those misconceptions are being baked into the conda-forge build process---to its detrement.

It's possible I'm misinterpreting some things being said; forgive me in advance if that is the case. But I definitely want to make sure one way or another.

cc: @msarahan

[jakirkham]

So, I think the problem here is there is no way in conda-build _yet_ to _dynamically_ configure one's dependencies based on what a recipe was built against. The reason we need this is while there is a large range of things we can build against; the runtime range is smaller and dependent on the build version to ensure ABI compatibility. Without addressing that, we are kind of at an impasse here. If you are saying that this is a general problem with conda recipes at this point, I completely agree with you. I would argue that this is the biggest problem facing recipes by far. However, it is hard to say this problem is a secret as people have known about and tried to fix it for many months.

Sure we can add some pinnings that we have determined work globally, which we have done in a few cases. We probably will continue to do this at present. There is some code to help automate this pinning across feedstocks and we have discussed furthering the infrastructure to accommodate this. Look at the recipes using HDF5 for an example and the batch generated PRs that have pinned them. However, this is at best a crutch. We can certainly improve on it and maybe it solves the whole problem here, but that is to the detriment of the larger conda and conda-build communities. The reason being it has not addressed this fundamental problem in recipes themselves.

Ultimately this dynamic pinning is really a problem that lives in conda-build. There have been multiple proposals to address this ( https://github.com/conda/conda-build/pull/728 ) ( https://github.com/conda/conda-build/pull/747 ) ( https://github.com/conda/conda-build/pull/848 ) ( https://github.com/conda/conda-build/issues/857 ) and I am sure I have missed a few in this list (though not intentionally). Right now this one ( https://github.com/conda/conda-build/pull/848 ) seems to be the most powerful, but is also quite complex in what it expects an end user to know and do (especially one with little knowledge of what goes on in conda/conda-build). That all being said, I am encouraged to see that a few of the people behind these proposals are going to sit down and discuss them in a chat. I hope that from this discussion there will be a simple, clean, effective, and reasonable proposal set forth that everyone can agree on.

[mcg1969]

Actually, I think the focus on automation may be part of the problem here. I am _in no way_ convinced that dynamic pinning is going to solve the problem I am outlining here.

There are two ways that dependencies can be improperly built: they can be _too relaxed_ and they can be _too tight_. My concern is that any sort of dynamic pinning approach is going to tend to produce specifications that are _too tight_. Furthermore, these errors are not equal in severity. In our experience, specifications that are too tight are _worse_, because they reduce the user's ability to carefully construct complex environments, and they produce strange and often unnecessary "unsatisfiable specification" scenarios.

When I take a 50,000-foot view of conda-forge, I see that its _primary_ benefit is not the level of automation you're providing---although make no mistake, it is fantastic. Rather, the primary benefit is the way this automation enables _crowdsourced_ development, testing, and _debugging_ of recipes. We at Continuum know far too well that conda recipes can sometimes be a challenge to get right. Some of them simply require tweaking and iteration. No automated system is going to be intelligent enough to do this.

The way you're doing this tweaking and debugging of recipes, however, is sub-optimal. For instance, take the RIOS PR again. There were Travis CI failures caused by gdal 2.0. What was the response? _Blame conda_. The solution to pin gdal to 2.0.2 was seen as a _band-aid_, when really, _it is the proper solution_. Well, not _quite_---the proper solution was to specify gdal >2.0 or gdal >=2.0.2, not gdal 2.0.2; the latter is far too tight.

Even though you considered the differences between the build and test environments to be a problem---and make no mistake, I've fixed that in master---the point is that _it performed an important service_ by revealing an incompatibility between packages. Indeed, a _proper_ testing regime, if we had unlimited compute resources to do so, would involve running tens or possibly hundreds of tests over an entire _matrix_ of dependency version combinations.

What I _can_ tell you is that running your tests solely on the same pristine, sterile, version-maximized environment that you used to build a package is an easy way to guarantee issues when users end up installing your package in environments with thousands of different dependency combinations. That seems pretty common-sensical to me. I see no practical alternative to being prepared to respond to those failure modes that your tests just didn't (and can't) catch.

[mcg1969]

I have been staying out of a lot of the conda-build discussions you linked to above, because so much of it involves implementation details that I frankly don't have a stake in---and too many cooks spoil the broth.

That said, when I look at the list of questions in https://github.com/conda/conda-build/pull/848, it is clear that for me getting Question 3 right ("How do we _relax_ the build specifications properly") is the piece I am most invested in. I worry that whatever the solution ends up being, it is going to encourage _too little_ relaxation. And that's going to be worse than the current situation.

There is no substitute for human experience and intervention. Let's make sure to make such interventions as simple as possible.

[jakirkham]

I'll respond to your earlier comment in a bit. Will stick with your more recent one for now as it is shorter and I think there are still many things we agree on and I would rather highlight those for now to avoid confusion.

I have been staying out of a lot of the conda-build discussions you linked to above, because so much of it involves implementation details that I frankly don't have a stake in---and too many cooks spoil the broth.

I tried to also, but it was getting stagnant and it is too important to not have it moving forward. Also, some important concerns were being ignored.

I worry that whatever the solution ends up being, it is going to encourage to little relaxation. And that's going to be worse than the current situation.

I am also worried about that too. It has already started to happen here too unfortunately. Though I doubt we are getting to hit the right solution immediately on such a scale, but I do hope we converge close to if not on it exactly.

Though I think of one the nice things about ( https://github.com/conda/conda-build/pull/848 ) is it does allow one to programmatically define what constraints should be used in a far more flexible manner. However, it feels a bit complex for all users to be doing this and am a bit worried to see everyone using this functionality. It would be nice if conda-build could handle some generalized forms of version constraint functions and expose a simpler interface to its users.

There is no substitute for human experience and intervention.

Of course. Not questioning that.

To briefly diverge and respond to one point in your previous comment.

...the primary benefit is the way this automation enables crowdsourced development, testing, and debugging of recipes.

➕ 💯 Completely agree. Leveraging GitHub's social coding mentality is huge at making this easy. All the automation merely entices people to come here and makes it easier to manage their contributions. Though that is totally secondary to the work people do here.

Let's make sure to make such interventions as simple as possible.

Absolutely, this is my biggest concern. Hence question 7, which I am not sure has been well addressed in that thread yet.

[msarahan]

Overly specific pinning has been the cause of many problems I've dealt with in supporting anaconda. Please avoid pinning to a specific version. Automation might make it easier, but I still agree with @mcg1969 that this is more trouble down the road. Please be patient, and let's do this right. I'll push this internally as best I can, and having @mcg1969 on board should help. I will invite him to the meeting for discussing this.

[pelson]

I think this issue has served its purpose of raising awareness of the issue. In the interest of keeping things clean, I'm going to go ahead and close this. I think there is a place for these kinds of lessons in documentation, be it on conda somewhere, or in the fledgling docs at conda-forge.

Thanks for raising. :+1: