Stack: Handling of the ^>= operator

[snoyberg]

Cabal 2.0 introduces a new operator, ^>=. The next release of Stack will be the first release to support this format. The documented meaning of ^>=, and the behavior of the built-in helper functions in Cabal (the library), is to treat ^>= x.y.z as identical to >= x.y.z && < x.(y + 1) (with some more clarification around single component version numbers).

There has additionally been discussion ongoing in a number of places about this operator implying a "soft bound," and instead of requiring build tools to respect a bound, use it as evidence of prior build success to support the dependency solving case.

This leaves the question for Stack: how should we treat this operator. I can think of four options:

1. Do nothing special (currently implemented on master), and let the built-in behavior of a hard bound fall through. This means that a package that if a package depends on foo ^>= 1.2.3, Stack will refuse to build it with foo-1.3 (unless allow-newer is used).
2. Implemented the newly discussed behavior of a soft bound, and completely ignore ^>= bounds. Note that this is in complete contradiction of the Cabal docs themselves, but appears to be the intended direction for the future.
3. Disallow usage of the ^>= operator entirely. This may sound nice as a stop-gap until this situation is sorted out upstream, but Hackage is already allowing packages to be uploaded with this new operator, so such a move will disallow their usage with Stack.
4. Choose one of (1) or (2) as the implementation, but include a warning in Stack for usage of the ^>= operator indicating that the semantics of the operator may change in a future release.

I'm opening this issue on the Stack issue tracker, but it also impacts Stackage. In the Stackage case, it's easier to argue for a ruling of (3), since we already impose restrictions on which packages from Hackage are accepted.

I _think_ my preferences are:

• Stackage goes with (3). Note that, currently, Stackage disallows Cabal file format 2.0 entirely, so it's a moot point until we relax that restriction.
• Stack goes with the warning (4) on top of the behavior of (1) (respecting the bounds). It seems less confusing to turn failing build plans into successful ones in a future release than the other way around.

My feelings aren't strong on this though, I'd love to hear other thoughts.

[Blaisorblade]

I 👍'ed this (weakly), meaning: I wasn't aware of the discussion but your argument's convincing.

[snoyberg]

The fact that almost no one is aware of the discussion is a big problem, major changes to the way we do versioning should have more community awareness around them. But it's not my discussion to try and open up.

[tfausak]

For reference, the Cabal docs describe "soft" bounds as:

The solver will attempt to satisfy these preferences on a “best-effort” basis.

The PR that introduced ^>= (https://github.com/haskell/cabal/pull/3705) doesn't mention soft bounds.

A more recent PR (https://github.com/haskell/cabal/pull/4575) added the ability to ignore only ^>= bounds when using --allow-newer.

Anyway, all that to say: I can't find any stated direction for the ^>= operator and soft bounds in Cabal. As far as Stack/Stackage is concerned, I support option 1 (hard bounds). It matches the colloquial explanation of the operator (^>=x.y.z = >=x.y.z && <x.(y+1)), and it's easy enough to relax the bounds in the future if desired.

[Blaisorblade]

Please don't count me as "not aware" (I had seen and ignored rumours on Twitter, maybe your feed? — also not Haskelling enough right now) but feel free to use more solid evidence :-). OTOH, you made me curious.

OTOH, the plan is actually semi-documented. Let me quote the Cabal 2 changelog on this feature: http://coldwa.st/e/blog/2017-09-09-Cabal-2-0.html

New caret-style version range operator ^>= (#3705) that is equivalent to >= intersected with an automatically inferred major upper bound. For example, foo ^>= 1.3.1 is equivalent to foo >= 1.3.1 && < 1.4. Besides being a convenient syntax sugar, ^>= allows to distinguish “strong” and “weak” upper bounds: foo >= 1.3.1 && < 1.4 means “I know for sure that my package doesn’t work with foo-1.4”, while foo ^>= 1.3.1 means “I don’t know whether foo-1.4, which is not out yet, will break my package, but I want to be cautious and follow PVP”. In the future, this feature will allow to implement automatic version bounds relaxation in a formally sound way (work on this front is progressing on matrix.hackage.haskell.org). See this section of the manual for more information.

Note the link is to a different section of the manual, and it describes behavior 1 here (no hint about future changes).

OTOH, while that post (from a Cabal maintainer) is the "Cabal 2 release announcement", I could only find it from /r/haskell. Took me ~30 actual minutes. I had even seen it once before. So yes, I understand why most people haven't seen this.

I've pinged people on https://github.com/haskell/cabal/issues/4807 and https://github.com/haskell/cabal/issues/4013.

Last, available public discussion of ^>= is here:
https://www.reddit.com/r/haskell/comments/6z2gja/whats_new_in_cabalcabalinstall_20_improved/dmsa6ch/

I guess people are busy doing feasibility studies on their plans, that's why they aren't public—which is fine. Hopefully people will get to discuss what to do when results are in.

OTOH, with the current docs Cabal 2 will work as an "extended beta" because docs are limited, which is not the worst result (though it doesn't seem intentional) till Nix-style builds are working.

[snoyberg]

Thanks for the release announcement link, that was the first pubic mention of the soft bound concept I've seen.

[23Skidoo]

Treating ^>= x.y.z as a syntax sugar for >= x.y.z && < x.(y + 1) should be fine, now and in the future. I do not think that a warning is necessary, but it's up to you.

Once automatic version bound relaxation is implemented, I think that you can treat it in the same way you treat manual version bound updates via .cabal file revisions, since the information about which bounds are safe to relax in practice will come from an external source. However, it's a bit early to worry about that.

[snoyberg]

Thanks for the feedback @23Skidoo. I'm pretty confused about what the plans are around the ^>= operator, and the comments you've just provided don't really clarify anything to me. Documentation, code, and private discussions all describe it as anywhere from syntactic sugar for existing bounds to something brand new which will be treated differently by different kinds of tooling. I'm not very excited about the idea of accepting and treating this syntax as a hard bound today, when there is active discussion around changing its semantics in a way that will cause changes to future versions of Stack and affect the Stackage build process.

In other words, I disagree with the idea that "it's a bit early to worry about that." Frankly, before release was the right time to worry about this.

[Blaisorblade]

Seems like the semantics (however open) are nothing different from Cabal package updates, as mentioned; in fact updates from soft bounds are more limited (since you opt in in advance and only allow relaxing bounds). I guess the open problem is "how do we check a bound is safe to relax". It's not enough to check if a package A still builds after relaxing the dependency on B, one must check if the API is still compatible (PVP-wise). I can imagine counterexamples using CPP (and questionable choices); more examples might be possible with typeclass instances.

[23Skidoo]

@tfausak

The --preference flag you linked to is unrelated to the caret operator and in fact predates it, it's basically like --constraint, but there is no guarantee that the resulting install plan will satisfy the soft constraint. Thanks for mentioning it, I'll think about how to rephrase that section to avoid confusion with soft upper bounds.

@snoyberg

I'm pretty confused about what the plans are around the ^>= operator, and the comments you've just provided don't really clarify anything to me. Documentation, code, and private discussions all describe it as anywhere from syntactic sugar for existing bounds to something brand new which will be treated differently by different kinds of tooling.

The "syntax sugar" description refers to how the caret operator works in Cabal/cabal-install 2.0 today. Interpreting it in this way is guaranteed to continue to be valid in the future.

The "automatically inferred upper bound" description is also valid today -- in that sense, there will be no changes to the semantics of the caret operator in the future. However, current implementation is able to infer only trivial upper bounds (i.e. < x.(y+1) for ^>= x.y.z).

The long-term plan is to implement more advanced automatic inference of upper bounds -- releasing text-1.3, for example, shouldn't result in a cascade of manual version bound updates in every package that has a text ^>= 1.2.x.y dependency (text >= x.y.z && < 1.3 will continue to work as before). This will require a server-side solution -- once text-1.3 comes out, a build bot will try to build each reverse dependency of text with --allow-newer=^text --constraint="text ==1.3" (this means "use text-1.3 and disregard upper bounds in all ^>=-style dependencies on text"). Then the information about which packages can have their upper bounds on text safely bumped will be published on Hackage.

The reason we need new syntax to distinguish between weak (caret-style) and strong (<) upper bounds is to allow package authors to opt out of automatic upper bound inference and to limit the search space.

So to me (and to @Blaisorblade, it seems) this future scheme looks not much different from .cabal file revisions, which are already supported/worked around by Stack, but maybe there are additional complications I'm missing.

[tfausak]

That makes sense and is about what I expected based on https://github.com/haskell/cabal/pull/4575. It sounds like the Hackage matrix builder will make use of relaxing ^>= bounds to try out install plans. If the plan works, a Hackage trustee will revise the package to loosen the bounds. Is that about right?

Also, does that mean that the ^>= operator also implies soft lower bounds? It seems like it does since you can say --allow-older=^text. I'm not quite sure what to make of that.

[23Skidoo]

@tfausak

It sounds like the Hackage matrix builder will make use of relaxing ^>= bounds to try out install plans. If the plan works, a Hackage trustee will revise the package to loosen the bounds. Is that about right?

More or less, though the process will likely be fully automatic and it is not certain that the existing .cabal revision mechanism will be used, maybe it will be a separate DB.

edit: It will most likely work via .cabal file revisions.

you can say --allow-older=^text

So this will transform text ^>= 1.2 to text < 1.3, but will not affect text >= 1.2 && < 1.3. IIRC caret modifier support for --allow-older was implemented just for symmetry with --allow-newer and because it was easy to do.

Also, does that mean that the ^>= operator also implies soft lower bounds?

I'm not convinced that it is a good idea, I think that behaviour would be confusing.

edit: I talked with @hvr, and he thinks that automatically relaxing lower bounds will be also feasible, since the machinery required for that is essentially the same as for relaxing upper bounds.

[snoyberg]

Thanks again for clarifying @23Skidoo. My confusion comes from the fact that the private discussions I've been included in have been discussing soft bounds/^>= in the absence of Hackage revisions. Now it sounds like the plan is to make the caret operator only special for the purposes of the build bot, and only to be exposed to users vis-a-vis revisions. I'm not commenting on whether or not that's a good plan, but it's drastically different from the plans that have been shared with me until now.

[23Skidoo]

I talked with @hvr about future plans for automatic bounds relaxation, and while the full solution will have some additional parts (like a tool for PVP compliance checking), the big picture I described in the comment above is basically correct. I also updated my reply to @tfausak above.

[tfausak]

Sounds like Hackage will treat ^>= as discretionary bounds. ^>=1.2.3 "means" >=1.2.3 && <1.3, except that Hackage might automatically revise your package to raise the upper bound or lower the lower bound. Some package foo-1.2.3 with a dependency on bar ^>=1.2.3 could instead end up having bar >=1 && <2 after the matrix is done with it. I can't say I'm thrilled about that.

However, it sounds like Cabal will actually treat ^>=1.2.3 as a sugary form of >=1.2.3 && <1.3 (ignoring --allow-{newer,older}). That tells me that Stack should do the same. Stackage could do something else with ^>= bounds if it wants to. (For example, it could try relaxing them and rebuilding before bugging maintainers about restrictive bounds.)

[snoyberg]

Alright, I'm going to close this issue and leave the behavior as is currently implemented (doing whatever Cabal-the-library does). If there are changes in the future, it will affect the entire ecosystem, so may as well join the crowd.