Cabal: Cabal 2.0 release planning

[ezyang]

In https://github.com/haskell/cabal/issues/3545#issuecomment-233979093 @23Skidoo suggested that we should cut Cabal 2.0 next release. The primary implication of this is that the default Cabal upper bound for setup dependencies will EXCLUDE this new version: this means backwards incompatible changes can be made without instantly breaking old Custom Setup scripts.

Of course, that doesn't mean we should go breaking backwards compatibility will nilly. This is a tracking ticket for specific incompatibility we plan to introduce in 2.0, and the motivation behind it. Please edit with other instances.

• 62c3aa627c99aefb09605bb59507b925151625dc configPrograms from ProgramConfiguration to Last' ProgramConfiguration @hvr (subsequently undone in 0077e2c476a8f7a6fa5b86bd70b408de50d92568)
• e1c39fc9ff4bf4eb8e956e2ee77da95d6cc73bbe drops GHC 6.12 and earlier support @ezyang
• cc63fbf9ac5e4f033c943a4395f7c3e1359a05c2 adds PACKAGE_VERSION macro for currently being compiled package itself @erikd
• Convenience libraries #3574 @ezyang see 2040c1c937956f7bbb9be61e2a655211a862db38 90e908b80b327c4636c21627b0e2a54a6a8d4e4f 3e781503df84ebbcd40523d608e7dfe5dca95852 (undone in )
• 4260a385d80a8f5140fb80190bd7607d9ce19f0f z- prefixed packages are now reserved @ezyang
• f88f502d85b60b7ca5993fdccfa069d7b313e174 CopyFlags has new copyArgs field @ezyang
• 40d6f0afd5b86d698abb876d4f5ed0bb452b0c4b registerPackage takes ADT MultiInstance rather than Bool @ezyang
• db9ccb5b2ba5f7c7ca1ca4fdbd1ccc820d5ff6a9 OriginalModule is now called Module @ezyang
• bddc053a1b9e13c8db37f13f0437534ca81cc929 componentPackageRenaming is now called componentIncludes @ezyang
• b946c05e749fb0acb78ed8ae135b98e576df9d55 Windows short path names @23Skidoo
• --assume-deps-up-to-date flags @ezyang 85a76c9dc45808abe5856960b64e1b5ce5b7f716 1d27ac7118844e4c161b7817675645073f1cb359 2fac2ec6a7ae87d965375a55c997e8f8286a4d86
• 58b4a6512269480998067bdbb82a0c110a89ce57 SetupBuildInfo gets defaultSetupDepends field @23Skidoo
• 0252a0aed79e881d4f6210d9296216aef3c301d0 Warns if a package has no exposed modules @23Skidoo (e9883dced04e6a16e25752cc8513ce26325b4b4b)
• 09753724b8f2e90ce25bae1af1995e4d279ab760 hscolourPref/haddockPref takes extra HaddockTarget, haddockForHackagetype changed toFlag HaddockTarget` @bennofs
• b9fdc1bf2b131b4c02d1e1af36e9710da2f58ac5 AllowNewer generalized into RelaxDeps @hvr
• #3595 LocalBuildInfo refactor @ezyang

Planned work:

• #3600 hook redesign
• #3578 new test interface

[acfoltzer]

What's the timeline expected on this? Such a major version change would perhaps be a good moment to replace build with new-build, but I expect that's more backwards incompatibility than you have in mind in the near term.

[ezyang]

Well, we'd want to hook this up with GHC 8.2. However note that s/new-build/build/ is a _cabal-install_ change, and so the setup upper bounds is not applicable. cabal-install can break its outwards interface whenever it wants to distinct from the library.

[23Skidoo]

Yeah, I don't think nix-local-build will be completely ready for prime time in time for the next release. I plan to cut the next release in October/November, so most likely it won't be synced with GHC 8.2 (but the next release after that will).

[ttuegel]

Will there be a Cabal 1.26 release? In other words, should we avoid pull-request-ing branches that are intended for Cabal 2.0?

Is there a deprecation procedure for things we'd like to remove from Cabal 2.0?

[ezyang]

I'll defer to @23Skidoo on the Cabal 1.26, but given how early convenience libraries were merged into master this cycle, it probably makes more sense to backport useful fixes to 1.24. (As in, master already has a number of BC incompatibilities that are best put into 2.0) So I think PR'ing Cabal 2.0 branches should be OK.

I wrote deprecation/BC guidelines in #3573 but amendments welcome.

[ttuegel]

I left a suggestion on the deprecation guidelines, but I still have some question about Cabal 2.0 in particular. I'm thinking of the hooks interface here. It would be very convenient to clean that up for Cabal 2.0; given that we will need an upper bound on Cabal for old setup scripts anyway, can we break the hooks interface?

[ezyang]

I am open to fixing the hooks interface, but only if it's very clear what _specific_ problems with the existing hooks interface we are fixing, and how the new design addresses them. I don't have a good idea what an improved hooks interface looks like. And we'd need docs for how to migrate to the new interface, or a shim, etc.

[23Skidoo]

My proposal was to call the next release 2.0, i.e. release 2.0 _instead of_ 1.26. Since we seem to agree that this is the way to go, I'll rename the milestone to avoid confusion.

[hvr]

I was expecting we'd bump to 2.0 to mark the nix-local-build transition, since that's gonna be a very visible and disruptive UI change (as well as some subtle semantic changes regarding the meaning of .cabal files). Can we then use a bump to 3.0 instead to advertise making new-build the new default, even if that means that we'll have a quick succession of 1.x -> 2.x -> 3.x versions?

[23Skidoo]

@hvr

Can we then use a bump to 3.0 instead to advertise making new-build the new default, even if that means that we'll have a quick succession of 1.x -> 2.x -> 3.x versions?

Sure.

[dcoutts]

Yes! I've been suggesting the same. And yes, to clarify this is the Cabal library API.

Yes I'd suggest the terrible hooks API as something we should break. There's a bunch of other TODOs/FIXMEs in the code that are there because we could not yet break the Setup.hs API.

[ezyang]

OK, fine by me.

[ezyang]

@ttuegel re the hook redesign, please look at the concern raised by this ticket #2910. Any new hook interface needs to accommodate the fact that hooks are best applied at the component level.

[dcoutts]

Also ought to remove overall buildDepends field from PackageDescription

[Ericson2314]

This, is super drastic, but what if we _removed_ custom setup until we found a better design (IMO additional shake rules or something else declarative)? That could maybe help avoid 2->3->>? major version churn as these things get worked out.

[ezyang]

The downside of dropping Custom script is that any packages which use Custom scripts today (and there are multitudes) will be forced to use an old version of the Cabal library and will not get any future bugfixes. The payoff doesn't seem so good: it is true that now you can change the Cabal API without worrying about BC, but if you are going to reimplement Cabal with Shake you are probably going to end up with a complete rewrite, so no point in reusing the old code.

On September 20, 2016 9:13:09 AM GMT+09:00, John Ericson [email protected] wrote:

This, is super drastic, but what if we _removed_ custom setup until we
found a better design (IMO additional shake rules or something else
declarative)? That could maybe help avoid a 2->3->>? major version
churn as these things get worked out.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/haskell/cabal/issues/3576#issuecomment-248165924

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Ericson2314]

That a convincing counter argument :).

I'd still like to get people off custom installs and rethink the relationship between Cabal and cabal-install (and stack)—Cabal's remaining users. But more version bumps is a small price to pay and that can still be schieved more slowly.

[ezyang]

Short term, the best way to get people off Custom setups is to (1) add features to Cabal build system which subsume the use cases people use for Custom (see also #3788), and (2) make it possible to version-gate a Custom setup so that if the Cabal library is sufficiently new you don't do anything non-standard (maybe a pile of ifdefs is enough, though you don't get per-component builds that way.)

[bgamari]

What is the status here? GHC 8.2 is quickly approaching; is the Cabal tree currently near a release-able state?

[ezyang]

My read on the situation is that the Cabal tree is releasable today, we have not done any "break the tree pending some future change" changes.

[23Skidoo]

Yep, I think it's time to create the 2.0 branch.

[bgamari]

Great news. It would be great if someone could ping me when this happens.

[23Skidoo]

@bgamari Sure, I will. How quickly is GHC 8.2 approaching? Are the dates in https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-8.2.1 accurate?

[bgamari]

@23Skidoo yes, ideally we would have a final release tag for Cabal by late February.

[bgamari]

How close are we to cutting a 2.0 branch at this point?

[23Skidoo]

@bgamari I think I'll do it during the weekend.

[bgamari]

@23Skidoo are we getting close yet? The 8.2 branch is going to be cut any second now.

[ezyang]

The 2.0 branch has been cut. Also FYI, I'd quite like this patchset https://github.com/haskell/cabal/pull/4337 to make it in the 2.0 branch.

[23Skidoo]

@bgamari I'll tag the Cabal 2.0 release next week (there are a couple PRs that need to be merged), for now you can point the submodule at the 2.0 branch.

[bgamari]

What is the plan regarding cutting a 2.0.0 release?

[23Skidoo]

@bgamari In principle can be done right now, but there are some patches that I want to cherry-pick first. When do you need it?

[Ericson2314]

I'd still really like to get the last bit of work for local build-depends syntax in. It's been in limbo as I've been busy in work. There's just a few test failures it might be good to pair program on :).

[bgamari]

@23Skidoo, I would ideally like to have the submodules stabilized for the next RC, which I would like to cut at some point this week.

[hvr]

@bgamari can we make it a more concrete deadline, say "on sunday" or something like that? I keep finding bugs in Cabal... and I still have to finalise a few other ghc submodules too... :-)

[bgamari]

Herbert Valerio Riedel [e-mail] writes:

@bgamari can we make it a more concrete deadline, say "on sunday" or something like that? I keep finding bugs in Cabal... and I still have to finalise a few other ghc submodules too... :-)

Sure, "On Sunday" sounds fine to me.

[23Skidoo]

OK, it'll be Sunday then.

[bgamari]

How are things looking here?

[phadej]

For the future: I'd like to have a feature (and API) freeze at about the same time as GHC releaase is frozen. I won't fix cabal-doctest until there is tag in Cabal.

Sorry for causing inconvenience, but my time is limited, and making compat package for a moving target just won't work.

Should have added Kmett's custom setup script as a test case; didn't think we will break API that much. I'll add it after there is a tag.

[phadej]

ping @23Skidoo

[23Skidoo]

@bgamari @phadej In principle, 2.0 is in releasable state right now, I just want to squeeze in one other small update today or tomorrow.

[phadej]

@23Skidoo Can we have the tag + release tomorrow (on Friday) with or without an update. The "deadline" (Sunday) passed already, the delays in releases/freezes accumulate.

[23Skidoo]

OK, sure.

[bgamari]

Thanks @23Skidoo! Assuming this gets done I think we should have an rc2 by the weekend.

[angerman]

As I was made aware of by @ezyang, cabal release are infrequent. I'd like to see if there is support for adding #4480 (Adding cabal doctest) to the release. I know that @phadej is rather against this. I'd like to still layout my reasoning:

My reason for implementing cabal doctest, was to reduce the number of packages that depend on build-type: Custom just for doctest, as custom setups make cross compiling packages harder. This adding cabal doctest is not the ultimate solution, which is rather to teach cabal about cross compiling. It is rather low hanging fruit that makes cross compiling cabal packages easier, if there is no custom Setup. Of course there are other benefits as well.

Assuming this makes it into the release, cabal doctest will be widely available starting with 8.2, and therefore I will be able to lobby maintainers to drop the custom setups, if they are only required for doctest. So that when 8.4 is released (with the improved cross compilation support), cross compiling will be less painful than it is today. (E.g. fetching packages by hand, rewriting the build-type to Simple and hoping for the best).

[ezyang]

I'll chime in that I would also look positively upon this making it into 8.2. It's a simple matter to look at the commit set and see that it is very low risk: even if the implementation of doctest itself is buggy (it probably is; not enough testing yet ;) it is highly unlikely to affect other functionality, or Cabal API users (the only risk is the extra hooks, but we already encourage people to use emptyUserHooks to be stable over additional hooks.)

[23Skidoo]

Merged doctest patches into the 2.0 branch. The feature seems to lack documentation, which is probably for the better for now.

[23Skidoo]

Merged everything I wanted into 2.0, all that remains now is to wait for CI to become green and create a tag.

[23Skidoo]

Cabal-v2.0.0.0 tag has been created.

/cc @bgamari

[hvr]

@angerman

Assuming this makes it into the release, cabal doctest will be widely available starting with 8.2, and therefore I will be able to lobby maintainers to drop the custom setups, if they are only required for doctest.

If they use my PPA with the new-build powered script generator (which is the only one I'm actively supporting at this point), then I'm afraid you still won't be able to lobby them... for that you'd need cabal new-doctest :-/

[angerman]

@hvr I've been really bad with getting ahead with cabal new-doctest. I don't think I will have a chance at that before July. I guess I have to accept it will have to be a subsequent cabal release then :-/

[BardurArantsson]

(This may be a tangent:)

What are we generally aiming for wrt. release timing? I understand that there's a good reason to synchronize Cabal releases with GHC, but is there anything (technically) preventing more frequent releases of "cabal-install"?[1]

The reason I ask is that several projects have had great success with frequent (e.g. tri-monthly) time-based releases so that if a feature misses its release target, it'll just get pushed to the next (predictable) release date rather than being pushed into some uncertain "some day" future release. Does this sound like something which it might be reasonable to adopt for cabal-install?

[1] I also understand that there's quite a tight versioning relationship between Cabal and cabal-install right now, but that doesn't seem like a technical problem per se.

[hvr]

@BardurArantsson are you thinking about major or minor releases of cabal-install? If it's about minor releases, I think the major thing limiting us (after a new major version has been released) is the lack of manpower to handle the release management which can be time-demanding (testing, reviewing changelog, producing executables, double-checking, ...). If you look at https://matrix.hackage.haskell.org/package/cabal-install you can see there were several releases per major release (for 1.18/1.20/1.22) in the past; it's just with 1.24.* that things slowed down. And also note that Cabal 1.22.* was an "odd" major release that wasn't bundled with any GHC.

[23Skidoo]

GHC 8.2 release got delayed, and cabal-install/Cabal 2.0 got delayed too as a consequence. I think we should release 2.2 ~in the middle of the next GHC release cycle to avoid getting into this situation again. I'd also like us to move to ~six-month release cycles.

[BardurArantsson]

@hvr I really just mean regular, predictable and above all frequent releases.

@23Skidoo Is every six months frequent enough? That still leads to a big (disproportionate) disappointment if $MY_PULL_REQUEST doesn't make it in. I think it's worth aiming for 3 months and then trying to automate things enough that 3-month release cycles aren't overwhelming in terms of work for the Release Manager (i.e. you :) ).

(Don't get me wrong, I get that there are good reasons for the current situation. What I'm asking is if we could change it.)

[23Skidoo]

I think three months is a bit too short given the current level of developer activity.

[23Skidoo]

2.0 is out, I believe this can be closed.