Stack: Add support for cabal custom-setup

[mgsloan]

Currently we have explicit-setup-deps as a hack to say "Use the package's deps to build Setup.hs". It'd be much more principled to use the new custom-setup / setup-depends stuff.

• [ ] Update library dep to Cabal 1.24
• [ ] Figure out how it works. Are there docs? Hopefully it's just a component stanza..
• [ ] Re-use the same logic used for generating ghc arguments for stack ghci

Split off from https://github.com/commercialhaskell/stack/issues/2093#issuecomment-216644420

[rcook]

+1: The various gtk2hs projects (https://github.com/gtk2hs/gtk2hs) make extensive use of Setup.hs along with custom-setup and setup-depends and are quite tricky to build with Stack as a consequence.

[mgsloan]

Hmm, so now gtk3 can only be built with the very latest cabal?!

The cabal file could be modified to work with explicit-setup-deps. The library deps just need to be a superset of the Setup.hs deps. So in https://github.com/gtk2hs/gtk2hs/blob/master/gtk/gtk3.cabal, gtk2hs-buildtools needs to be a dep of the library. I can't think of many reasonable cases where this solution could be applied.

So basically, right now cabal has its way of addressing this (which we should probably add support for), and stack has its way of addressing this. Gtk2hs currently does not define things such that we can make them work via explicit-setup-deps.

[rcook]

Yes. I already used this workaround here: https://github.com/rcook/gtk-mac-integration/tree/p-rcook-stackify-1. It works for now!

[hvr]

fwiw, the semantically wrong hack of adding gtk2hs-buildtools to build-depends won't work anymore with future cabals, as build-depends only requests lib: components, but no exe: components.

[garetxe]

I am running into this issue with the gobject-introspection based bindings for gtk: https://hackage.haskell.org/package/gi-gtk .

The bindings are generated at cabal configure time using the haskell-gi dependency. In order to make this work with cabal new-build one needs to specify the haskell-gi dependency in custom-setup and set the cabal-version field to >= 1.24.

But then stack build rejects this, with

setup: This package description follows version 1.24 of the Cabal
specification. This tool only supports up to version 1.22.8.0.

So as far as I have been able to tell so far, the current situation is that I am forced to choose between supporting stack and supporting cabal new-build.

My problem could be worked around by stack accepting newer versions of the cabal spec, without really supporting them, but better would be to actually support the custom-setup syntax.

[rcook]

@garetxe: You can upgrade the version of Cabal used by Stack using stack setup --upgrade-cabal. This will upgrade to Cabal-1.24.0.0: this is how I was able to get gtk-mac-integration building (https://github.com/rcook/gtk-mac-integration/blob/p-rcook-stackify-1/gtk3-mac-integration.cabal).

[hamishmack]

I have prepared an update to Gtk2Hs that I hope will address the concerns @hvr raised. It will still use custom-setup, but everything will be in the library component of gtk2hs-buildtools.

[garetxe]

@rcook Thanks for the suggestion. But my issue is how to distribute the package reliably to end-users, not so much how to build it myself. Unless I misunderstand, I would have to ask every user of the library to run stack setup --upgrade-cabal before being able to use the libraries in stackage, which is far from ideal.

Anyway, so far I could not figure out how to make all of stack build, cabal build and cabal new-build work simultaneously, so I worked around this by making the stackage version have the explicit-setup-deps workaround, and releasing a newer version in hackage that does not have the workaround, and rather uses the new custom-setup syntax.

[garetxe]

Ah, I just saw that stackage nightly moved to cabal 1.24. So maybe now it is possible to support all options.

[mgsloan]

Ah, I just saw that stackage nightly moved to cabal 1.24. So maybe now it is possible to support all options.

Does Cabal 1.24 support an explicit way of building Setup.hs? If not, then we'll need to do the things I listed when this issue as created.

I've requested that Cabal add this to its CLI: https://github.com/haskell/cabal/issues/3471

[garetxe]

@mgsloan Sorry for not being clear. I just meant that for a stack compiled against Cabal-1.24 it may be possible to add a workaround (in my own packages) for the lack of support for custom-setup in stack while keeping support for cabal new-build. As it is right now, I have to maintain two versions of every package using custom-setup, one that works with new-style cabal and one for stack...

I was not commenting on how easy or hard it would be to implement proper support for custom-setup in stack.

[mgsloan]

@hamishmack , it looks like that didn't work - https://github.com/commercialhaskell/stack/issues/2512

Is it because stack doesn't have custom setup deps yet? Please add it back to build-tools, I'm not sure when we will add support for custom setup deps.

[mgsloan]

This should actually be quite easy to implement. As described in the original post, it will look like including the setup deps as part of the deps of building the package. Then, the deps just need to be passed in while building Setup.hs.

Want to take it on @Blaisorblade ? I can do it as well, but looks like you're digging into this.

[Blaisorblade]

Want to take it on @Blaisorblade ? I can do it as well, but looks like you're digging into this.

I'm taking a look... but unlike my other fixes, this requires actually getting Stack internals. Status:

• add setup-deps together with the other package dependencies, though it's not perfect (https://github.com/commercialhaskell/stack/issues/2515#issuecomment-241196356).
• then, try reusing explicit-setup-deps—though its logic seems rather confusing (I thought it reused the same dependencies as the library, but it doesn't).
• then, tried following a bit Ghci.hs, even though for quite a few pieces of the context it's unclear what I should exactly take.

EDIT: neither approach works (as implemented). Stack currently takes lots of care to stick to the Cabal version from the global package database ("for reproducibility"), and that's used in lots of places. Now instead Stack must use instead Cabal from setup-depends (if specified), which currently means from the build plan.

[mgsloan]

Stack currently takes lots of care to stick to the Cabal version from the global package database ("for reproducibility"

It's really the opposite, instead of reproducibility, flexibility

Now instead Stack must use instead Cabal from setup-depends (if specified), which currently means from the build plan.

Hmm, yeah, that is tricky. How about instead:

1) Use global Cabal for simple setup (for flexibility / ability to upgrade. Worth the inconsistency?)

2) Use environment Cabal for custom setup

This'd potentially be a breaking change.

[Blaisorblade]

A Simple setup literally works for any Cabal version I guess, so it could also use the environment cabal. What's upgrading needed for.

I confess I'm not sure about all the involved tradeoffs—how many Cabals are involved and which ones must match? Are there docs?

What I understand is that we have versions from stack, GHC and setup, and they seem to be all independent. stack should use the latest to support newer features if any, and GHC and setup versions vary independently for flexibility. I suppose something has to match to be able to save the LocalBuildInfo in dist/setup-config, but not sure what.

That doesn't account for --upgrade-cabal, but what's the real use case for it if you can pick the version your setup needs, and if stack uses the latest cabal? It seems the only use case for --upgrade-cabal is breaking setup scripts that don't compile with newer cabal versions—I bet for instance base-noprelude wouldn't be installable.

And in general, I'm not sure what the user stories to support are, and how many of those make sense nowadays.

[mgsloan]

In current stack, there are 3 cabal versions to consider:

1) The version in the global DB, which is used to build Setup.hs for both simple and custom.

2) The version in the project's DB (snapshot or local), which is the one packages will be built against. This means that there's a diamond dep problem that can cause some Setup.hs not to build. Happily it usually doesn't seem to be a big problem.

3) For --solver, sometimes the version of the cabal-install executable on the PATH is relevant.

[Blaisorblade]

In case there's any doubt, I didn't make much progress and I'm unlikely to tackle this in the near term, sorry.

[mgsloan]

@snoyberg actually already implemented this - #2866, so no worries @Blaisorblade !

[fumieval]

gl doesn't build with lts-9.0, probably because it seems to ignore setup-depends.

--  While building package gl-0.8.0 using:
      /Users/fumieval/.stack/programs/x86_64-osx/ghc-8.0.2/bin/ghc --make -odir /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup -hidir /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup -i -i. -package=Cabal-1.24.2.0 -clear-package-db -global-package-db -package-db=/Users/fumieval/.stack/snapshots/x86_64-osx/lts-9.0/8.0.2/pkgdb /Users/fumieval/hs/gl/Setup.hs /Users/fumieval/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs -main-is StackSetupShim.mainOverride -o /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup/setup -threaded
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/fumieval/hs/glassy/.stack-work/logs/gl-0.8.0.log

    [1 of 7] Compiling Utils            ( Utils.hs, /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup/Utils.o )
    [2 of 7] Compiling Registry         ( Registry.hs, /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup/Registry.o )
    [3 of 7] Compiling Parser           ( Parser.hs, /Users/fumieval/hs/gl/.stack-work/dist/x86_64-osx/Cabal-1.24.2.0/setup/Parser.o )

    Parser.hs:18:1: error:
        Failed to load interface for ‘Text.XML.HXT.Core’
        Use -v to see a list of the files searched for.

After adding hxt to build-depends in the library, it succeeded.

[snoyberg]

@fumieval I cannot reproduce the issue. Please ensure that you are using the latest Stack (which supports setup-depends), and if still present open a new issue with all relevant information (such as OS, Stack version, etc).

[fumieval]

@snoyberg Sorry, my stack command was pointing to the old version. Latest stack seems to work

[nh2]

Sorry, my stack command was pointing to the old version

I also just got tricked by that; I ran stack upgrade but due to shell path hashing (see here, here), the globally installed stack in my docker container got used.

hash -d stack fixed it.