cabal-install doesn't consider build-tools as dependencies

[bos]

Summary by @ezyang. When you write build-tools: alex, Cabal only checks that an executable of this name is installed; cabal-install does not know that if there is no such executable, alex needs to be installed. One complication is that build-tools does not necessarily correspond to a package, so it is not a priori obvious to cabal-install what should be installed to get an executable.

Here is Proposal A: Build-tools only

• We introduce a new type of qualified goal, a build-tools goal, similar to how setup qualified goals work.
• We modify the solver to interpret each field of build-tools as a package name, and we solve for each build-tool as a separate build-tools dependency (so we don't do something goofy like try to solve for happy and alex's dependency sets consistently when they really couldn't care less.)
• Some build-tools fields don't correspond to actual packages. We give two escape hooks in this situation. First, Cabal has an internal mapping of build tools to Maybe PackageName. If Nothing, this indicates that the build-tool is not managed by Cabal and we shouldn't try to solve for it. If Just pn, then it means that installing pn will also install the requested executable. This mapping will be configurable using the build-tool-packages field in a cabal-project and also modifiable by command line argument. (Names can be bikeshedded of course.) Our second escape hook is, if a build-tool refers to an unknown package name, we print a warning but continue to solve, simply ignoring that package. A user can suppress the warning by disabling this using the aforementioned flag.

Consider these examples:

• build-tools: mytoolpkg: cabal-install will install all executables defined by a package named mytoolpkg, and ensure they are available prior to building this package.
• build-tools: gtk2hsc2hs: cabal-install is hard-coded to know that the gtk2hsc2hs executable is provided by the gtk2hs-buildtools, so it ensures that it builds and installs all executables from the package before building this package.
• build-tools: someprog, plus a flag --provides-build-tool someprog=somepkg, cabal-install sees someprog (which has no package), but using the flag mapping it knows that somepkg is the package that provides this build tool.

Here is Proposal B: tool-depends

• We introduce a new type of qualified goal, a build-tools goal, similar to how setup qualified goals work. (Like before)
• We introduce a new tool-depends field, which consists of pkg:exename version-constraints specifications, e.g., tool-depends: gtk2hs-buildtools:gtk2hs2hs < 2.0. This means that we must build the gtk2hsc2hs executable from gtk2hs-buildtools before building this package. In principle cabal-install need only build the gtk2hsc2hs executable, although today's setup interface doesn't allow one to selectively enable executables, so we'd probably just build all executables. If qualification is omitted it is assumed that you need ALL executables from the package.
• build-tools refers exclusively to non-packaged executables, e.g. provided by the system. However, for backwards compatibility, we introduce an internal mapping hard-coded into GHC which maps some build-tools fields to package names. The intended interpretation is that build-tools: happy > 0.4 actually elaborates to tool-depends: happy:happy > 0.4. This mapping will be configurable using the build-tool-packages field in a cabal-project and also modifiable by command line argument. (Names can be bikeshedded of course.)

Examples:

• tool-depends: mytoolpkg: cabal-install will install all executables defined by a package named mytoolpkg, and ensure they are available prior to building this package.
• tool-depends: mytoolpkg:mytool: cabal-install will install the executable mytool from mytoolpkg before building this package
• build-tools: happy: this is equivalent to writing tool-depends: happy:happy.
• build-tools: someprog: as before this does not effect solver behavior; there just simply needs to be a someprog in the path.

---

(Imported from Trac #227, reported by guest on 2008-02-01)

On #haskell, nelhage mentioned that one of the bugs he ran into while trying to install Yi-0.3 through cabal-install was an error which looked like this:

Installing: /home/gwern/.cabal/lib/regex-posix-0.72.0.2/ghc-6.8.2/regex-posix-0.72.0.2/ghc-6.8.2
Registering regex-posix-0.72.0.2...
Reading package info from "dist/installed-pkg-config" ... done.
Saving old package config file... done.
Writing new package config file... done.
'yi-0.3' is cached.
[1 of 1] Compiling Main             ( Setup.hs, dist/setup/Main.o )
Linking dist/setup/setup ...
Configuring yi-0.3...
setup: alex version >=2.0.1&&<3 is required but it could not be found.

That is, the dependency on Alex (a version which satisfied >=2.0.1&&<3) was not met. Perfectly sensible as he did not have Alex installed. Puzzled, he did a 'cabal install alex', which worked, and then re-started. (I guess he had his GHC installed locally?).

He and I were puzzled, because cabal-install is supposed/usually does track down dependencies. Is build-tools excluded for a reason, or was it just accidentally omitted from dependency-tracking? If the latter, I think this is a bit of a bug.

(I use Cabal-1.2.3.0, btw, and the error message above was my reproduction of Nelhage's problem. I had to uninstall Alex from my system, but that did the trick, and reinstalling bypassed the issue just fine - which is why I hadn't seen it before.)

gwern

[bos]

(Imported comment by @dcoutts on 2008-02-01)

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

[bos]

(Imported comment by @dcoutts on 2008-02-28)

This limitation clearly confuses people.

• [http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html](http://www.haskell.org/pipermail/cabal-devel/2008-June/003220.html)
• [http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html](http://www.haskell.org/pipermail/haskell-cafe/2008-June/044222.html)

[bos]

(Imported comment by darrint on 2008-06-16)

Seems like what you need is to be able to determine which packages provide which executables. Alex, for instance, declares that it provides an executable by that name. For the next release of cabal-install, maybe change the package repository metadata format to include more (all?) the cabal info instead of a file list. That will allow you to solve more situations and do a full install plan up front.

Would that work?

[bos]

(Imported comment by @dcoutts on 2008-06-16)

See also #342

[bos]

(Imported comment by @mcandre on 2009-01-05)

I have the same problem with ncurses and c2hs. I'd really like dependencies to be handled automatically whenever possible.

[mietek]

Looking forward to this being fixed.

[tibbe]

@mietek right now this is blocked on no one having the time to work on this. If you're interested perhaps you could take a look? If you have any questions you can write to [email protected].

[mietek]

@tibbe: I can't right now, but I am interested in solving this, in one way or another.

[TheunisKotze]

:+1:

[mietek]

Halcyon supports declaring and automatically installing build-tools and other Haskell apps for use at build-time with the HALCYON_SANDBOX_EXTRA_APPS option. Version constraints for these apps can be declared with the HALCYON_SANDBOX_EXTRA_APPS_CONSTRAINTS option.

Similarly, you can use HALCYON_EXTRA_APPS and HALCYON_EXTRA_APPS_CONSTRAINTS to declare additional apps to be installed for use at run-time.

See Haskell Language for an example of declaring _alex_ and _happy_ as sandbox extra apps.

[danse]

This is quite a big flaw. I am an Haskell newbie, but i will try to put my two cents in order to keep the discussion alive

One problem is that not all build-tools correspond to haskell packages. Some do some don't. We have a hard coded list of them at the moment (which can be extended in Setup.hs files) so we could extend that with what haskell package if any the tools correspond to. Any better suggestions to make it a tad more generic?

What is the purpose of build-tools if not to list items to be installed? Correct me if i am wrong: if a build tool is declared as a build dependency in build-depends, it will be correctly installed, but if it is listed within build-tools it might be ignored?

[oneeyedziggy]

"Every little bit helps," said the gnat and it pissed in the sea.
here's my little bit... I'm also a first timer like danse above (who i suspect is going to OsCon 2015), this kind of puts a sour taste in the mouth of anyone trying to pick up haskell...

[hvr]

IMHO to keep this KISS-y, build-tools should be taken into account when solving for an install-plan, and try to find an install-plan that is possible with the currently available tools. I.e. consider build-tools a static (non-upgradeable/installable) property of the build-environment (like e.g. the GHC version -- which ultimately is a build-tool as well).

If, at some point in the future, we want to support auto-installing build-tools, we'd have to massively extend cabal-installs support for tracking package executables.

/cc @kosmikus

[mietek]

@hvr’s suggestion would certainly make the experience less frustrating.

As it is, you’ll only find out you’re missing a build-tool halfway through building your dependencies — and that’s if you’re paying attention, because https://github.com/haskell/cabal/issues/1924 means unless you manually abort the build, you’ll continue burning CPU cycles on something which has no chance of succeeding.

[ezyang]

CC @hvr

I implemented this in #3662 and I realized that we'll need to change build-tools in a bit more major way than I originally suggested in proposal A. So, in no particular order, here are things I observed:

1. Today, most names you put in build-tools are rejected by Cabal, saying "Unknown build tool blah". The reason for this is Cabal has a ProgramDb which exhaustively enumerates all of the build-tools that Cabal actually knows about. Today, the only way to extend this database is to write a Custom setup script that sets hookedPrograms to add some extra programs to the database (this is dumb, by the way). So, what I decided to is this: IF an executable in a build-tools is not already in the program database, we attempt to configure it as a SimpleProgram, instead of immediately giving up. (But maybe this is wrong and counter to the intent of the people who wrote this interface originally.) One problem with doing it this way is that the program database is not serialized to disk (to my knowledge), so the program won't be known in subsequent invocations. This is not too much of a problem at the moment because you never actually try to invoke the program from Cabal; it's always done through a PATH invocation, e.g., a GHC flag (perhaps another indication that doing ProgramDb is the wrong thing.)
2. OK, so that's nice, but now there's a second problem: the executable you want from a package may NEVER coincide with the package name. An example is cabal, from the cabal-install package. So you could say build-tools: cabal-install which will make the dep-solver happy, but then Cabal will say that it can't find any executable named cabal-install. And if you say cabal, well, someone needs to say what package it comes from, and _clearly_ (except for legacy cases) this information should be in the Cabal file, not some user configuration.) So there really needs to be syntax as flexible as tool-depends (from proposal B) to finger the specific executable you want.
3. Given that you can finger the correct executable, I don't see any reason for the "legacy mapping" to be configurable outside the Cabal file: putting it in the Cabal file is basically always better. It only poses a problem for backwards compatibility, but the way to handle this is to NOT extend the build-tools syntax, and instead provide package name to executable mappings out of band (in a field that old versions of Cabal can ignore). So, something like tool-depends, but just specifying a bunch of packages which you should look at to find the executables you actually want. Perhaps you could even have thinning and renaming Backpack style.

[hvr]

as we discussed on IRC, it turns out that "proposal B" is starting to look a lot more attractive given the pointed out issues, as requires less confusing magic/fallback tricks