Cabal: Builds depending on cpphs fail occasionally.

[vmchale]

Occasionally, builds with new-build will fail with the following:

cabal: The following package dependencies were requested
--dependency='cpphs=cpphs-1.20.8-df266940be4ee1f49d22445541267998ca9452a1a9acff3e608ae8cee98d4624'
however the given installed package instance does not exist.

Builds with GHC 8.4.3 always work, but builds with past versions (such as 8.2.2, 8.0.2, etc.) will sometimes fail if cpphs has already been depended by a project using GHC 8.4.3.

This works on e.g. shake-literate with GHC 8.0.2, viz.

Resolving dependencies...
Build profile: -w ghc-8.0.2 -O1
In order, the following will be built (use -v for more details):
 - shake-literate-0.1.0.0 (lib) (first run)
Configuring library for shake-literate-0.1.0.0..
cabal: The following package dependencies were requested
--dependency='cpphs=cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf'
however the given installed package instance does not exist.

I'm using cabal-head from the PPA.

cabal-install version 2.3.0.0
compiled using version 2.3.0.0 of the Cabal library

[quasicomputational]

Does, e.g., using 8.2.2 in a clean build and then rebuilding with 8.2.2 always work?

-v output might also be helpful to track this down.

[vmchale]

Here is the verbose output:

this build was affected by the following (project) config files:
- /home/vanessa/programming/haskell/done/shake-literate/cabal.project.local
Build profile: -w ghc-8.0.2 -O1
In order, the following will be built:
 - shake-literate-0.1.0.0 (lib) (first run)
creating
/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/build
creating
/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/tmp
creating
/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/build/x86_64-linux/ghc-8.0.2/shake-literate-0.1.0.0
creating
/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/build/x86_64-linux/ghc-8.0.2/shake-literate-0.1.0.0/cache
Using self-exec internal setup method with build-type Simple and args:
["act-as-setup","--build-type=Simple","--","configure","--verbose=2","--builddir=/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/build/x86_64-linux/ghc-8.0.2/shake-literate-0.1.0.0","--ghc","--prefix=/home/vanessa/.cabal","--bindir=/home/vanessa/.cabal/bin","--libdir=/home/vanessa/.cabal/lib/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0-inplace","--libsubdir=","--dynlibdir=/home/vanessa/.cabal/lib/x86_64-linux-ghc-8.0.2","--libexecdir=/home/vanessa/.cabal/libexec/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0","--libexecsubdir=","--datadir=/home/vanessa/.cabal/share/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0","--datasubdir=","--docdir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0","--htmldir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0/html","--haddockdir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0/html","--sysconfdir=/home/vanessa/.cabal/etc","--enable-library-vanilla","--disable-library-profiling","--enable-shared","--disable-static","--disable-executable-dynamic","--disable-profiling","--profiling-detail=default","--library-profiling-detail=default","--enable-optimization","--disable-debug-info","--disable-library-for-ghci","--disable-split-sections","--disable-split-objs","--disable-executable-stripping","--disable-library-stripping","--package-db=clear","--package-db=global","--package-db=/home/vanessa/.cabal/store/ghc-8.0.2/package.db","--package-db=/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/packagedb/ghc-8.0.2","--flags=-development","--cid=shake-literate-0.1.0.0-inplace","--extra-prog-path=/home/vanessa/.cabal/bin","--dependency=base=base-4.9.1.0","--dependency=cpphs=cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf","--dependency=shake=shake-0.16.4-9e6708c5802ab338c5aace79838c373a8ae081f507beadc3e69a471f220df3be","--disable-coverage","--exact-configuration","--with-ghc=/opt/ghc/bin/ghc-8.0.2","--with-ghc-pkg=/opt/ghc/bin/ghc-pkg-8.0.2","--ghc-option=-hide-all-packages","lib:shake-literate"]
/opt/cabal/head/bin/cabal act-as-setup --build-type=Simple -- configure
--verbose=2
--builddir=/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/build/x86_64-linux/ghc-8.0.2/shake-literate-0.1.0.0
--ghc --prefix=/home/vanessa/.cabal --bindir=/home/vanessa/.cabal/bin
--libdir=/home/vanessa/.cabal/lib/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0-inplace
--libsubdir= --dynlibdir=/home/vanessa/.cabal/lib/x86_64-linux-ghc-8.0.2
--libexecdir=/home/vanessa/.cabal/libexec/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0
--libexecsubdir=
--datadir=/home/vanessa/.cabal/share/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0
--datasubdir=
--docdir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0
--htmldir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0/html
--haddockdir=/home/vanessa/.cabal/share/doc/x86_64-linux-ghc-8.0.2/shake-literate-0.1.0.0/html
--sysconfdir=/home/vanessa/.cabal/etc --enable-library-vanilla
--disable-library-profiling --enable-shared --disable-static
--disable-executable-dynamic --disable-profiling --profiling-detail=default
--library-profiling-detail=default --enable-optimization --disable-debug-info
--disable-library-for-ghci --disable-split-sections --disable-split-objs
--disable-executable-stripping --disable-library-stripping --package-db=clear
--package-db=global
--package-db=/home/vanessa/.cabal/store/ghc-8.0.2/package.db
--package-db=/home/vanessa/programming/haskell/done/shake-literate/dist-newstyle/packagedb/ghc-8.0.2
--flags=-development --cid=shake-literate-0.1.0.0-inplace
--extra-prog-path=/home/vanessa/.cabal/bin --dependency=base=base-4.9.1.0
--dependency=cpphs=cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf
--dependency=shake=shake-0.16.4-9e6708c5802ab338c5aace79838c373a8ae081f507beadc3e69a471f220df3be
--disable-coverage --exact-configuration --with-ghc=/opt/ghc/bin/ghc-8.0.2
--with-ghc-pkg=/opt/ghc/bin/ghc-pkg-8.0.2 --ghc-option=-hide-all-packages
lib:shake-literate
Using Parsec parser
Configuring library for shake-literate-0.1.0.0..
CallStack (from HasCallStack):
  die', called at ./Distribution/Simple/Configure.hs:439:20 in Cabal-2.3.0.0-3demcrOUkh3LONSMbNeyOx:Distribution.Simple.Configure
  configure, called at ./Distribution/Simple.hs:596:20 in Cabal-2.3.0.0-3demcrOUkh3LONSMbNeyOx:Distribution.Simple
  confHook, called at ./Distribution/Simple/UserHooks.hs:67:5 in Cabal-2.3.0.0-3demcrOUkh3LONSMbNeyOx:Distribution.Simple.UserHooks
  configureAction, called at ./Distribution/Simple.hs:178:19 in Cabal-2.3.0.0-3demcrOUkh3LONSMbNeyOx:Distribution.Simple
  defaultMainHelper, called at ./Distribution/Simple.hs:120:19 in Cabal-2.3.0.0-3demcrOUkh3LONSMbNeyOx:Distribution.Simple
  defaultMainArgs, called at main/Main.hs:1214:18 in main:Main
cabal: The following package dependencies were requested
--dependency='cpphs=cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf'
however the given installed package instance does not exist.

It still fails if I run rm -rf dist-newstyle but I have not tried deleting ~/.cabal and ~/.ghc.

[gbaz]

I have a guess here. I suspect that it recognizes the _library_ dependency on cpphs as a tool dependency instead (c.f. https://github.com/haskell/cabal/blob/0c72bc88c838d8139e2896a6f9dfa5d60f12756d/Cabal/Distribution/Simple/BuildToolDepends.hs#L54).

It then determines that the program exists, but then omits to check that the library exists for use with the particular compiler configuration.

I think this is a rare issue that we haven't seen before because people tend to depend on the executable and not the library, and so the "clever" hack that correctly turns the library into a built-tool depends for _most_ files, in this case actually does the wrong thing.

[hvr]

I've been trying to reproduce this locally, but so far I haven't had any luck

the "clever" hack that correctly turns the library into a build-tool

I don't understand what you mean, can you point to the code that implements this logic? afaik if you build-depends on a package (which is what shake-literate does), you'll only get access to its primary library component (if one exists), the exe components aren't requested, and may not be built at all.

Also note there's quite a few packages combining executables & libraries where users typically want the library component (where the executables are just relatively uninteresting CLI frontends or test programs). And we haven't observed issues of this kind there yet.

I also don't see how

will sometimes fail if cpphs has already been depended by a project using GHC 8.4.3.

can interact with with build-plans for other GHC versions, as each GHC version has its own nix-style store which are isolated from each other. I.e. whatever the store for GHC 8.4.3 cached cannot have any influence on a build process for say GHC 8.0.2.

[vmchale]

I am still unsure as to why the global package state gets into an inconsistent state, but building shake-literate with cabal new-builld -w ghc-8.0.2 actually works fine if I run rm -rf ~/.cabal ~/.ghc beforehand. I will attempt to provide a reproducible set of steps for the failure.

If it helps, some of my builds have failed for this reason on the matrix CI in the past (including shake-literate).

[gbaz]

@hvr yeah i think my theory isn't exactly right. rereading the code I linked I think I misread one instance of build-tools as build-depends. I hadn't thought that this occurred in general -- just that there might be some special treatment of cpphs in particular. But the place that it occurs in cabal code in a "white list" isn't translating build-depends into build-tool-depends, but rather build-tools in to build-tool-depends. However, I still suspect something particular to _cpphs_. Recall that we _don't_ have isolated nix-style stores for built-tools, iirc, just for libraries. So my suspicion is that something _upstream_ of the shake-literate package is pulling in cpphs as a build tool -- which isn't hard, anything with any cpp will do that somehow. But then the use of it as a build-tool is getting mixed up with the use of it as a library, and thus throwing off the test for it?

At least that gives one approach to try to narrow down to a smaller repro.

[hvr]

@vmchale I have a suspicion that your nix-style store got into an inconsistent state (in the sense of #5125), thus I'd suspect that you have a folder cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf somewhere in ~/.cabal/store/, but if you pointed ghc-pkg -f ... list to the respective pkg-db folder, the registration for that cpphs unit would be missing. Can you confirm or refute my suspicion?

[hvr]

I was able to reproduce this myself now; there appears to be a subtle bug in new-build's logic which can be reproduced as follows:

Start with an empty ~/.cabal/store/$compiler for the specific $compiler, and create the following dummy package description:

cabal-version: 2.2
name: z
version: 0

library
  build-tools: cpphs

run cabal new-build which will result in

$ cabal new-build -w ghc-8.0.2
Resolving dependencies...
Build profile: -w ghc-8.0.2 -O1
In order, the following will be built (use -v for more details):
 - old-locale-1.0.0.7 (lib) (requires build)
 - text-1.2.3.0 (lib) (requires build)
 - old-time-1.1.0.3 (lib:old-time) (requires build)
 - polyparse-1.12 (lib:polyparse) (requires build)
 - cpphs-1.20.8 (exe:cpphs) (requires build)
 - z-0 (lib) (first run)
Configuring old-locale-1.0.0.7 (lib)...
Configuring text-1.2.3.0 (lib)...
Building old-locale-1.0.0.7 (lib)...
Building text-1.2.3.0 (lib)...
Configuring old-time-1.1.0.3 (all, legacy fallback)...
Building old-time-1.1.0.3 (all, legacy fallback)...
Configuring polyparse-1.12 (all, legacy fallback)...
Building polyparse-1.12 (all, legacy fallback)...
Configuring cpphs-1.20.8 (all, legacy fallback)...
Building cpphs-1.20.8 (all, legacy fallback)...
Configuring library for z-0..
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Preprocessing library for z-0..
Building library for z-0..

Notice how the build-plan mentions only exe:cpphs but not lib:cpphs,

 - cpphs-1.20.8 (exe:cpphs) (requires build)

but also

Building cpphs-1.20.8 (all, legacy fallback)...

which means that cpphs is built in monolithic mode, meaning that per-component mode is disabled, and thus all lib&exe components of cpphs are built and installed under the same hash.

After this procedure, we end up with the situation that a folder

~/.cabal/store/ghc-8.0.2/cpphs-1.20.8-5073bb53f1c45a8d04e54e55ee8e2500a032fca11038320e52b495d33ca6cbbf/

gets populated with library and executable component, however it doesn't get registered via ghc-pkg. This is a bug.

If we had used build-depends: cpphs instead; we would have ended up with the same result except that also ghc-pkg register would have been called.

In summary, the bug is exposed if the following conditions are simultaneously satisfied:

1. A package with executable & library components is depended upon via build-tools (or build-tool-depends) -- i.e. its executable component is requested -- and experiences a nix-style store cache-miss
2. Said lib+exe package requires to disable per-component build-mode (because it's either cabal-version:1.6 (or less), or it's a non-Simple build-type)

...then the store cache entry is fully populated but but its library component fails to be registered! (this leaves the cache in an inconsistent state, as the invariant that each cached unit (i.e. for which a folder was created and populated) must have its library component (if it contains any) registered via ghc-pkg is violated -- and the cache-logic relies on that invariant.

[23Skidoo]

/cc @dcoutts

[hvr]

@vmchale this should be fixed now and if it doesn't get reverted will be included in the next 2.4 point release; this should also be available for testing via my PPA in the next couple hours once the PPA builds of cabal-install have finished