Cabal: ghc envs not properly ignored by `new-build`

[hvr]

I just ran into an issue which with the help of @ezyang was diagnosed to be caused
by an unlucky combination of custom Setup.hs and a GHC 8.0.2 snapshot with
a not yet updated Cabal lib

The symptom is the following:

given a trivial package such as

name:                dummy
version:             0
build-type:          Simple
cabal-version:       >=1.10

library
  build-depends: tasty

Running cabal new-build for the first time succeeds, and creates a file .ghc.environment.x86_64-linux-8.0.1.20161014 with the content

-- This is a GHC environment file written by cabal. This means you can
-- run ghc or ghci and get the environment of the project as a whole.
-- But you still need to use cabal repl $target to get the environment
-- of specific components (libs, exes, tests etc) because each one can
-- have its own source dirs, cpp flags etc.
-- 
clear-package-db
global-package-db
package-db /home/hvr/.cabal/store/ghc-8.0.1.20161014/package.db
package-db dist-newstyle/packagedb/ghc-8.0.1.20161014
package-id ansi-terminal-0.6.2.3-e8175c6cd482d5c9e7b92ba68d6363b841b6627942fc6368b7980e5944277f9d
package-id base-4.9.0.0
package-id ghc-prim-0.5.0.0
package-id rts
package-id integer-gmp-1.0.0.1
package-id unix-2.7.2.0
package-id bytestring-0.10.8.1
package-id deepseq-1.4.2.0
package-id array-0.5.1.1
package-id time-1.6.0.1
package-id async-2.1.0-65beb88cd84537d902b2bb0f834cec19c2fc7a501af9de365360f8d1d3224d6a
package-id stm-2.4.4.1-4fb640c22a961fa87e0f7ca5aaa4a3d040a98d232ed1827dd3d2c534ddf5a059
package-id clock-0.7.2-e14c7ff6e51334938f8f342093fa84a10e8a685303527380a41ac7c5b9b8b278
package-id containers-0.5.7.1
package-id mtl-2.2.1-09ef0d4c975ddca7091852fddcdf250bc2243b8c25c9e267e3c43a1a2aeb8f14
package-id transformers-0.5.2.0
package-id optparse-applicative-0.13.0.0-fa1c47887f70127fe79d25c05562b407fe6f73cba4ff63d861ded51d43251a61
package-id ansi-wl-pprint-0.6.7.3-36d4937e3d5870272653a83aaa85f2b99285ab119ddcbde6b50d4f726cbab73a
package-id process-1.4.2.0
package-id directory-1.2.6.2
package-id filepath-1.4.1.0
package-id transformers-compat-0.5.1.4-795f7daed226c475a0f42d64d6a997ab25d35660d2882080b8c3d27ea749cfeb
package-id text-1.2.2.1-7d47767d7e9da8b3b49e4a3e24e417492dede5a9823bcec3284cc8b02d22c04e
package-id binary-0.8.3.0
package-id regex-base-0.93.2-e578239124026e62ea947d16d81304c8ef35f86d34d23706baae8b5d9282929f
package-id tagged-0.8.5-7c78a1c83a8b86913c62b59d223ee6d945f5b40f6dd9a65bedaa0c736b1cfe02
package-id template-haskell-2.11.0.0
package-id ghc-boot-th-8.0.1.20161014
package-id pretty-1.1.3.3

However, after resetting the store via e.g. rm -rf ~/.cabal/store/ghc-8.0.1.20161014/ and removing dist-newstyle (but leaving the .ghc.env* file as-is), and re-running cabal new-build we now run into a weird & confusing build failure:

$ cabal new-build
In order, the following will be built (use -v for more details):
 - ansi-terminal-0.6.2.3 (lib) (requires build)
 - clock-0.7.2 (lib) (requires build)
 - mtl-2.2.1 (lib) (requires build)
 - old-locale-1.0.0.7 (lib) (requires build)
 - stm-2.4.4.1 (lib) (requires build)
 - text-1.2.2.1 (lib) (requires build)
 - transformers-compat-0.5.1.4 (lib) (requires build)
 - ansi-wl-pprint-0.6.7.3 (lib) (requires build)
 - regex-base-0.93.2 (lib) (requires build)
 - old-time-1.1.0.3 (lib) (requires build)
 - async-2.1.0 (lib) (requires build)
 - parsec-3.1.11 (lib) (requires build)
 - tagged-0.8.5 (lib) (requires build)
 - optparse-applicative-0.13.0.0 (lib) (requires build)
 - unbounded-delays-0.1.0.9 (lib:unbounded-delays) (requires build)
 - regex-tdfa-1.2.2 (lib) (requires build)
 - tasty-0.11.0.4 (lib) (requires build)
 - dummy-0 (lib) (first run)
Configuring ansi-terminal-0.6.2.3 (lib)...
Configuring clock-0.7.2 (lib)...
Configuring mtl-2.2.1 (lib)...
Configuring old-locale-1.0.0.7 (lib)...
Configuring transformers-compat-0.5.1.4 (lib)...
Configuring stm-2.4.4.1 (lib)...
Configuring text-1.2.2.1 (lib)...
Building clock-0.7.2 (lib)...
Building mtl-2.2.1 (lib)...
Building ansi-terminal-0.6.2.3 (lib)...
Building old-locale-1.0.0.7 (lib)...
Building stm-2.4.4.1 (lib)...
Building text-1.2.2.1 (lib)...
Building transformers-compat-0.5.1.4 (lib)...
Configuring tagged-0.8.5 (lib)...
Configuring old-time-1.1.0.3 (lib)...
Building tagged-0.8.5 (lib)...
Configuring async-2.1.0 (lib)...
Building async-2.1.0 (lib)...
Building old-time-1.1.0.3 (lib)...
Configuring regex-base-0.93.2 (lib)...
Building regex-base-0.93.2 (lib)...
Configuring ansi-wl-pprint-0.6.7.3 (lib)...
Building ansi-wl-pprint-0.6.7.3 (lib)...
Configuring optparse-applicative-0.13.0.0 (lib)...
Building optparse-applicative-0.13.0.0 (lib)...
Configuring unbounded-delays-0.1.0.9 (all, due to Custom setup)...

Failed to build unbounded-delays-0.1.0.9. The failure occurred during the
configure step.
Build log (
/home/hvr/.cabal/logs/ghc-8.0.1.20161014/unbounded-delays-0.1.0.9-3018361f6c155727e30a06ab7e7c36258ec4708e839db0dd5e568528427ca8b8.log
):
[1 of 1] Compiling Main             ( /home/hvr/Haskell/dummy/dist-newstyle/tmp/build-24031/setup/setup.hs, /home/hvr/Haskell/dummy/dist-newstyle/tmp/build-24031/setup/Main.o )
Linking /home/hvr/Haskell/dummy/dist-newstyle/tmp/build-24031/setup/setup ...
Configuring unbounded-delays-0.1.0.9...
<command line>: cannot satisfy -package-id optparse-applicative-0.13.0.0-fa1c47887f70127fe79d25c05562b407fe6f73cba4ff63d861ded51d43251a61
    (use -v for more information)
cabal-head: Failed to build unbounded-delays-0.1.0.9 (which is required by
dummy). See the build log above for details.

[ezyang]

CC @dcoutts who is captain of this feature.

[dcoutts]

Sigh. I guess we can probably work around it for older Cabal Setup.hs scripts by passing --ghc-options=-hide-all-packages. This will also be helped by the things being back-ported to the 1.24.x branch.

[ezyang]

So, this is because GHC walks up directories until it finds the environment file? Harsh. (You would quite prefer it if you could run Setup script with the environment file unconditionally hidden.) Maybe we could add some sort of environment variable escape hatch to GHC?

[bgamari]

I wonder whether @ezyang is right; using -hide-all-packages to disable environment file support seems diabolically roundabout. If we are going to do this it should be very soon; the window for 8.0.2 is very rapidly closing.

[dcoutts]

@bgamari it could be more explicit with its own new flag, and perhaps that's nice, but it does not seem essential, and certainly not something to hold up 8.0.2 for. The -hide-all-packages while not very explicit does have the merit that it is what is done in 99% of cases already when setting up a specific environment.

[alexbiehl]

I really like this feature and would want it to get into GHC as soon as possible. I have created a patch for GHC: https://github.com/alexbiehl/ghc/commit/42bbc38940699fe0d271dcdbc9bf492ffe37f2c4

If this approach is okay I will submit this patch to GHCHQ and modify cabal accordingly.

[ezyang]

Hi @alexbiehl, the GHC-level change doesn't really solve the core problem with this bug, because the issue here is old Cabal Setup scripts aren't passing enough flags to environment file enabled GHCs to ensure that the environment file is ignored on certain operations.

So really, your fix "accidentally" solves the problem, but only because we never write out environment variables for GHC 8.0 which ships with Cabal 1.24 which doesn't pass -hide-all-packages in enough places. Maybe that's good enough, but I am still concerned that there is no consistent strategy in Cabal's codebase for making sure we don't forget to pass -hide-all-packages to GHC to ensure that the package environment is not applied.

[hvr]

There appear to be at least two ways to workaround this:

• either setenv("GHC_ENVIRONMENT","/dev/null") (or rather set env-var via execve(2)) before executing a Setup.hs linked against a Cabal < 1.24.2 (NB: this results in an implicit -hide-all-packages), or
• inject --ghc-options=-hide-all-packages into configPrograms (in ProjectPlanning.setupHsConfigureFlags), e.g.

    configProgramArgs
        | elabSetupScriptCliVersion < mkVersion [1,24,2]
          -- workaround for https://github.com/haskell/cabal/issues/4010
                              = Map.toList $ traceShowId $
                                Map.insertWith (++) "ghc" ["-hide-all-packages"]
                                               elabProgramArgs
        | otherwise           = Map.toList elabProgramArgs

---

Note however, that injecting --ghc-options=-hide-all-packages appears to result in duplication of the -hide-all-packages flag:

("/opt/ghc/bin/ghc",["--make","-v","-fbuilding-cabal-package","-O","-static","-dynamic-too","-dynosuf","dyn_o","-dynhisuf","dyn_hi","-outputdir","dist/build","-odir","dist/build","-hidir","dist/build","-stubdir","dist/build","-i","-idist/build","-i.","-idist/build/autogen","-Idist/build/autogen","-Idist/build","-optP-include","-optPdist/build/autogen/cabal_macros.h","-this-unit-id","unbounded-delays-0.1.0.10-a6e8369d7f8cef5fd41d8a3108ced8cc60762d154f538655988622577ad44308"

,"-hide-all-packages","-no-user-package-db","-package-db","/home/test/.cabal/store/ghc-8.0.2/package.db","-package-db","dist/package.conf.inplace","-package-id","base-4.9.1.0","-XHaskell98","Control.Concurrent.Thread.Delay","Control.Concurrent.Timeout","-Wall"

,"-hide-all-packages"])

---

PS: For a proof of concept, the shell invocation GHC_ENVIRONMENT=/dev/null cabal new-build can be used to test the workaround