Cabal: Cannot install pango with new-build when using local copy of gtk2hs-buildtools

[RyanGlScott]

I'm experiencing this issue with both:

$ cabal --version
cabal-install version 2.0.0.1
compiled using version 2.0.1.1 of the Cabal library 
$ cabal-head --version
cabal-install version 2.1.0.0
compiled using version 2.1.0.0 of the Cabal library

To trigger the issue:

$ cabal get pango-0.13.4.0
Unpacking to pango-0.13.4.0/
$ cd pango-0.13.4.0/
$ cabal get gtk2hs-buildtools-0.13.3.1
Unpacking to gtk2hs-buildtools-0.13.3.1/
$ cat > cabal.project << EOF
> packages: .
>           ./gtk2hs-buildtools-0.13.3.1
> EOF
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - gtk2hs-buildtools-0.13.3.1 (lib) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsTypeGen) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsHookGenerator) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsC2hs) (first run)
 - glib-0.13.5.0 (lib:glib) (requires build)
 - cairo-0.13.4.2 (lib:cairo) (requires build)
 - pango-0.13.4.0 (lib:pango) (first run)
Configuring library for gtk2hs-buildtools-0.13.3.1..
Preprocessing library for gtk2hs-buildtools-0.13.3.1..
<build output elided>
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/Main.o )

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:51:18: warning: [-Wdeprecations]
    In the use of ‘autogenModulesDir’
    (imported from Distribution.Simple.BuildPaths):
    Deprecated: "If you can, use 'autogenComponentModulesDir' instead, but if you really wanted package-global generated modules, use 'autogenPackageModulesDir'.  In Cabal 2.0, we avoid using autogenerated files which apply to all components, because the information you often want in these files, e.g., dependency information, is best specified per component, so that reconfiguring a different component (e.g., enabling tests) doesn't force the entire to be rebuilt.  'autogenPackageModulesDir' still provides a place to put files you want to apply to the entire package, but most users of 'autogenModulesDir' should seriously consider 'autogenComponentModulesDir' if you really wanted the module to apply to one component."
   |
51 |     targetDir  = autogenModulesDir lbi
   |                  ^^^^^^^^^^^^^^^^^

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:55:3: warning: [-Wdeprecations]
    In the use of ‘die’ (imported from Distribution.Simple.Utils):
    Deprecated: "Messages thrown with die can't be controlled with Verbosity; use die' instead, or dieNoVerbosity if Verbosity truly is not available"
   |
55 |   die $ "unexpected pango version number: " ++ display version
   |   ^^^
Linking /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup ...
Configuring pango-0.13.4.0...
/u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template: openFile: does not exist (No such file or directory)cabal: Failed to build pango-0.13.4.0. The failure occurred during the
configure step.

Note that this bug will not be triggered if you have installed gtk2hs-buildtools-0.13.3.1 previously using cabal install, as that will cause ~/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template to have been created.

Moreover, this bug does _not_ occur if gtk2hs-buildtools-0.13.3.1 is not put into cabal.project and is instead downloaded from Hackage.

I suspect the issue may have something to do with this line of code in gtk2hs-buildtools-0.13.3.1:

templateFile <- getDataFileName "hierarchyGen/Hierarchy.chs.template"

[RyanGlScott]

I've created https://github.com/RyanGlScott/cabal-gh5164, which minimizes this bug. To use it:

$ git clone https://github.com/RyanGlScott/cabal-gh5164
$ cd cabal-gh5164/
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - setup-lib-0.1 (lib) (first run)
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Configuring library for setup-lib-0.1..
Preprocessing library for setup-lib-0.1..
Building library for setup-lib-0.1..
[1 of 2] Compiling Paths_setup_lib  ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/autogen/Paths_setup_lib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/Paths_setup_lib.o )
[2 of 2] Compiling SetupLib         ( src/SetupLib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/SetupLib.o )
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/Main.o )
Linking /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup ...
setup: /u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1/example.txt: openFile: does not exist (No such file or directory)

[RyanGlScott]

In contrast, here's what happens with cabal install:

$ cabal install setup-lib/
Resolving dependencies...
Configuring setup-lib-0.1...
Building setup-lib-0.1...
Installed setup-lib-0.1
$ cd fails-with-new-build/
$ cabal build
Resolving dependencies...
[1 of 1] Compiling Main             ( dist/setup/setup.hs, dist/setup/Main.o )
Linking ./dist/setup/setup ...
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, dist/build/FailsWithNewBuild.o )

[hvr]

I believe this to be a long-standing issue w/ local packages, because we construct a Paths_ that points to paths that make no sense for nix-style local builds... but I can't seem to find the issue which contains the discussion (and fwiw, a related issue exists for local build-tools #5104).

So the solution would be either to

• have Paths_... contain the actual local filepaths, or
• for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

[RyanGlScott]

Out of curiosity, is there an environment variable that I can set manually to work around this issue?

[hvr]

Sure, it's documented here:

http://cabal.readthedocs.io/en/latest/developing-packages.html?highlight=datadir#accessing-data-files-from-package-code

So you need to set an env-var named normalised_pkgname_datadir (you can cheat by looking at the autogenerated Paths_....hs source code)

And for reference, this here is the place were e.g. new-runsets those env-vars automatically:
https://github.com/haskell/cabal/blob/abaf365d9ee1405652f124827758915f08abc12f/cabal-install/Distribution/Client/CmdRun.hs#L226-L256

[RyanGlScott]

Thanks, that's very helpful.

A related question: does new-build put the data-files somewhere in dist-newstyle? I just searched for example.txt, but couldn't find it anywhere in my dist-newstyle.

[RyanGlScott]

For the time being, I'm finding myself having to hardcode the directory in which the data-files live to make this work. For instance:

$ setup_lib_datadir="$(pwd)/setup-lib" bash -c 'cabal new-build all'
In order, the following will be built (use -v for more details):
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/build/FailsWithNewBuild.o )
Example data file

Example data file

Obviously, this is quite ugly, and won't scale if your data-files are spread out over multiple directories. But since new-build apparently doesn't put data-files into dist-newstyle, I don't see any other way to make this work at the moment.

[hvr]

A related question: does new-build put the data-files somewhere in dist-newstyle?

Maybe for (non-local) "inplace" packages (I'd have to check), but certainly not for local "inplace" ones.

[RyanGlScott]

Note that stack does the correct thing here:

$ stack build fails-with-new-build                   
setup-lib-0.1: configure (lib)
setup-lib-0.1: build (lib)
setup-lib-0.1: copy/register
fails-with-new-build-0.1: configure (lib)
[1 of 2] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/Setup.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/Main.o )
[2 of 2] Compiling StackSetupShim   ( /home/rgscott/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/StackSetupShim.o )
Linking /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/setup ...
Example data file

Configuring fails-with-new-build-0.1...
fails-with-new-build-0.1: build (lib)
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/FailsWithNewBuild.o )
fails-with-new-build-0.1: copy/register
Example data file

Installing library in /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/fails-with-new-build-0.1-BoDEqEiYoIHEwENsULGL10
Example data file

Registering library for fails-with-new-build-0.1..
Completed 2 action(s).

$ more ./setup-lib/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/autogen/Paths_setup_lib.hs
{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -fno-warn-missing-import-lists #-}
{-# OPTIONS_GHC -fno-warn-implicit-prelude #-}
module Paths_setup_lib (
    version,
    getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir,
    getDataFileName, getSysconfDir
  ) where

import qualified Control.Exception as Exception
import Data.Version (Version(..))
import System.Environment (getEnv)
import Prelude

#if defined(VERSION_base)

#if MIN_VERSION_base(4,0,0)
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#else
catchIO :: IO a -> (Exception.Exception -> IO a) -> IO a
#endif

#else
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#endif
catchIO = Exception.catch

version :: Version
version = Version [0,1] []
bindir, libdir, dynlibdir, datadir, libexecdir, sysconfdir :: FilePath

bindir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/bin"
libdir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/setup-lib-0.1-8hekJdedMBc9fTEHUxKQOX"
dynlibdir  = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2"
datadir    = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
libexecdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/libexec/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
sysconfdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/etc"

getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir, getSysconfDir :: IO FilePath
getBinDir = catchIO (getEnv "setup_lib_bindir") (\_ -> return bindir)
getLibDir = catchIO (getEnv "setup_lib_libdir") (\_ -> return libdir)
getDynLibDir = catchIO (getEnv "setup_lib_dynlibdir") (\_ -> return dynlibdir)
getDataDir = catchIO (getEnv "setup_lib_datadir") (\_ -> return datadir)
getLibexecDir = catchIO (getEnv "setup_lib_libexecdir") (\_ -> return libexecdir)
getSysconfDir = catchIO (getEnv "setup_lib_sysconfdir") (\_ -> return sysconfdir)

getDataFileName :: FilePath -> IO FilePath
getDataFileName name = do
  dir <- getDataDir
  return (dir ++ "/" ++ name)

[hvr]

@RyanGlScott this looks a lot like what the old cabal sandbox did as well...

[RyanGlScott]

Indeed, I can confirm that this works with sandboxes as well.

This is an annoying enough issue (I experience it several times per day) that I'm inclined to figure out how to fix it. stack fixes this issue by wiring in a different datadir. Is there an equivalent code path for new-build that would accomplish the same thing?

[hvr]

Well, first we'd need to agree which of the two variants mentioned in my comment https://github.com/haskell/cabal/issues/5164#issuecomment-368245245 we want to pursue (there's a reason we didn't just go w/ the obsolete cabal sandbox approach, but I don't remember the details); I remember some discussion about this in another ticket, and it would be worth locating that again to learn about the conclusion from there.

PS: related #4120 and https://github.com/haskell/cabal/issues/4558#issuecomment-315457767 (but I don't think that's the discussion I was thinking of)

/cc @phadej @23Skidoo

[23Skidoo]

• for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

I'm +1 on this idea, what are the arguments against?

[RyanGlScott]

I'd be in favor of that as well.

[hvr]

@23Skidoo Can't think of any showblocker arguments against either.

[RyanGlScott]

Great! Now the question arises of how to implement this.

I'm a neophyte when it comes to the Cabal codebase—are there any places where we currently set these environment variables? And where in cabal new-build would I need to set them?

[23Skidoo]

@RyanGlScott

are there any places where we currently set these environment variables?

new-run does it in https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/CmdRun.hs#L217

And where in cabal new-build would I need to set them?

https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectBuilding.hs#L930 is the place where new-build runs the compiled setup script.

[RyanGlScott]

A question for the Cabal folks. I looked into implementing this recently and was presented with a design concern that scared me. Presumably, we're going to want to set these environment variables with something like withEnv. But as the comment for that notes:

-- Warning: This operation is NOT thread-safe, because current
-- environment is a process-global concept.

This is pretty worrying—if we're going to be building things in parallel (which is often the case), it means that we don't have a particularly strong guarantee that when we build a particular component, <component>_datadir _et al._ will be set to the right thing.

Note that new-run doesn't suffer from this issue since it runs an executable in a separate process, so it only needs to set environment variables once. In a new-build sort of situation, however, we might need different <component>_datadir values for each thread.

Thoughts?

[23Skidoo]

Setting the vars should be fine when using the external setup method (running a compiled setup script) or the self-exec setup method (cabal act-as-setup), because in both cases you're running a separate process. IIUC, this bug manifests only when using build-type: Custom, which means the former.

Though you're right about withEnv, you shouldn't use it. Instead, pass the env arg to runProcess in D.C.SetupWrapper when running the setup script. Adding a useExtraEnvVars setting to SetupScriptOptions is IMO the logical place for this, there's already a useExtraPathEnv setting.