Nixpkgs: Haskell/Stack: terminfo library cannot be built because of missing ncurses

[fmthoma]

Issue description

When building a haskell project depending on the terminfo library (possibly through other libraries like vty) using stack --nix, the build fails since the ncurses library cannot be found during the configure phase of terminfo. Adding

nix:
  packages: [ncurses]

to stack.yaml remedies the problem, however this is a very ad-hoc solution. The problem also already occurred for ghc (#2969, #5616) and others (https://github.com/ekmett/wl-pprint-terminfo/issues/7).

Steps to reproduce

• Start with a clean stack environment (mv ~/.stack ~/.stack.bkup)
• Set up a new stack/cabal project (stack new foo)

• Make the project depend on terminfo by adding in your foo.cabal:

  [...]
  executable foo-exe
  hs-source-dirs:      app
  main-is:             Main.hs
  ghc-options:         -threaded -rtsopts -with-rtsopts=-N
  build-depends:       base
                     , terminfo -- <- add this line
                     , foo
  default-language:    Haskell2010
  [...]
  

• stack --nix build

Result:

terminfo-0.4.0.2: download
terminfo-0.4.0.2: configure
Progress: 1/2
--  While building package terminfo-0.4.0.2 using:
      /run/user/1000/stack13135/terminfo-0.4.0.2/.stack-work/dist/x86_64-linux-nix/Cabal-1.22.5.0/setup/setup --builddir=.stack-work/dist/x86_64-linux-nix/Cabal-1.22.5.0 configure --with-ghc=/nix/store/n962p1idgzh07wvhxam5r6icv894kj61-ghc-7.10.3/bin/ghc --with-ghc-pkg=/nix/store/n962p1idgzh07wvhxam5r6icv894kj61-ghc-7.10.3/bin/ghc-pkg --user --package-db=clear --package-db=global --package-db=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/pkgdb --libdir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/lib --bindir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/bin --datadir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/share --libexecdir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/libexec --sysconfdir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/etc --docdir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/doc/terminfo-0.4.0.2 --htmldir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/doc/terminfo-0.4.0.2 --haddockdir=/home/fthoma/.stack/snapshots/x86_64-linux-nix/lts-5.0/7.10.3/doc/terminfo-0.4.0.2 --dependency=base=base-4.8.2.0-0d6d1084fbc041e1cded9228e80e264d --extra-include-dirs=/nix/store/n962p1idgzh07wvhxam5r6icv894kj61-ghc-7.10.3/include --extra-lib-dirs=/nix/store/n962p1idgzh07wvhxam5r6icv894kj61-ghc-7.10.3/lib
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/fthoma/projects/foo/.stack-work/logs/terminfo-0.4.0.2.log

    [1 of 1] Compiling Main             ( /run/user/1000/stack13135/terminfo-0.4.0.2/Setup.lhs, /run/user/1000/stack13135/terminfo-0.4.0.2/.stack-work/dist/x86_64-linux-nix/Cabal-1.22.5.0/setup/Main.o )
    Linking /run/user/1000/stack13135/terminfo-0.4.0.2/.stack-work/dist/x86_64-linux-nix/Cabal-1.22.5.0/setup/setup ...
    Configuring terminfo-0.4.0.2...
    configure: WARNING: unrecognized options: --with-compiler, --with-gcc
    checking for gcc... gcc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables... 
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ISO C89... none needed
    checking how to run the C preprocessor... gcc -E
    checking for grep that handles long lines and -e... /nix/store/dp6c8mcsashywfkppdzic3l1qz4n9paq-gnugrep-2.22/bin/grep
    checking for egrep... /nix/store/dp6c8mcsashywfkppdzic3l1qz4n9paq-gnugrep-2.22/bin/grep -E
    checking for ANSI C header files... yes
    checking for sys/types.h... yes
    checking for sys/stat.h... yes
    checking for stdlib.h... yes
    checking for string.h... yes
    checking for memory.h... yes
    checking for strings.h... yes
    checking for inttypes.h... yes
    checking for stdint.h... yes
    checking for unistd.h... yes
    checking ncurses.h usability... no
    checking ncurses.h presence... no
    checking for ncurses.h... no
    checking curses.h usability... no
    checking curses.h presence... no
    checking for curses.h... no
    configure: error: in `/run/user/1000/stack13135/terminfo-0.4.0.2':
    configure: error: curses headers could not be found, so this package cannot be built
    See `config.log' for more details

Technical details

• System: NixOS 16.03.886.0068260 (Emu)
• Nix version: nix-env (Nix) 1.11.2
• Nixpkgs version: 16.03.886.0068260
• Stackage: tried with LTS-5.0 through LTS-5.9

[jagajaga]

cc @peti

[peti]

I am not sure how Nixpkgs could solve that problem. The terminfo build does not declare its dependency on ncurses, hence the library is not included when terminfo is built. This seems to be well outside of the scope of thing Nix can influence; all the decisions here are made by stack.

[peti]

This issue should probably be reported to the Stack project.

[fmthoma]

@peti Actually, the dependency is declared (nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix, line 222662), but not propagated.

[peti]

The log snippet you've posted above suggests to me that terminfo is a package compiled by _Stack_, not by Nix. What dependencies Nix does or does not declare for that build makes no difference.

[fmthoma]

@peti it is compiled by stack --nix, which uses nix-shell under the hood. This she'll misses the ncurses dependency. Although I am not sure what mechanism exactly Stack uses to compute the actual dependencies to include in the Nix shell.

[peti]

I am not sure what mechanism exactly Stack uses to compute the actual dependencies to include in the Nix shell.

There you go. Maybe you should find out?

[puffnfresh]

Stack only uses Nix for system dependencies, not cabal dependencies. I had the same expectation that it would use it for all dependencies, but I was wrong. I believe Stack should.

See https://github.com/commercialhaskell/stack/issues/1683

[fmthoma]

@puffnfresh Thank you for the clarification, this is very helpful!

@peti The documentation on this is not exactly stellar, so I am working my way through this issue in a way that may solve the problem not only for me, but for others as well.

The terminfo build does not declare its dependency on ncurses, hence the library is not included when terminfo is built.

Can I read from this that you have a rough idea how this issue can be solved on terminfo side?

[PierreR]

While I understand that stack --nix build failure might be a stack related issue, I am currently in a situation where the simpler stack build terminfo (or stack build --no-nix terminfo) is failing with the same message ...

Is there a workaround ?

[PierreR]

@puffnfresh I am confused. If the nix stack integration only serves for stack to find the system dependencies, does it mean that it is not possible to work with stack without the nix integration (I mean stack --nix) on nixos ?

Given that even on master I can only get <= lts-5.9 when I use --nix, it looks like currently stack and nixos don't do along together very much.

I am probably missing something obvious (I hope @peti will forgive my ignorance given the state of the documentation for everything nix related nowadays ;-)

[peti]

If the nix stack integration only serves for stack to find the system dependencies, does it mean that it is not possible to work with stack without the nix integration (I mean stack --nix) on nixos ?

Stack works just fine on NixOS with or without the --nix flag. If you don't use that feature, then you'll simply have to install the necessary system dependencies manually.

Given that even on master I can only get <= lts-5.9 when I use --nix, it looks like currently stack and nixos don't do along together very much.

Huh? Nixpkgs master has all LTS versions up to and including version 6.5.

[puffnfresh]

I can't use Stack without the --nix flag due to https://github.com/commercialhaskell/stack/issues/826. I have to use the --nix flag.

When I use the --nix flag, I have to add nix.packages: [ ncurses.dev, ncurses.out, zlib.dev, zlib.out ] to the initial Stack project I work on, so it caches things correctly (see https://github.com/commercialhaskell/stack/issues/440) - if I accidentally only include the headers then Stack will compile and cache the packages but I have to manually unregister them and recompile with the libraries included correctly.

Stack is very broken on NixOS for me. It's a huge pain to use.

I've started writing a script which will compile all my dependencies using Nix, then link them to the right places in my project's .stack-work directory. This skips all of Stacks dependency compilation and reuses the huge amount of Haskell work that nixpkgs provides.

[PierreR]

Stack works just fine on NixOS with or without the --nix flag. If you don't use that feature, then you'll simply have to install the necessary system dependencies manually.

Thanks ! That's good to know because it didn't seem to work that way. Now that I know how it is supposed to work I am ready to try it harder ;-)

Huh? Nixpkgs master has all LTS versions up to and including version 6.5.

You're right. Sorry for all the side notes. But for what it's worth, I keep getting this error in emacs:

Suspicious state from syntax checker haskell-stack-ghc: Checker haskell-stack-ghc returned non-zero exit code 1, but no errors from output: error: attribute ‘lts-6_5’ missing, at (string):1:536

But yeah stack build does work correctly with lts-6.5. I still have to figure out what's going on exactly with the checker and how I can fix it (I do enable nix in ~/.stack/config.yaml)

PS:

• I am trying to keep my base nixos with the stable channel while I am using stack from nixpkgs master in user space (setting nixpkgs master to be found on NIX_PATH).
• I am actually trying to get intero up and running within emacs on nixos.

[peti]

I can't use Stack without the --nix flag due to commercialhaskell/stack#826. I have to use the --nix flag.

Actually, you need a working compiler in $PATH and the --nix flag just happens to provide one. If you use Nix to install a working compiler instead, however, then stack will work just fine without --nix.

[PierreR]

Actually, you need a working compiler in $PATH and the --nix flag just happens to provide one. If you use Nix to install a working compiler instead, however, then stack will work just fine without --nix.

Do you mean that given the original issue setting, if I try to build terminfo without the --nix flag, using a ghc compiler installed by nix, it is supposed to work ?

I have been trying a couple of times, but the only way for me to successfully build terminfo is with the --nix flag (ncurses and ghc have been installed by nix)

[peti]

if I try to build terminfo without the --nix flag, using a ghc compiler installed by nix, it is supposed to work ?

Yes, it is.

[jbaum98]

This is still an issue for me. Any progress?

[fmthoma]

For me too, unfortunately. There has been some change in the terminfo lib, which made matters even worse for me. See also: judah/terminfo#15, commercialhaskell/stack#3233, and this comment

[judah]

A few bits of background, since it sounds like there's some misunderstanding in this thread:

• The C library that provides terminfo-related functionality has different names on different operating systems (and/or versions of a given OS), including: ncurses, curses, ncursesw or tinfo.
• The Haskell terminfo package solves this ambiguity by using a configure script which checks whether any of those libraries are available, and writes whichever one it finds to a buildinfo file. More information here.
• stack --nix runs all its build commands (including the configure script) inside of a nix-shell. So stack needs to be told somehow to provide the ncurses dependency to that shell. (By the time the Haskell terminfo package starts configuring, it's too late the since nix-shell has started already.).

The standard fix, as mentioned above and in the Stack documentation, is to add an explicit nix.packages section in stack.yaml. (If that doesn't work, unfortunately tracking down the issue is probably beyond beyond my limited understanding of Nix.)

However, it sounds like you really want something like commercialhaskell/stack#2943. That issue suggests stack could keep track of a hand-curated mapping from Haskell packages to Nix dependencies. (And as it mentions, you run into this problem with pretty much any Haskell library with a C dependency, not just the terminfo package.)

[asvyazin]

Something indeed happened (recently?) and now explicit adding ncurses in nix.packages doesn't work for terminfo anymore.

Also interesting that if you manually unpack terminfo-0.4.1.0 (with stack unpack terminfo) and run nix-shell -p gcc -p ncurses in terminfo-0.4.1.0 directory and then run ./configure script in this nix-shell then the script succeeds. But stack build doesn't work even in this working nix-shell.

[puffnfresh]

@asvyazin does using ncurses.lib help? I think I had a problem with that while I used Stack, though I think I might have fixed that.

[asvyazin]

Unfortunately ncurses.lib doesn't work, I get error error: attribute 'lib' is missing. Perhaps you meant ncurses.dev and ncurses.out (which also do not work).

[judah]

@asvyazin, can you give more information about what breaks in that setup with stack build? Manually adding a trivial stack.yaml file worked for me (stack unpack doesn't provide that automatically):

$ stack unpack terminfo
$ cd terminfo-0.4.1.0
$ vim stack.yaml  # manually create it
$ cat stack.yaml
resolver: lts-8.23
packages:
  - '.'
$ nix-shell -p gcc -p ncurses
$ stack build
# succeeds
$ stack ghci
# loads successfully
> setupTermFromEnv >>= flip runTermOutput (termText "Hello\n")
Hello

[asvyazin]

In my case stack build can't find ghc without --nix flag (or nix.enable: true in stack.yaml) and with this flag it fails at configure step configure: error: curses library not found, so this package cannot be built.
But if I run nix-shell -p gcc -p ncurses -p ghc and then stack build --system-ghc (without nix.enable: true in stack.yaml of course) then it indeed works.

[jbaum98]

I've gotten stack build intero to work using the following shell.nix:

{ ghc }:
with (import <nixpkgs> {});

haskell.lib.buildStackProject {
  inherit ghc;
  name = "groupMaker-env";
  buildInputs = [ ncurses ];
  LANG = "en_us.UTF-8";
}

[ChrisPenner]

This whole thing is still a bit of an issue for me, but I got moving by running stack build intero with the following stack.yaml

resolver: lts-8.18

nix:
  enable: true
  packages:
    - libcxx
    - icu
    - gcc
    - ncurses

packages:
- '.'

extra-deps: []

flags: {}

extra-package-dbs: []

[ChrKM]

On Ubuntu 17.04 stack build intero (no nix) works for me after sudo apt install libtinfo-dev