Nix: Manual's build example doesn't build

[pushcx]

Chapter 13. A Simple Nix Expression (from doc/manual/expressions/simple-expression.xml) introduces building packages with a worked example of building GNU hello. I attempted to use this to learn to build nix packages and was unable to build the package. I don't know where I went wrong, so this bug report is something of a step-by-step. (For context, I'm a longtime professional linux sysadmin and developer who's totally new to nix. I was able to set up a NixOS box from its manual to play around with.)

The chapter is structured around three steps: writing a nix expression (13.1), writing a build script (13.2), and adding the package (13.3).

13.1 doesn't explain where to create default.nix, but from the note "It is customary to place each package in a separate directory" I figured I should create a test directory with this default.nix in it. In 13.2 I also put builder.sh in this directory.

Then instead of the section heading for 13.3 ("Arguments and Variables") matching the third bullet point in the intro, it's kind of a random name. It talks about editing all-packages.nix but the syntax is invalid. As a newbie, I don't have the experience to fill in ... so the example is unusable. Additionally it talks about editing pkgs/top-level/all-packages.nix like that directory already exists, and the only file by that name that find turned up is /nix/store/ilagh0i1vxhp68hd9r73lh0f2mjqbj5x-nixos-18.09pre143771.a8c71037e04/nixos/pkgs/top-level/all-packages.nix. The description of the nix store in 5.3 has a wishy-washy warning against editing files in /nix/store and I don't see any way to regenerate that hash (or maybe I don't need to? The introduction to the hash at the top of chapter 1 says it's only based on dependencies, not contents). Still, it isn't clear at all what I'm supposed to be doing to what file, or why this is required for building hello.

Then section 13.4 starts "You can now try to build Hello." and nix-build -A hello looks like it's assuming I put those files in a directory named hello, though after renaming my directory I got the error error: getting status of '/home/pushcx/default.nix': No such file or directory from outside the directory and from inside I got error: cannot auto-call a function that has an argument without a default value ('stdenv') (with no file name or line-number). The error message is useless because I'm trying to build a package, not auto-call a function, and "auto-call" does not appear in the manual. I don't see anything more to experiment with to try to fix this.

After chatting with infinisil in #nixos, I think the two causes of my confusion are:

1. The manual does not provide a complete, worked example of building a package.
2. The manual conflates building a package and contributing publicly to nixpkgs.

He suggested (via the bot, so is this a frequent problem?), "If a Nix file ./foo.nix starts with something like { bar, baz }:, you can build it with nix-build -E '(import <nixpkgs> {}).callPackage ./foo.nix {}'" This doesn't match the description of callPackage in the manual, which only explains callPackage as something you'd write in all-packages.nix rather than a command-line tool. And it doesn't account for writing a builder.sh.

Additionally, it would probably be worthwhile for someone on the team to watch a newbie try to work through the doc without answering any questions, because this wasn't the first issue I had to puzzle out, just the issue I couldn't puzzle out.

[vaibhavsagar]

It is frustrating that the example in the manual doesn't build, especially when only small changes are required to default.nix:

let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "hello-2.1.1";
  builder = ./builder.sh;
  src = pkgs.fetchurl {
    url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  perl = pkgs.perl;
}

[pushcx]

Thanks very much for the improvement, @vaibhavsagar, I can see how this looks more like a complete example. What command are you using to build? Do you have it in a hello directory? Are you running nix-build from within that directory, or a parent? Are you able to run nix-shell or nix-env?

(Asking because I've tried a bunch of variations like nix-build -A hello and nix-build -A . with no luck.)

[pushcx]

I sort of got it to start a build, but got a strange failure out of configure. I tried to run nix-shell because that seemed like the right tool but it also lacks newbie docs. There's a section over in the "nixpkgs" manual (though I'm not trying to contribute to a public package so I don't understand why it's there) but its description of "phases" don't at all match what's present in the shell and lead to a whole 'nother set of bugs and underdocumented topics. One again, I don't see any way forward with building a package.

I appreciate the folks who've tried to help, but I've reached my limit of time and frustration on this experiment. I'm unsubscribing from this issue; I don't know what your project standards are around closing or keeping issues open, so I'll leave it to a contributor to handle.

[vaibhavsagar]

Oops, that's meant to go in a default.nix file that you then run by executing nix-build default.nix or just nix-build (which looks for a default.nix if no filename is provided).

[cmal]

@vaibhavsagar Hi, Thanks for your fix. I am stepping over the errors mentioned by the original author. I am using nix-build default.nix to build GNU Hello, but still errors:

...
...
make[2]: Entering directory '/private/tmp/nix-build-hello-2.1.1.drv-0/hello-2.1.1/src'
source='hello.c' object='hello.o' libtool=no \
depfile='.deps/hello.Po' tmpdepfile='.deps/hello.TPo' \
depmode=gcc3 /nix/store/n9hba031gjky8hpjgx9fnlaxhidyzxbz-bash-4.4-p23/bin/bash ../depcomp \
clang -DLOCALEDIR=\"/nix/store/vh1c9dv71ggh4viyvlvjr4gbvvnv6az8-hello-2.1.1/share/locale\" -DHAVE_CONFIG_H -I. -I. -I.. -I. -I. -I.. -I../intl -I../intl    -g -O2 -c `test -f 'hello.c' || echo './'`hello.c
hello.c:332:17: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
        printf (_("hello, world\n"));
                ^~~~~~~~~~~~~~~~~~~
hello.c:47:19: note: expanded from macro '_'
#define _(String) gettext (String)
                  ^~~~~~~~~~~~~~~~
hello.c:332:17: note: treat the string as an argument to avoid this
        printf (_("hello, world\n"));
                ^
                "%s",
hello.c:47:19: note: expanded from macro '_'
#define _(String) gettext (String)
                  ^
hello.c:340:10: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
        printf (_("\
                ^~~~
hello.c:47:19: note: expanded from macro '_'
#define _(String) gettext (String)
                  ^~~~~~~~~~~~~~~~
hello.c:340:10: note: treat the string as an argument to avoid this
        printf (_("\
                ^
                "%s",
hello.c:47:19: note: expanded from macro '_'
#define _(String) gettext (String)
                  ^
2 errors generated.
make[2]: *** [Makefile:199: hello.o] Error 1
make[2]: Leaving directory '/private/tmp/nix-build-hello-2.1.1.drv-0/hello-2.1.1/src'
make[1]: *** [Makefile:175: all-recursive] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-hello-2.1.1.drv-0/hello-2.1.1'
make: *** [Makefile:131: all] Error 2
builder for '/nix/store/6h7c2np6z1j29fci1ck721q1x8s533qw-hello-2.1.1.drv' failed with exit code 2
error: build of '/nix/store/6h7c2np6z1j29fci1ck721q1x8s533qw-hello-2.1.1.drv' failed

How can I fix this?

$ uname -a                                                                                                   
Darwin xxx.local 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64

[vaibhavsagar]

These look like macOS-specific issues with clang. I don't think I can help.

[gloaming]

Out of curiosity I tried to reproduce this by building with clang on NixOS, but it works fine, even though the same warnings are enabled. I would guess that the compiler has special logic for the use of gettext that doesn't work on Darwin for some reason, maybe.

@cmal Could you try with this line inside the mkDerivation?

hardeningDisable = stdenv.lib.optional stdenv.isDarwin "format";

Relevant links I found:
https://github.com/NixOS/nixpkgs/pull/12895
https://fedoraproject.org/wiki/Format-Security-FAQ

[mawis]

I also have problems just at this very basic point of trying to learn how to write packages. I was following the manual with the same problems as the original author of this issue. Indeed it is very frustrating if even the hello world example in the documentation cannot be build.

With the changes in default.nix mentioned above I came a bit futher. (Thanks @vaibhavsagar ). Still it doesn't build successfully. I get the following error message:

[…]
source='getopt1.c' object='getopt1.o' libtool=no \
depfile='.deps/getopt1.Po' tmpdepfile='.deps/getopt1.TPo' \
depmode=gcc3 /nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash ../depcomp \
gcc -DLOCALEDIR=\"/nix/store/qwqg5ah3n4wg29h4m1h65b4sxbydk3v0-hello-2.1.1/share/locale\" -DHAVE_CONFIG_H -I. -I. -I.. -I. -I. -I.. -I../intl -I../intl    -g -O2 -c `test -f 'getopt1.c' || echo './'`getopt1.c
gcc  -g -O2   -o hello  hello.o version.o getopt.o getopt1.o
make[2]: Leaving directory '/build/hello-2.1.1/src'
Making all in man
make[2]: Entering directory '/build/hello-2.1.1/man'
help2man --name="Friendly Greeting Program" ../src/hello >hello.1
/nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash: help2man: command not found
make[2]: *** [Makefile:282: hello.1] Error 127
make[2]: Leaving directory '/build/hello-2.1.1/man'
make[1]: *** [Makefile:175: all-recursive] Error 1
make[1]: Leaving directory '/build/hello-2.1.1'
make: *** [Makefile:131: all] Error 2
builder for '/nix/store/mvg07i5xl1g71bpj7ic9q2r1wrh2565v-hello-2.1.1.drv' failed with exit code 2
error: build of '/nix/store/mvg07i5xl1g71bpj7ic9q2r1wrh2565v-hello-2.1.1.drv' failed

Maybe I should also mention, that I changed the hash value in default.nix to c510e3ad0200517e3a14534e494b37dc0770efd733fc35ce2f445dd49c96a7d5 is this is the sha256sum I get for ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz

I'm trying this on NixOS 19.03.

[mawis]

Okay, finally I got it building. I needed to make the help2man package accessible to the build process. So what worked for me is this:

file default.nix:

let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "hello-2.1.1";
  builder = ./builder.sh;
  src = pkgs.fetchurl {
    url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "c510e3ad0200517e3a14534e494b37dc0770efd733fc35ce2f445dd49c96a7d5";
  };
  perl = pkgs.perl;
  help2man = pkgs.help2man;
}

file builder.sh:

source $stdenv/setup                                                             

PATH=$perl/bin/:$help2man/bin/:$PATH                                             

tar xvfz $src                                                                    
cd hello-*                                                                       
./configure --prefix=$out                                                        
make                                                                             
make install

And building this with nix-build in the folder these two files are in.

[burbon]

Following documentation further

Discerning readers will note that the buildInputs could just as well have been set in the Nix expression, like this:
buildInputs = [ perl ];
which seems to solve help2man dependency so

  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "hello-2.1.1";
  buildInputs = [ pkgs.perl ];
  src = pkgs.fetchurl {
    url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
}

this is just building fine for me using nix-build

[bburdette]

Ok I found a relevant stackoverflow for this. A quote:

And finally, there are two common formats for derivations in default.nix: derivation, and callPackage derivation. You can't nix-build the latter. Almost any package in nixpkgs is written in this style, see hello. But you can
nix-build -E 'with import { }; callPackage ./path/to/default.nix { }'
as a workaround. nix-shell also supports this -E argument.

Sure enough, with the original default.nix and builder.sh in a directory, I can make it work:

bburdette@BB-5520:~/op-code/nix-example$ ls
builder.sh  default.nix
bburdette@BB-5520:~/op-code/nix-example$ nix-build -E 'with import <nixpkgs> { }; callPackage ./default.nix { }'
/nix/store/fkr5ab4lc1k0cn98lp13615pvc8r1nyh-hello-2.1.1
bburdette@BB-5520:~/op-code/nix-example$ ls
builder.sh  default.nix  result
bburdette@BB-5520:~/op-code/nix-example$ ./result/bin/hello 
Hello, world!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-on-prgmr-and-failing-to-learn-nix-push-cx/9672/3