Nixpkgs: depending on derivation (built with Meson): derivation not added to PKG_CONFIG_PATH; derivation's own PKG_CONFIG_PATH not inherited

[siriobalmelli]

Issue description

1. dependency chain:
2. xdpacket, derivation: https://github.com/siriobalmelli/xdpacket/blob/sirio/default.nix
3. depends on nonlibc, derivation: https://github.com/siriobalmelli/nonlibc/blob/sirio/default.nix

4. nonlibc depends on liburcu in nixpkgs

5. nonlibc builds correctly, seeing liburcu in PKG_CONFIG_PATH; see 'Nonlibc build is OK' below

6. xdpacket build sees nonlibc referenced in these environment variables:

   • HOST_PATH
   • buildInputs
   • nonlibc
   • NIX_TARGET_CFLAGS_COMPILE
   • NIX_CFLAGS_COMPILE
   • _PATH
   • PATH
     ... but unfortunately PKG_CONFIG_PATH is not in this list.
     details in 'xdpacket build not ok' below

7. xdpacket does not see liburcu referenced anywhere.

8. My understanding of the documentation saying "run-time dependencies are taken from the top level attribute set" is the inherit statement here ... did I get that correctly?

9. both derivations build with Meson

10. In the derivation for xdpacket, I am importing nonlibc by:

    • taking it as an input argument
    • adding to buildInputs
    • inheriting inside the derivation
      I am uncertain which one of these is preferable ... but even all 3 possible approaches do not seem to accomplish the desired effect.

11. I have tried many Google searches, there seems to be no mention of a method to explicitly declare additions to pkgconfig path for anyone inheriting/importing a derivation.

Expected Behavior

Importing nonlibc:

1. adds /nix/store/[nonlibc]/lib/x86_64-linux-gnu/pkgconfig to PKG_CONFIG_PATH
2. add all dependencies of nonlibc (e.g. liburcu) to PKG_CONFIG_PATH as well,
   without the importer having to declare them.

Steps to reproduce

mkdir ~/pandora && cd ~/pandora
git clone https://github.com/siriobalmelli/nonlibc.git
cd nonlibc && nix-build && cd ..
git clone https://github.com/siriobalmelli/xdpacket.git
cd xdpacket
nix-build -I ~/pandora

Technical details

sirio@container:~/nonlibc$ nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/m2mwfr63zh28wxy6zzmgns4vhjmyw0p1-nix-info
copying path '/nix/store/m2mwfr63zh28wxy6zzmgns4vhjmyw0p1-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-29-generic, Ubuntu, 18.04.1 LTS (Bionic Beaver)`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0.4`
 - channels(sirio): `"nixpkgs"`
 - channels(root): `""`
 - nixpkgs: `/home/sirio/.nix-defexpr/channels/nixpkgs`

sirio@container:~/nonlibc$ 

Also note the full travis output of both builds:

1. nonlibc build
2. xdpacket build

Nonlibc build is OK

nix-shell in nonlibc gives:

sirio@container:~/nonlibc$ pkg-config --modversion liburcu
Package liburcu was not found in the pkg-config search path.
Perhaps you should add the directory containing `liburcu.pc'
to the PKG_CONFIG_PATH environment variable
No package 'liburcu' found
sirio@container:~/nonlibc$ echo $PKG_CONFIG_PATH

sirio@container:~/nonlibc$ nix-shell

[nix-shell:~/nonlibc]$ pkg-config --modversion liburcu
0.10.1

[nix-shell:~/nonlibc]$ echo $PKG_CONFIG_PATH
/nix/store/1g97qw2czbakl2crdkbgmqhsffmqfb7b-liburcu-0.10.1/lib/pkgconfig:/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/pkgconfig:/nix/store/b46xsr24d671qymr5wbnyznka7y7v1wg-dpkg-1.19.0.5/lib/pkgconfig:/nix/store/p2vw003g0czpdpclbqsjjmcia11xci86-rpm-4.14.1-dev/lib/pkgconfig:/nix/store/n17a3q4npsvg7wvchf96ayg1a8dmk85y-popt-1.16/lib/pkgconfig:/nix/store/j20xyf0cbv81s5y1zc91wzwamkg7di0i-nss-3.38-dev/lib/pkgconfig:/nix/store/f28n054prjz3hrl4q773qfdgpp7jpiai-nspr-4.19-dev/lib/pkgconfig:/nix/store/97dgpgfsa3l0mh3ajld9p8wgkwbqnyxp-bzip2-1.0.6.0.1-dev/lib/pkgconfig:/nix/store/sk1bmar9nq013b2kn898lizljjlc8asz-libarchive-3.3.2-dev/lib/pkgconfig:/nix/store/v19c13n4zng6nyb844z5gxgn2wmp416x-elfutils-0.173/lib/pkgconfig

[nix-shell:~/nonlibc]$ 

xdpacket build not ok

# the `-I ~` stanza to force build of current nonlibc branch in my home dir
sirio@container:~/xdpacket$ nix-shell -I ~

# we don't see pkg-config dependency for nonlibc
[nix-shell:~/xdpacket]$ pkg-config --modversion nonlibc
Package nonlibc was not found in the pkg-config search path.
Perhaps you should add the directory containing `nonlibc.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nonlibc' found

[nix-shell:~/xdpacket]$ echo $PKG_CONFIG_PATH 
/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/pkgconfig:/nix/store/b46xsr24d671qymr5wbnyznka7y7v1wg-dpkg-1.19.0.5/lib/pkgconfig:/nix/store/p2vw003g0czpdpclbqsjjmcia11xci86-rpm-4.14.1-dev/lib/pkgconfig:/nix/store/n17a3q4npsvg7wvchf96ayg1a8dmk85y-popt-1.16/lib/pkgconfig:/nix/store/j20xyf0cbv81s5y1zc91wzwamkg7di0i-nss-3.38-dev/lib/pkgconfig:/nix/store/f28n054prjz3hrl4q773qfdgpp7jpiai-nspr-4.19-dev/lib/pkgconfig:/nix/store/97dgpgfsa3l0mh3ajld9p8wgkwbqnyxp-bzip2-1.0.6.0.1-dev/lib/pkgconfig:/nix/store/sk1bmar9nq013b2kn898lizljjlc8asz-libarchive-3.3.2-dev/lib/pkgconfig:/nix/store/v19c13n4zng6nyb844z5gxgn2wmp416x-elfutils-0.173/lib/pkgconfig

# but we DO see nonlibc has been included in this environment
[nix-shell:~/xdpacket]$ env | grep nonlibc | sed -nE 's/^([^=]+).*/\1/p'
HOST_PATH
buildInputs
nonlibc
CPATH
NIX_TARGET_CFLAGS_COMPILE
NIX_CFLAGS_COMPILE
_PATH
PATH

# liburcu however is NOT referenced at all
[nix-shell:~/xdpacket]$ env | grep rcu | sed -nE 's/^([^=]+).*/\1/p'

[nix-shell:~/xdpacket]$ pkg-config --modversion liburcu
Package liburcu was not found in the pkg-config search path.
Perhaps you should add the directory containing `liburcu.pc'
to the PKG_CONFIG_PATH environment variable
No package 'liburcu' found

[nix-shell:~/xdpacket]$ 

[jtojnar]

1. PKG_CONFIG_PATH is populated by pkg-config setup hook – nothing to do with meson.
2. Well, it is not listed in its expression, nor is it propagated by any of its dependencies.
3. Nope. Top-level here refers to pkgs.gcc, as opposed to pkgs.buildPackages.gcc. Usually, runtime dependencies go to buildInputs attribute and stdenv will take care of linking them correctly. If the dependency cannot be linked (for example libraries dlopened in run-time) you will need to do it manually. inherit will just pass the liburcu in an environment variable of the same name, see nix pills.
4. You can use dontUseMesonConfigure = true attributes dontUseMesonCheck = true so you do not have to override the setupHook. But I would just rely on it. If it does not do the right thing it is a bug.
5. Inheriting it inside the derivation will do absolutely nothing, see 5. You need to add the dependency to buildInputs where it is picked up by stdenv. The first link is orthogonal, it is just a way of passing the dependency to the expression.
6. Just add the dependencies to buildInputs and pkg-config setup hook will do the right thing.

[jtojnar]

Actually, /nix/store/[nonlibc]/lib/x86_64-linux-gnu/pkgconfig would not get added to PKG_CONFIG_PATH packages are expected to install .pc files to lib/pkgconfig or share/pkgconfig.

[siriobalmelli]

Thank you @jtojnar - turns out Meson's default pkg-config path of /nix/store/[nonlibc]/lib/x86_64-linux-gnu/pkgconfig was breaking things.

Fixed by forcing pkgconfig path with install_dir : get_option('prefix') + '/lib/pkgconfig' here

1. Thank you for clarification :)

2. I see. But nonlibc lists liburcu as a buildInput here ... where should I declare liburcu so that a dependent of nonlibc will get it as well?
   Right now I am obliged to declare it in xdpacket's derivation.
   It's not the end of the world but would be cleaner to just inherit liburcu transparently when inheriting nonlibc.

3. Ah, makes sense - thank you.

4. Need to dig into this more ... will come back and clean this up - I need to read the docs for the Meson derivation. Info much appreciated.

5. Thanks; removed superfluous inherit stanzas.

[jtojnar]

1. If liba is needed to use libb, you can add liba to propagatedBuildInputs of libb.
2. The setup hooks tent to be undocumented since their source code is short and quite readable. See the meson and ninja ones.

[siriobalmelli]

Thanks.
propagatedBuildInputs does not seem to work as expected.

1. implemented in nonlibc here

2. added to propagatedBuildInputs in xdpacket here

results

1. from a nix-shell -I ~ in xdpacket:

[nix-shell:~/xdpacket]$ pkg-config --modversion nonlibc
0.2.4

[nix-shell:~/xdpacket]$ pkg-config --modversion liburcu
Package liburcu was not found in the pkg-config search path.
Perhaps you should add the directory containing `liburcu.pc'
to the PKG_CONFIG_PATH environment variable
No package 'liburcu' found

[nix-shell:~/xdpacket]$

1. from a nix-shell in nonlibc:

[nix-shell:~/nonlibc]$ pkg-config --modversion liburcu
0.10.1

[nix-shell:~/nonlibc]$ 

[jtojnar]

You do not need to propagate nonlibc from xdpacket but otherwise the code looks fine. Cannot test it because nonlibc does not build for me:

building '/nix/store/mdsrsncbb5bqmmi15lim9lisrhix830p-nonlibc.drv'...
unpacking sources
unpacking source archive /nix/store/7fcx7brxc7ry7iw9v3369sr0p24f405z-nonlibc
source root is nonlibc
patching sources
configuring
pkgconfig: /nix/store/1g97qw2czbakl2crdkbgmqhsffmqfb7b-liburcu-0.10.1/lib/pkgconfig:/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/pkgconfig:/nix/store/b46xsr24d671qymr5wbnyznka7y7v1wg-dpkg-1.19.0.5/lib/pkgconfig:/nix/store/p2vw003g0czpdpclbqsjjmcia11xci86-rpm-4.14.1-dev/lib/pkgconfig:/nix/store/n17a3q4npsvg7wvchf96ayg1a8dmk85y-popt-1.16/lib/pkgconfig:/nix/store/j20xyf0cbv81s5y1zc91wzwamkg7di0i-nss-3.38-dev/lib/pkgconfig:/nix/store/f28n054prjz3hrl4q773qfdgpp7jpiai-nspr-4.19-dev/lib/pkgconfig:/nix/store/97dgpgfsa3l0mh3ajld9p8wgkwbqnyxp-bzip2-1.0.6.0.1-dev/lib/pkgconfig:/nix/store/sk1bmar9nq013b2kn898lizljjlc8asz-libarchive-3.3.2-dev/lib/pkgconfig:/nix/store/v19c13n4zng6nyb844z5gxgn2wmp416x-elfutils-0.173/lib/pkgconfig
flags:  --buildtype=release -Ddep_type=shared
prefix: /nix/store/0qyqmaj9if48r0aaim7l1n496ijq8rb4-nonlibc
The Meson build system
Version: 0.46.1
Source dir: /build/nonlibc
Build dir: /build/nonlibc/build
Build type: native build
Project name: nonlibc
Native C compiler: clang (clang 5.0.2 "clang version 5.0.2 (tags/RELEASE_502/final)")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Found pkg-config: /nix/store/5pb4a6hv8lc2vfr44ab4mhbljd500csw-pkg-config-0.29.2/bin/pkg-config (0.29.2)
Native dependency liburcu found: YES 0.10.1
Configuring nonlibc.h using configuration
Program test_fnvsum.py found: YES (/build/nonlibc/util/test_fnvsum.py)
Program test_ncp.py found: YES (/build/nonlibc/util/test_ncp.py)
Program pandoc found: YES (/nix/store/c26qlmjm46c8ysc9b5nw0n9ss652fi2h-pandoc-2.2.1/bin/pandoc)
Build targets in project: 26
Found ninja-1.8.2 at /nix/store/m9mj2m6i0s6gs10sbvv2w9d05s73v9s3-ninja-1.8.2/bin/ninja
building
[1/69] Compiling C object 'lib/lib@@nonlibc@sha/hx2b.c.o'.
[2/69] Compiling C object 'lib/lib@@nonlibc@sha/fnv.c.o'.
[3/69] Compiling C object 'lib/lib@@nonlibc@sha/nmem.c.o'.
[4/69] Compiling C object 'lib/lib@@nonlibc@sha/b2hx.c.o'.
[5/69] Compiling C object 'lib/lib@@nonlibc@sha/npath.c.o'.
[6/69] Compiling C object 'lib/lib@@nonlibc@sha/pcg_rand.c.o'.
[7/69] Compiling C object 'lib/lib@@nonlibc@sha/lifo.c.o'.
[8/69] Compiling C object 'lib/lib@@nonlibc@sta/hx2b.c.o'.
[9/69] Compiling C object 'lib/lib@@nonlibc@sha/posigs.c.o'.
[10/69] Compiling C object 'lib/lib@@nonlibc@sha/nmem_linux.c.o'.
[11/69] Compiling C object 'lib/lib@@nonlibc@sta/b2hx.c.o'.
[12/69] Compiling C object 'lib/lib@@nonlibc@sta/fnv.c.o'.
[13/69] Compiling C object 'lib/lib@@nonlibc@sha/epoll_track.c.o'.
[14/69] Compiling C object 'lib/lib@@nonlibc@sta/nmem.c.o'.
[15/69] Compiling C object 'lib/lib@@nonlibc@sta/npath.c.o'.
[16/69] Compiling C object 'lib/lib@@nonlibc@sta/pcg_rand.c.o'.
[17/69] Compiling C object 'lib/lib@@nonlibc@sta/lifo.c.o'.
[18/69] Compiling C object 'lib/lib@@nonlibc@sta/posigs.c.o'.
[19/69] Linking target lib/libnonlibc.so.
[20/69] Compiling C object 'lib/lib@@nonlibc@sta/nmem_linux.c.o'.
[21/69] Compiling C object 'test/test@@atop_test_shared@exe/atop_test.c.o'.
[22/69] Compiling C object 'lib/lib@@nonlibc@sta/epoll_track.c.o'.
[23/69] Linking static target lib/libnonlibc.a.
[24/69] Compiling C object 'test/test@@atop_test_static@exe/atop_test.c.o'.
[25/69] Compiling C object 'test/test@@fnv_test_shared@exe/fnv_test.c.o'.
[26/69] Compiling C object 'test/test@@hex2bin2hex_test_shared@exe/hex2bin2hex_test.c.o'.
[27/69] Compiling C object 'test/test@@fnv_test_static@exe/fnv_test.c.o'.
[28/69] Compiling C object 'test/test@@hex2bin2hex_test_static@exe/hex2bin2hex_test.c.o'.
[29/69] Linking target test/atop_test_static.
[30/69] Compiling C object 'test/test@@lifo_test_shared@exe/lifo_test.c.o'.
[31/69] Compiling C object 'test/test@@lifo_test_static@exe/lifo_test.c.o'.
[32/69] Linking target test/fnv_test_static.
[33/69] Linking target test/hex2bin2hex_test_static.
[34/69] Compiling C object 'test/test@@nlc_urand_test_shared@exe/nlc_urand_test.c.o'.
[35/69] Compiling C object 'test/test@@nlc_urand_test_static@exe/nlc_urand_test.c.o'.
[36/69] Linking target test/lifo_test_static.
[37/69] Generating symbol file 'lib/lib@@nonlibc@sha/libnonlibc.so.symbols'.
[38/69] Compiling C object 'test/test@@nmath_test_shared@exe/nmath_test.c.o'.
[39/69] Compiling C object 'test/test@@nmath_test_static@exe/nmath_test.c.o'.
[40/69] Linking target test/nlc_urand_test_static.
[41/69] Compiling C object 'test/test@@npath_test_shared@exe/npath_test.c.o'.
[42/69] Linking target test/atop_test_shared.
[43/69] Compiling C object 'test/test@@npath_test_static@exe/npath_test.c.o'.
[44/69] Linking target test/hex2bin2hex_test_shared.
[45/69] Linking target test/fnv_test_shared.
[46/69] Linking target test/lifo_test_shared.
[47/69] Linking target test/nlc_urand_test_shared.
[48/69] Linking target test/nmath_test_static.
[49/69] Linking target test/nmath_test_shared.
[50/69] Linking target test/npath_test_static.
[51/69] Linking target test/npath_test_shared.
[52/69] Compiling C object 'test/test@@pcg_rand_test_static@exe/pcg_rand_test.c.o'.
[53/69] Compiling C object 'test/test@@pcg_rand_test_shared@exe/pcg_rand_test.c.o'.
[54/69] Compiling C object 'test/test@@zed_dbg_test_shared@exe/zed_dbg_test.c.o'.
[55/69] Compiling C object 'test/test@@zed_dbg_test_static@exe/zed_dbg_test.c.o'.
[56/69] Linking target test/pcg_rand_test_static.
[57/69] Compiling C object 'test/test@@epoll_track_test_shared@exe/epoll_track_test.c.o'.
[58/69] Linking target test/pcg_rand_test_shared.
[59/69] Compiling C object 'test/test@@epoll_track_test_static@exe/epoll_track_test.c.o'.
[60/69] Linking target test/zed_dbg_test_shared.
[61/69] Compiling C object 'util/util@@fnvsum@exe/fnvsum.c.o'.
[62/69] Linking target test/epoll_track_test_shared.
[63/69] Linking target test/zed_dbg_test_static.
[64/69] Linking target test/epoll_track_test_static.
[65/69] Compiling C object 'util/util@@ncp@exe/ncp.c.o'.
[66/69] Linking target util/fnvsum.
[67/69] Linking target util/ncp.
[68/69] Generating man fnvsum with a meson_exe.py custom command.
[69/69] Generating man ncp with a meson_exe.py custom command.
running tests
[0/1] Running all tests.
 1/22 atop test (shared)                      OK       0.02 s
 2/22 atop test (static)                      OK       0.02 s
 3/22 fnv test (shared)                       OK       0.02 s
 4/22 fnv test (static)                       OK       0.03 s
 5/22 hex2bin2hex test (shared)               OK       0.01 s
 6/22 hex2bin2hex test (static)               OK       0.00 s
 7/22 lifo test (shared)                      OK       1.67 s
 8/22 lifo test (static)                      OK       1.67 s
 9/22 nlc urand test (shared)                 OK       1.18 s
10/22 nlc urand test (static)                 OK       1.16 s
11/22 nmath test (shared)                     OK       0.01 s
12/22 nmath test (static)                     OK       0.02 s
13/22 npath test (shared)                     OK       0.01 s
14/22 npath test (static)                     OK       0.01 s
15/22 pcg rand test (shared)                  OK       0.47 s
16/22 pcg rand test (static)                  OK       0.33 s
17/22 zed dbg test (shared)                   OK       0.01 s
18/22 zed dbg test (static)                   OK       0.01 s
19/22 epoll track test (shared)               OK       0.05 s
20/22 epoll track test (static)               OK       0.05 s
Traceback (most recent call last):
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/bin/meson", line 27, in <module>
    sys.exit(main())
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/bin/meson", line 24, in main
    return mesonmain.run(sys.argv[1:], launcher)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mesonmain.py", line 290, in run
    return mtest.run(remaining_args)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 712, in run
    return th.doit()
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 463, in doit
    self.run_tests(tests)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 621, in run_tests
    self.drain_futures(futures)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 637, in drain_futures
    self.process_test_result(result.result())
  File "/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 232, in run
    return self._run_cmd(wrap + cmd + self.test.cmd_args + self.options.test_args)
  File "/nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/lib/python3.6/site-packages/mesonbuild/mtest.py", line 276, in _run_cmd
    preexec_fn=preexec_fn if not is_windows() else None)
  File "/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/build/nonlibc/util/test_fnvsum.py': '/build/nonlibc/util/test_fnvsum.py'
FAILED: meson-test
/nix/store/8wfgl06xbcsxs6pmwr3n04kyhnvsh31d-python3-3.6.6/bin/python3.6m -u /nix/store/dby8969lmx2n5pj209qxzsa1x79qhhy5-meson-0.46.1/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.
builder for '/nix/store/mdsrsncbb5bqmmi15lim9lisrhix830p-nonlibc.drv' failed with exit code 1

[siriobalmelli]

I really appreciate you looking at this :)

Found an undeclared python3 dependency for that test file; also made it optional so it won't break the build. Pushed a commit fixing this.

When cloning, use the 'sirio' branch since this work is not pushed to 'master' yet:

git clone --branch=sirio https://github.com/siriobalmelli/nonlibc.git

[jtojnar]

Still the same on 8d129bec17da71736e8b51826de7e1171b9ec518

[jtojnar]

You will need to run patchShebangs on it, since /usr/bin/ is not available in sandbox.

[siriobalmelli]

oh, I see.

I'm unfamiliar with 'sandbox' ... until now I have been assuming that if it builds using nix-build or inside a nix-shell --pure (both building and testing cleanly on my machine - for output from another machine, see travis build) then things are kosher.

Unfortunately, no mentions of 'sandbox' in man nix-build or man nix-shell or nix --help | grep sandbox ... while the nixos manual only uses the term once in the context of nixos-rebuild which does not apply here.

... how do I replicate "sandbox" on my system so I can test that it will build there?
Is there an indication of when are things built "sandboxed" and when are they build "normally" (aka: Travis or any machine I have with Nix installed)?

re: patchShebangs ... I see it mentioned here ... am I expected to explicitly run it on any script that the build will call, in this case util/test_fnvsum.py?

[jtojnar]

Unfortunately, no mentions of 'sandbox' in man nix-build or man nix-shell or nix --help | grep sandbox ... while the nixos manual only uses the term once in the context of nixos-rebuild which does not apply here.

See man nix.conf.

re: patchShebangs ... I see it mentioned here ... am I expected to explicitly run it on any script that the build will call, in this case util/test_fnvsum.py?

Yep. Usually, grepping nixpkgs will find you a good example of use.

[siriobalmelli]

Didn't know about nix.conf - this opens up a whole bunch of options for me thanks :)

Build fixed by adding a patchPhase with patchShebangs here - it now builds in sandbox mode 👍

... still failing to propagate liburcu though.
But hey - progress.

All the help is very appreciated!

[jtojnar]

Works for me with https://github.com/siriobalmelli/nonlibc/commit/c07f7a58b96fb1d5ec794e25babf7743c7380b5f and https://github.com/siriobalmelli/xdpacket/commit/2b3ce34633c9a8aecb72e9e2adcaede62d84f3bc

[siriobalmelli]

Unfortunately that's the master branch of xdpacket - I have a specific dependency on liburcu here.

The 'sirio' branch removes that dependency and fails to build on my machine and also on travis

[jtojnar]

The issue is that you are overriding the default phases. That is not recommended because they often do some extra things, like running pre/post-phase hooks, or, as in this case, propagating the dependencies.

[siriobalmelli]

Thank you - that's what I was missing.
Fixed in this commit

Specifically - clobbering fixupPhase broke dependency propagation and stripping paths from binaries etc.

It seems like all the other phases in setup.sh are Makefile-specific, so I will continue overriding those.

Again thanks so much for the assistance.

[jtojnar]

The other phases are overridden by meson and ninja setup hooks. You really should never use *Phase unless you know what you are doing. If you need to run some command, hooks like preConfigure are much better choice.

[siriobalmelli]

Agreed.
In this case however I'm driving Meson very differently from how the 'meson' derivation does, so not much choice.

I'm much obliged you took the time to walk through this with me.
Not only do I have a much better understanding of Nix, but I feel this is a sufficiently well documented thread for anyone who runs into similar problems in the future 💯

[jtojnar]

I do not see any difference, other than passing a different CC. And for that you should rather replace stdenv (with clangStdenv, in this case).

[siriobalmelli]

... except not quite - for example all of these meson attributes are being set "under the hood".

I am approaching it from a standpoint of:

• The project must build and install cleanly with just Meson; Nix must be orthogonal to this
• A Meson-only build and a Nix build should give very similar results in terms of final file placement, etc.

I don't remember the exact issue why the 'meson' package didn't build for me out-of-the-box, but my takeaway from reading setup-hook.sh was "probably easier to simply declare my build steps directly".

52+ lines of BASH -> 11 lines directly in the Nix derivation.

win

I'm certain that as I get more familiar with Nix I'll realize the error of my ways (and better understand the interaction between variables in nix and what is actually run in BASH when hooks are executed, where/when environment variables like @cc@ are constructed, etc) and move to the meson package.

I think part of the trouble is that, being a new and very fast-growing project, Nix itself requires some experience with internals - there is no single-source-of-truth like the GCC manual or the glibc man pages. Nix has lots of documentation (and some of the most helpful people I've met in OpenSource ... seriously, the amount of work you've done to help me resolve this is really appreciated - beer owed when we cross paths at a conference sometime), but the structure or "con-ops" is hard to glimpse coming from the outside.

From my part I'll keep trying to grok Nix as I integrate it with all my projects, hoping I can also start contributing and helping to improve the learning curve for newcomers.

[jtojnar]

This line replaces @var@s with environment variables in setup hooks:

https://github.com/NixOS/nixpkgs/blob/8bd8c09f98ece814185a0e9b0285ac3c5e51d062/pkgs/stdenv/generic/setup.sh#L1138

---

https://github.com/NixOS/nixpkgs/blob/8bd8c09f98ece814185a0e9b0285ac3c5e51d062/pkgs/development/tools/build-managers/meson/setup-hook.sh#L14-L21 is just used for multiple outputs. It is a no-op when not using that.

---

-Dauto_features=disabled is currently a no-op. I will make it configurable when we update to meson which supports it.