Nixpkgs: Closure size of musl C++ programs bloated due to location of libstdc++

[gustavderdrache]

Describe the bug

When cross-compiling C++ programs for musl targets, the resulting closure is significantly larger (about 750-800MB extra) than programs compiled for my host machine (generic x86_64 linux). This closure includes both musl-gcc and my host gcc toolchain thanks to the cross-compilation.

I've done some digging and I believe that the root cause is due to the location of libstdc++.so and libstdc++.a in the musl stdenv.

To Reproduce
Steps to reproduce the behavior:

1. Clone or copy the files in this gist: https://gist.github.com/gustavderdrache/72af4fc8d01f67c59c822eb84e57109f
2. nix-build -A host
3. nix-build -A musl and compare closure sizes.

Expected behavior
The closure for a C++-compiled binary should only rely on the gcc-lib output, not the full gcc itself.

Additional context
Here's my sleuthing on the problem. When building a C++ "hello world" with my host toolchain, I get the following closure:

$ nix-store -qR $(nix-build -A host) | xargs du -hcs
30M /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27
5.6M    /nix/store/1220kf6lvlswh677wvizp9p51c6rcp3x-gcc-8.3.0-lib
28K /nix/store/s285860zxigbsxlc36kh995j301vvd39-hello
35M total

However, if I build the _same_ program in cross-compiling mode for musl gcc, everything gets carried along:

$ nix-store -qR $(nix-build -A musl) | xargs du -hcs
3.8M    /nix/store/7vs9r8304a6a2g70algfqcgikwhz4qsm-musl-1.1.24-x86_64-unknown-linux-musl
6.9M    /nix/store/v6y3vd1yk8p76g6hrm5nbn0s1ylaxrr9-linux-headers-4.19.16
30M /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27
1.4M    /nix/store/wd1jazzawjk4w1d31ism7fm7vdg4ma9l-bash-4.4-p23
1.3M    /nix/store/0nfmp5r1fkh1138ksa0d7w7rbavvqg3x-musl-1.1.24-x86_64-unknown-linux-musl-dev
5.6M    /nix/store/1220kf6lvlswh677wvizp9p51c6rcp3x-gcc-8.3.0-lib
700K    /nix/store/1fp2dwgv61f8hc76rrva11wlnv7q08rl-gmp-6.1.2
6.9M    /nix/store/fasbaxqn43vmzv58vb5z0rg90k4d2jli-linux-headers-4.19.16
2.7M    /nix/store/vl9ajbvls3y09i823cd66gsk9wrp6mw0-glibc-2.27-bin
3.5M    /nix/store/4a8wp97g0ddhjwx573k30x4fqzvkfn67-glibc-2.27-dev
152K    /nix/store/cg9l4lrvfc9azjsdzgfaxkcbfsmyyzmg-zlib-1.2.11
138M    /nix/store/nrisqq65br52ys4w6nly7vkjb2j10yy4-gcc-8.3.0
140K    /nix/store/ydaii11rxaycdkvybbaql34qhxm53hdn-zlib-1.2.11-dev
39M /nix/store/2fyhf3jfxp0xrbfabqbm610jgrynbvab-x86_64-unknown-linux-musl-binutils-2.31.1
28K /nix/store/3fadh5372m45h4h7y6d47fmx8aqz1wcq-expand-response-params
208K    /nix/store/mizf8jq9w3wkb6fk94wya8vc4mnzn4n8-attr-2.4.48
200K    /nix/store/6mzar1vxnd4g6wiabgjwpdd9mzhqrgwz-acl-2.2.53
548K    /nix/store/94mihb9rijc0yw5y98263f5185sdqs15-mpfr-4.0.2
216K    /nix/store/c77wf6mcprx3x8ncxpk9f59hk31wj8cj-gmp-6.1.2-dev
2.5M    /nix/store/cbxsbk38lsl04g3pik9hxn4m4lngnp3q-isl-0.17.1
88K /nix/store/f8wcg9lvrzzm2dh5fmnqdwn6i17h7fi3-mpfr-4.0.2-dev
244K    /nix/store/h88484hl6pr9x1il5g1isxnzr142x25j-libmpc-1.1.0
1.8M    /nix/store/gnw6yrqy249n62r4q8vy12ispviv3dav-coreutils-8.31
88K /nix/store/labmch9fr3ag9zwlvfc1s229x6iap1kb-x86_64-unknown-linux-musl-binutils-wrapper-2.31.1
1.3M    /nix/store/rhixry2rhx3ifciv74xcsrd423mynkkk--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0-lib
768M    /nix/store/gbz2mydglxzww5r4x9d6qdqbwadg00bn--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0
28K /nix/store/gyvg83g4sbcdq0vjajmnha9icbb2gd9r-hello-x86_64-unknown-linux-musl
1013M   total

From the output, it's pretty obvious that the entire closure of GCC has been carried along.

What I eventually noticed was that ldd says that libstdc++ is in gcc-lib for the host toolchain:

$ nix-build -A host
$ ldd result/bin/a.out
    linux-vdso.so.1 (0x00007ffc36db0000)
    libstdc++.so.6 => /nix/store/1220kf6lvlswh677wvizp9p51c6rcp3x-gcc-8.3.0-lib/lib/libstdc++.so.6 (0x00007f9c15463000)
    libm.so.6 => /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27/lib/libm.so.6 (0x00007f9c15233000)
    libgcc_s.so.1 => /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27/lib/libgcc_s.so.1 (0x00007f9c1501d000)
    libc.so.6 => /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27/lib/libc.so.6 (0x00007f9c14e67000)
    /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f9c153c9000)

However, the same is not true for cross-compiling against musl:

$ nix-build -A musl
$ ldd result/bin/a.out
    linux-vdso.so.1 (0x00007ffd3de58000)
    libstdc++.so.6 => /nix/store/gbz2mydglxzww5r4x9d6qdqbwadg00bn--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0/x86_64-unknown-linux-musl/lib64/libstdc++.so.6 (0x00007f49de3d0000)
    libgcc_s.so.1 => /nix/store/gbz2mydglxzww5r4x9d6qdqbwadg00bn--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0/x86_64-unknown-linux-musl/lib64/libgcc_s.so.1 (0x00007f49de3b7000)
    libc.so => /nix/store/7vs9r8304a6a2g70algfqcgikwhz4qsm-musl-1.1.24-x86_64-unknown-linux-musl/lib/libc.so (0x00007f49de2ad000)

Ultimately, what I see happening is that C++-compiled programs via the host toolchain have a RUNPATH that points to pkgs.stdenv.cc.cc.lib:

$ nix-build -A host
$ objdump -p result/bin/a.out
[snip]
Dynamic Section:
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  RUNPATH              /nix/store/qb6k4hp7gk331x9fydw0w7qj4dv09bwz-glibc-2.27/lib:/nix/store/1220kf6lvlswh677wvizp9p51c6rcp3x-gcc-8.3.0-lib/lib

The second path in the list is this:

nix-repl> "${(import <nixpkgs> {}).stdenv.cc.cc.lib}"
"/nix/store/1220kf6lvlswh677wvizp9p51c6rcp3x-gcc-8.3.0-lib"

On cross-compiled programs, however, the RUNPATH points to $out instead:

$ nix-build -A musl
$ objdump -p result/bin/a.out
[snip]
Dynamic Section:
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so
  RUNPATH              /nix/store/7vs9r8304a6a2g70algfqcgikwhz4qsm-musl-1.1.24-x86_64-unknown-linux-musl/lib:/nix/store/gbz2mydglxzww5r4x9d6qdqbwadg00bn--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0/x86_64-unknown-linux-musl/lib64

The REPL bears this out:

nix-repl> let
            musl = (import <nixpkgs/lib>).systems.examples.musl64;
            musl-pkgs = import <nixpkgs> { crossSystem = musl; };    
          in
          "${musl-pkgs.stdenv.cc.cc.out}"
"/nix/store/gbz2mydglxzww5r4x9d6qdqbwadg00bn--x86_64-unknown-linux-musl-stage-finalgcc-debug-8.3.0"

I was able to dig up two potentially-related issues. I didn't bump them because I wasn't sure how much like mine they were, but it's possible they've been bitten by this same bug:

• #35251
• #58981

Metadata
Please run nix run nixpkgs.nix-info -c nix-info -m and paste the result.

$ nix run nixpkgs.nix-info -c nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 5.3.0-20-generic, Pop!_OS, 18.04 LTS`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.1`
 - channels(user): `"nixpkgs-20.03pre202154.58fb23f72ad"`
 - nixpkgs: `/home/user/.nix-defexpr/channels/nixpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[grahamc]

cc @Ericson2314 and perhaps @dtzWill

[matthewbauer]

I think that https://github.com/NixOS/nixpkgs/pull/58606 would fix this.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[matthewbauer]

The situation is better with https://github.com/NixOS/nixpkgs/pull/81844, but still pulls in glibc:

$ nix-store -qR $(nix-build test.nix -A musl) | xargs du -hcs
7.3M    /nix/store/2hi983ysd510865ianwpwgkm0pzp4irn-linux-headers-5.5
1.6M    /nix/store/pgj5vsdly7n4rc8jax3x3sill06l44qp-libunistring-0.9.10
476K    /nix/store/9l6d9k9f0i9pnkfjkvsm7xicpzn4cv2c-libidn2-2.3.0
32M     /nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30
1.4M    /nix/store/81wybawvkr95c7j8gj5ab3y740mq1fli-bash-4.4-p23
3.8M    /nix/store/xh9awv1c9smj57p0df9p5542kr5kgp5x-musl-1.1.24-x86_64-unknown-linux-musl
1.3M    /nix/store/z2kigsl8nqdd9bc1ijwwh3041ivvidik-musl-1.1.24-x86_64-unknown-linux-musl-dev
4.8M    /nix/store/rvwa0ckpcyvkdymxpx6a2slc8gdprdyf-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0-lib
28K     /nix/store/hnbsd3nzv1aqj2mp746x5j7b08rwpv0x-hello-x86_64-unknown-linux-musl
52M     total

[matthewbauer]

Also:

$ nix why-depends -a ./result /nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30
/nix/store/hnbsd3nzv1aqj2mp746x5j7b08rwpv0x-hello-x86_64-unknown-linux-musl
└───bin/a.out: …known-linux-musl/lib:/nix/store/rvwa0ckpcyvkdymxpx6a2slc8gdprdyf-x86_64-unknown-linux-musl-stage…
    → /nix/store/rvwa0ckpcyvkdymxpx6a2slc8gdprdyf-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0-lib
    └───x86_64-unknown-linux-musl/lib/libgcc_s.so.1: …/../gcc-9.3.0/libgcc./nix/store/z2kigsl8nqdd9bc1ijwwh3041ivvidik-musl-1.1.24-x86_64-unknown-linu…
        x86_64-unknown-linux-musl/lib/libquadmath.so.0.0.0: …/../gcc-9.3.0/libgcc./nix/store/z2kigsl8nqdd9bc1ijwwh3041ivvidik-musl-1.1.24-x86_64-unknown-linu…
        → /nix/store/z2kigsl8nqdd9bc1ijwwh3041ivvidik-musl-1.1.24-x86_64-unknown-linux-musl-dev
        └───bin/ld.musl-clang: …#!/nix/store/81wybawvkr95c7j8gj5ab3y740mq1fli-bash-4.4-p23/bin/sh.cc="clang".…
            bin/musl-clang: …#!/nix/store/81wybawvkr95c7j8gj5ab3y740mq1fli-bash-4.4-p23/bin/sh.cc="clang".…
            bin/musl-gcc: …#!/nix/store/81wybawvkr95c7j8gj5ab3y740mq1fli-bash-4.4-p23/bin/sh.exec "${REA…
            → /nix/store/81wybawvkr95c7j8gj5ab3y740mq1fli-bash-4.4-p23
            └───bin/bash: …...................../nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib/ld-linux-x86-64.…
                lib/bash/basename: …basename.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/dirname: ….dirname.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/finfo: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/head: …BC_2.3.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/id: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/ln: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/logname: ….logname.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/mkdir: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/mypid: ….6.mypid.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/pathchk: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/print: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/printenv: …printenv.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/push: …o.6.push.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/realpath: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/rmdir: …ibc.so.6.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/setpgid: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/sleep: …LIBC_2.2.5.GLIBC_2.4./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/strftime: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/sync: …ibc.so.6.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/tee: …so.6.tee.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/truefalse: …ruefalse.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/tty: …so.6.tty.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/uname: …LIBC_2.4.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/unlink: …ibc.so.6.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                lib/bash/whoami: …6.whoami.GLIBC_2.2.5./nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30/lib.XXXXXXXXXXXXXXXX…
                → /nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30

[Ericson2314]

So we need a bin dir for musl?

[matthewbauer]

no i think we just need to strip / nuke-refs on libgcc_s.so.1 and libquadmath.so.0.0.0

[Ericson2314]

Oh, sweet.