Nixpkgs: Referencing anything in `pkgs.pkgsMusl` causes hours-long build.

[LBensman]

Issue

Synopsis

Referencing any package inside pkgs.pkgsMusl causes a massive rebuild that takes hours to run. E.g. Creating a MUSL stdenv with clang takes hours just to create the build environment, making it prohibitive. Even simply running nix-shell -E '(import <nixpkgs>).pkgsMusl.minixml' - a simple library reference - will take hours to complete.

Details

I'm trying to upgrade our nix-based development/CI build environment from release 16.03 to 19.03 or later. Our default.nix defines couple of packages (with their respective dependencies), and because at least one package we need to build fully static-linked, we setup stdenv with MUSL to build our binaries (using clang / llvm). To upgrade some of the code, we need latest llvm (9 preferably), and because packages definitions changes b/w 16.0x and 19.0x, I was unable to simply import llvm-9 definitions (we did import llvm-5 definitions from some post 16.03 release for w/e reason, but same no longer works for llvm-9). And it's just plain seems like a good idea to get everything to the latest.

Examining 19.03 and later releases, I found that nix seems to be now having a direct definition for MUSL based packages and pretty straight-forward way of creating MUSL based environment for building our own packages, i.e. muslStdenv = stdenvAdapters.overrideCC pkgsMusl.stdenv pkgsMusl.clang;, but noticed that build was taking forever. Trying to figure out what takes the most time, I reduced the default.nix to basically an equivalent of just evaluating pkgs.pkgs.Musl.minixml (one of the libraries that is a build input into one of our projects), and that triggers the same long build.

I then simply rebuilt docker image with plain vanila nixOS, and simply tried to run nix-shell -E '(import <nixpkgs>).pkgsMusl.minixml' and got the same result - hours long build that seems to be building 80+ packages.

Here is the output of declaration of what nix-shell is doing before messages from downloading and pulling:

these derivations will be built:
  /nix/store/6xq3i95sxyg14rhh58h41bik44jayngx-bootstrap-tools.tar.xz.drv
  /nix/store/r5fb6xnsg45m490zaj40z2z804crr5bh-busybox.drv
  /nix/store/ss114s0h3cpdzji536d21xd0fdcaakr0-bootstrap-tools.drv
  /nix/store/0gfxf884791yyhqjwdnpi3x31ppkzwmh-bootstrap-stage2-stdenv-linux.drv
  /nix/store/a3drsla370z3vvr0jih6rbmpd114gnaf-bootstrap-stage0-stdenv-linux.drv
  /nix/store/3xx6z36gzw7lqfj80lv6fhjfvbrwyk27-bootstrap-stage0-musl.drv
  /nix/store/1z9ry7fkkvnn10bzgxd69gfzj80l7iwk-bootstrap-stage0-binutils-wrapper.drv
  /nix/store/95ns6xh9ld4n45p4v2x993axas6a9j2d-bootstrap-stage1-gcc-wrapper.drv
  /nix/store/dfmzcdyyc1v877a3ilhbr5c9vykcmdvh-bootstrap-stage1-stdenv-linux.drv
  /nix/store/hddbyk3f20wiq7idpvhnsvgzsyxcqcp0-expand-response-params.drv
  /nix/store/c8607vm8v50m41r89bfh3bjbsi7i89p9-bootstrap-stage1-stdenv-linux.drv
  /nix/store/vhc6559rzxiawhmr7vjm7yvdx5znb2w4-zlib-1.2.11.drv
  /nix/store/1wbqgfxgcfa56anjp0mamnv4l89ldamr-perl-5.28.1.drv
  /nix/store/yviq51y6fafaa1wdjwd9h41b8avrv6fz-gnum4-1.4.18.drv
  /nix/store/y60pfgkacl4abil4c4lggdc9kav0fd00-bison-3.3.2.drv
  /nix/store/d27l41471jvak2pxvdwx4w20nqc74nj9-binutils-2.31.1.drv
  /nix/store/hxp05x2xl94z9arg9jqmvwa7qxzlfcg4-binutils-wrapper-2.31.1.drv
  /nix/store/fvhwkyfaj1i8y9ymjyj0sdpvggxm4ma1-bootstrap-stage2-gcc-wrapper.drv
  /nix/store/0pr7fnyn3yk3bhi5fc85z4ccbwlaax7d-bootstrap-stage2-stdenv-linux.drv
  /nix/store/14j85xr2sxyc3r9yhm155vrqwaiik3sk-patchelf-0.9.drv
  /nix/store/xyq3wlj7zr7hbc8qqk2nxn6sdxyf2jb4-gnu-config-2016-12-31.drv
  /nix/store/n5bw0h9ixh1qqbprm3n2xk2cj9h8mwms-hook.drv
  /nix/store/ib3svlabvi5yb8vv5ncdy90k4fc7blf7-linux-headers-4.19.16.drv
  /nix/store/hqrsx8l1i4c6bayhvdcprywb5pn1xjcw-musl-1.1.21.drv
  /nix/store/m75y8r42f7vd26kfrfpyzphh0m163pbw-binutils-wrapper-2.31.1.drv
  /nix/store/q7275f2wwhaz9205awz9vmw8jbmw9bna-expand-response-params.drv
  /nix/store/pg0k1sc0478h9r5silscvvdbck9bkmn5-bootstrap-stage3-gcc-wrapper.drv
  /nix/store/1iwvikpynzymz2r1yb4mnwrin33b9b48-bootstrap-stage3-stdenv-linux.drv
  /nix/store/i5nrnm8j5hb03451cr38y0d0v35x06gz-xz-5.2.4.drv
  /nix/store/g8sxlilj1smqnabd4gm92lj7q9204w9z-gettext-0.19.8.1.drv
  /nix/store/6gf9rkj6khy9mrhyipnl23wwszay38hi-which-2.21.drv
  /nix/store/cb6n6nlw1nzlmfis3b3d3p41q178i75x-gmp-6.1.2.drv
  /nix/store/gpsc265lfsn7rnhd7h5i67qrwv09r5fs-isl-0.17.1.drv
  /nix/store/yci1llck5c6pxwmmafjrh8hp86zg496p-mpfr-4.0.2.drv
  /nix/store/lj5cwjc7xid46r697fdnzbmw1cshqczf-libmpc-1.1.0.drv
  /nix/store/qivd1bfn47pwpkdh1s4f3a6vl5hrkdcc-texinfo-6.5.drv
  /nix/store/qlzm85y97dqrx4pf39x05h5aa8l9fz3j-libelf-0.8.13.drv
  /nix/store/ybsky6cmsil1rkdqhyjzx0yg9x2g91i7-zlib-1.2.11.drv
  /nix/store/1jk1gyymz2mnc7bxfp3jgjzd4dbf1alr-gcc-7.4.0.drv
  /nix/store/x4mjvbprx5wbviwqpzpn16kl2whbgdzg-expand-response-params.drv
  /nix/store/pmb7vmvysqvlbih882sdxfxpbyx04dy6-bootstrap-stage4-gcc-wrapper-7.4.0.drv
  /nix/store/dvmdw105si5k3rd3mhcdb5yic8bhxxl6-gnu-config-2016-12-31.drv
  /nix/store/qmnc4sfwnzllmj7dsnvjf87cqhma1mpq-bootstrap-stage3-stdenv-linux.drv
  /nix/store/yshlmj46ww99ij39wha15gzg1znvn87g-hook.drv
  /nix/store/lclgamqfsvldnqwqggsi88gdd0s4c50c-bootstrap-stage4-stdenv-linux.drv
  /nix/store/m2gfsn53x2g3vdhvc24z240mml601941-autoconf-2.69.drv
  /nix/store/vqij6gcyknfdk442akdbjw69djsxl1km-bootstrap-stage4-stdenv-linux.drv
  /nix/store/vrs1cbg8i68b5ci99567nsmpmbfhqq5c-automake-1.16.1.drv
  /nix/store/5mnc314pyi5jqk8ypb8fz60jaz2clv8a-perl-5.28.1.drv
  /nix/store/9qw99i2cw36r1nql2kw95pmdv4bzm644-perl5.28.1-gettext-1.07.drv
  /nix/store/lqhnj9kwdqbvp2dphyx4zxbccrpi4a7d-help2man-1.47.8.drv
  /nix/store/xmpdd9sg0zaahm6wh4fc591bl2rxj91i-libtool-2.4.6.drv
  /nix/store/1j9l5gd0qcfgy3mbkpw7vh5imzi0zrjq-hook.drv
  /nix/store/1kpnf6ivbg62lyq92rp5dkhpj4kgzgyp-binutils-2.31.1.drv
  /nix/store/sw9gcsyklbjrf891rphfjx6cnda1f41r-attr-2.4.48.drv
  /nix/store/d7dk4i5477qn219n5nmpdj13mkisznqz-acl-2.2.53.drv
  /nix/store/m845znawabd1wngijvzvhspg5bdy70if-xz-5.2.4.drv
  /nix/store/p8s1vkkmg51866yp8vaglhrnlhvx6n2v-coreutils-8.30.drv
  /nix/store/7kmd23giyld1qmk1fbscdqnjh3sbi2qh-findutils-4.6.0.drv
  /nix/store/7nqazisxw0l4cjba2g60rikw751k1xs6-musl-iconv-1.1.21.drv
  /nix/store/81c2skbgf30spi8gvz7vz4srcrss8imf-gawk-4.2.1.drv
  /nix/store/asriz7030bkc6ggx75l1nr41zi6531bc-patchelf-0.9.drv
  /nix/store/b83mvvgh2aprpaqdk8nx567dbr2byg1p-bzip2-1.0.6.0.1.drv
  /nix/store/hhd5gp5552zs1323sckq3b86ka07z6lq-pcre-8.42.drv
  /nix/store/cdj40nszv51fkppw9ka9vr4sdrdvyd8v-gnugrep-3.3.drv
  /nix/store/cjsid5p0wsik665nnwnavnj4gddf4zjx-patch-2.7.6.drv
  /nix/store/d8n16q9fyvbhnfpsc7mg5k7cpclx3w8v-lzip-1.21.drv
  /nix/store/ds8vjmzh8224pi0v1rq08l2w3v6d1vxm-bash-4.4-p23.drv
  /nix/store/m70p486xy389j96vjxalwn56442hzi9i-gzip-1.10.drv
  /nix/store/p451mkg0z4zxav3kn00wamk3lrs7x424-gnu-config-2016-12-31.drv
  /nix/store/ni0lly5c5wwrhg4xs4g1amm1ar83fbjg-hook.drv
  /nix/store/nms805b8abx5h59fhdyhammc50ihqp1i-gnutar-1.31.drv
  /nix/store/rphcf89dsxx1qcfp1vfbizkbi7ilaq3n-diffutils-3.7.drv
  /nix/store/rvz3h2zx5byzvd2pxhrml5vv3rv6i67s-ed-1.15.drv
  /nix/store/v4wh92l1m1cq24s6c081r45mmyyan5pk-gnumake-4.2.1.drv
  /nix/store/zsp1abl05ha3z720bgsq36i8ps0896y7-gnused-4.7.drv
  /nix/store/zz82bcgk6sxpcsv8cfhxv9w3wx968764-binutils-wrapper-2.31.1.drv
  /nix/store/zvhw854n9npp72ksrqvs1yxccwih9hdk-gcc-wrapper-7.4.0.drv
  /nix/store/l1j9csyk8pqfsyfz1ln8qrjvnwyms1j7-stdenv-linux.drv
these paths will be fetched (292.23 MiB download, 328.29 MiB unpacked):
  /nix/store/01850b04ksfk9svg24pqw7ysd2h1n2sv-mpc-1.1.0.tar.gz
  /nix/store/0fpf4zfxzkz5yxcdp4wbyy1z52yzj1kw-acl-2.2.53.tar.gz
  /nix/store/202rpp7v7f21bx2kq06dpc6gaic2gvjh-bash44-011
  /nix/store/22w7prrkdk70ll462csh1j02q5ajrr51-libtool-2.4.6.tar.gz
  /nix/store/2bim63kxh3vi6ka6inclfpba9rsjj6v9-iconv.c
  /nix/store/2fr3wiy7q2i4gan2f8w27c9hrsr9x13c-bash44-021
  /nix/store/349f140sh7m12vy9mp1wkp58axyp06c9-which-2.21.tar.gz
  /nix/store/420qx5p8cc7w4jakspms2c73430xwnyn-make-4.2.1.tar.bz2
  /nix/store/4qwnbkw0v5a6dkg5zj67znwa7ilj8362-texinfo-6.5.tar.xz
  /nix/store/5086pd6ljw546sahagxzjg53d5hdrq09-bash44-009
  /nix/store/53ar6y7s69wmgf5bsb2grphlq5hp2zfd-gettext-1.07.tar.gz
  /nix/store/5f2i9rhs0pi5g8m762x4j55llm3z60xg-0001-Fix-missing-build-dependency-for-pods.patch
  /nix/store/5rcakigix8xmpkmcx90a083pkf8v0gwp-sys-queue.h
  /nix/store/66cw9xz3havfa345mzk6vjjhallh02xx-bash-interactive-4.4-p23-info
  /nix/store/6afjbhgxgl3jrwiwackzslkjcn0bhd1z-?id=14adc898a36948267bfe5c63b399996879e94c98
  /nix/store/7danzrkb22gignb90jhhl508k0x74fid-bash44-002
  /nix/store/7ir1ygibs1ap5jca2mf403srz9jjqj2g-bzip2-1.0.6.0.1.tar.gz
  /nix/store/8javli69jhj3bkql2c35gsj5vl91p382-linux-4.19.16.tar.xz
  /nix/store/8lim9yzfpmxf0h62l2wylmgrwg7vx0x3-diffutils-3.7.tar.xz
  /nix/store/8lz4k7gnsj4mj44y7vwcrhxw92m00xwp-gmp-6.1.2.tar.bz2
  /nix/store/9mhyazyw3rv5lmibkyb1x61lsw29fqch-bash44-019
  /nix/store/9x7zwqc14g6knm77r6s23z2jk0plwp4c-bash44-010
  /nix/store/bn77kf4krrcmaigccmjgnvs1xpr6kg85-perl-5.28.1.tar.gz
  /nix/store/byib83r321c1fjgdk37kai1q1zxncra7-sys-cdefs.h
  /nix/store/d0hhqckk1az5b4ggkxair3abs1gspl9n-config.guess?id=b75cdc942a6172f63b34faf642b8c797239f6776
  /nix/store/d5iz3skv66k7vx15pnbjq8xgyfk7hy6w-bash44-022
  /nix/store/dnq5biyvz32biyyqckddr1b0q0gb0a1b-isl-0.17.1.tar.xz
  /nix/store/dpalvmd30im5cf7pbzsnldfq0f9pa3ny-patchelf-0.9.tar.bz2
  /nix/store/f5gj2g7zdabqc35335ry2whpkl5skd2w-gzip-1.10.tar.xz
  /nix/store/fbkz234v3qllw6xpmxvww0l9awrg5wa4-coreutils-8.30.tar.xz
  /nix/store/g82xlipjxqlxqjx0x8m9kb8n71fs6hzr-bison-3.3.2.tar.gz
  /nix/store/gby40lx6vzrzky77ca4r1pwwwnl2skc2-help2man-1.47.8.tar.xz
  /nix/store/gffadngj1b38vrs3cw1davbndlfccj8r-bash44-013
  /nix/store/ggpgsklddx6h8dk0wkmlnk4brcxdb0jh-libelf-0.8.13.tar.gz
  /nix/store/gpyhb16nhbr31g9mf0n45scyv31ay6pp-ed-1.15.tar.lz
  /nix/store/h0p0h3rh1q4i2yavzm3yqi716s9yaj2f-glibc-2.27
  /nix/store/h2lap34xv5znjb5wcgqcj813939gwd2h-bash44-018
  /nix/store/hhxq0jb8dbkw68z5dvarwgdrliamr76c-bash44-003
  /nix/store/hl7wxzav5znk5wxfqncdrzpcb3hqy0y0-patch-2.7.6.tar.xz
  /nix/store/hvzczmqkw3pfjws27xz0q538d4ikms9y-gawk-4.2.1.tar.xz
  /nix/store/i95jrwl41kb92jai94vagzfgrg2vpww3-lzip-1.21.tar.gz
  /nix/store/id71yr7x2cbrrr8pb51qf2jwjq19rkfq-xz-5.2.4.tar.bz2
  /nix/store/ih1alzqkqpinbw745lrm48f7jfk5ab3b-bash44-008
  /nix/store/imbfbmn3kfk8k41h6zac62n9l6al3q1b-mpfr-4.0.2.tar.xz
  /nix/store/ipnh8pg3radfnhkx684ikysrvdnwr9rz-CVE-2018-1000156.patch
  /nix/store/issgq20n7k0z2dvgdsdgijrbd7j5fqvn-ncurses-6.1-20190112
  /nix/store/izjx6828l37dd9487zx3ik43zl3dz23j-Allow_input_files_to_be_missing_for_ed-style_patches.patch
  /nix/store/jkj92hfcl8kqfsvsk53pl1ksi1g9sqkh-tar-1.31.tar.xz
  /nix/store/kqykzisj9ll5dva7lr6hhjf0v1l5svar-bash44-007
  /nix/store/kzv56yx369j0jvcza2cphj2pv8xmqga4-bash-4.4-p23
  /nix/store/l0bbyc97wvycscj56rwsnskhag8zygi5-gcc-7.4.0.tar.xz
  /nix/store/lbzm411zx6ccqyxpgfkxrmdxvfs59dkl-findutils-4.6.0.tar.gz
  /nix/store/lfdjgq0mighjgs16wjrdl56f7zlgm18g-config.sub?id=b75cdc942a6172f63b34faf642b8c797239f6776
  /nix/store/li9vlg439jbbfz3anb7nx0a8dp2adsgc-bash44-012
  /nix/store/lk2db7rmwmbny9gwfzzvn42mlfl4vs47-musl-1.1.21.tar.gz
  /nix/store/lpv8wihhk84lx9vn0wa0m4hil8gx3cwd-patchJxugOXrY2y.patch
  /nix/store/lr5586viy3k10cphp6ka39r7qr8cirgg-bash44-017
  /nix/store/lsxbyvlsqk38fka1p2f9vlfnmxvhs8vi-bash44-016
  /nix/store/m7caj73xhn1ihald9ik025360ip8j4z1-pcre-8.42.tar.bz2
  /nix/store/mdmbvb93z195n3bij8pa838v3safw346-300-relative.patch
  /nix/store/mpsn3chi0m6y1xrxr3qy4q2sj6knl0d6-bash-interactive-4.4-p23-doc
  /nix/store/n1pp66v5mya20j0rq5mbzkak0pf6sal0-binutils-2.31.1.tar.bz2
  /nix/store/n781p1a8193xl6vf7y8z6s0rgi8gmg9h-bash44-004
  /nix/store/nca55pyd81bylc46s0h5mjvb6dk8azwa-bash44-014
  /nix/store/nqvgwsxabymj6k9cap1n4lnsbnpz9dx9-bash44-015
  /nix/store/r08yflna5b3x8f0fa8n2nkpw51slm411-bash44-005
  /nix/store/rn5dmlak31vg3fjv94y8gwy1pn6vbdab-bash-interactive-4.4-p23
  /nix/store/rnay1kyfxqvjhmk0bfj7g85q6zskk1xy-sys-tree.h
  /nix/store/rrsx38x833s7cihd6l0w1a1sq38hazrm-bash-interactive-4.4-p23-dev
  /nix/store/rrzja19jbqxbfryp2zchb2ma5h041rxh-zlib-1.2.11.tar.gz
  /nix/store/rv68r40mwx7xa7vrlmanczkkjcnkc452-bash-4.4.tar.gz
  /nix/store/skadi5jryx45iabyzyipk4zhv50n42c1-automake-1.16.1.tar.xz
  /nix/store/sm3m6c71sxa8rwzk85zllm9xjqrdvdb4-autoconf-2.69.tar.xz
  /nix/store/v03dcpm4mhgv2k327i2i6c62wmbxd7fv-readline-7.0p5
  /nix/store/v3l7j8ydmf998m1wilz05c561wz1gjqw-bash-interactive-4.4-p23-man
  /nix/store/v4zzrwb6m2wl3lpy8ycf82i49dgm947l-bash44-001
  /nix/store/v9lnrjcmf6bm655yjfamjvc82ydmgck9-m4-1.4.18.tar.bz2
  /nix/store/vfm3618g5cd5321y1vc5vaq9mg5lspc6-grep-3.3.tar.xz
  /nix/store/wdqd6k7fil667yvi6yarj9mpz2qqn1dl-sed-4.7.tar.xz
  /nix/store/wmq4fmib2r79gxmhis01cghcd7phrcgw-bash44-020
  /nix/store/x9cyj78gzd1wjf0xsiad1pa3ricbj566-bash44-023
  /nix/store/y6w1k6z7nzxhddj5dwj20clfd2q3pxb7-attr-2.4.48.tar.gz
  /nix/store/yha5ld53dgjnplaxjkmxdvgf6pmfd3mk-gettext-0.19.8.1.tar.gz
  /nix/store/zlvfcsdwmsfvyi9c3irsp1d83w0hz8gc-source
  /nix/store/zskj1jz5s2iix97iz5kxjyh3kqd8v28m-bash44-006

This seems excessive and, more importantly, impractical. Am I missing something? Shouldn't predefined pkgsMusl packages be either downloaded from cache.nixos.org just like their counterparts in pkgs collection, or at least just that package be compiled rather than entire environment? What's the proper way of creating MUSL build environment with clang/llvm, if not directly accessing it?

To Reproduce

Steps to reproduce the behavior:

1. Build Docker container with the following definition:

FROM lnl7/nix:2019-05-04
ADD . /srv/base-build/
ENV NIX_PATH nixpkgs=/root/.nix-defexpr/channels/nixpkgs
RUN mkdir -p ~/.config/nixpkgs && \
    echo '{ allowUnfree = true; }' > ~/.config/nixpkgs/config.nix && \
    cd /srv/base-build && \
    nix-channel --add https://nixos.org/channels/nixos-20.03 nixpkgs && \
    nix-channel --update

1. Start and attach to the new container, and in its shell execute: nix-shell -E '(import <nixpkgs>).pkgsMusl.minixml'.
2. Grab some ☕️ , learn fencing, and read some xkcd relevant comics.

Expected Behavior

Referencing packages in pkgs.pkgsMusl collection doesn't cause full environment build with 80+ packages to be compiled from source. Environment and packages should be a quick download, or at least environment - ok if individual package itself is built from source, but MUSL environment for building should be readily available (download is OK).

Metadata

bash-4.4# nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.9.125-linuxkit`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.2.2`
 - channels(root): `"nixpkgs-20.03beta-410.67c24f6f8a7"`
 - nixpkgs: `/root/.nix-defexpr/channels/nixpkgs`

P.S. I struggled if this was best to post here or over at Discourse channel, but it seemed lengthy enough and possibly suggestive of an existing issue that I opted to post here. If wrong forum, please kindly directed to a more appropriate one.

[FRidh]

In short, you are asking why most of pkgs.pkgsMusl need to be build from source, contrary to main pkgs? That's because we do not build adaptions on Hydra because of the huge amount of resources that would take.

[LBensman]

@FRidh , well, I don't necessarily mean that _every_ package that hangs of off pkgsMusl should be prebuilt, as I realize that's a lot of resources to generate and host. But I think general MUSL environment that all those packages need in order to build should be prebuilt and available to help avoid prohibitive lengthy builds. IOW, just have those ~80 derivations prebuilt and hosted on cache.nixos.org so that only the package referenced is built against a readily available MUSL build environment. This should be relatively minimal in resource demands.

The only other thing I'd suggest is to add MUSL-clang/llvm to those 80 prebuilt packages since it is likely something that others would often want in their build environment as well.

WDYT?

[FRidh]

For master the stdenv is built, however, channels are not gated on the results
https://hydra.nixos.org/jobset/nixpkgs/cross-trunk

[LBensman]

@FRidh, I'm not familiar with hydra and its configurations, so I'm not sure what you're trying to say with the cross-trunk link, nor what do you mean by not gated channels, nor which stdenv for which master?

I've been working with setup that someone else has created for our build/development environment, but otherwise I (still) have somewhat limited knowledge of nix. I've poked around to more-or-less understand our setup, but not enough to understand your references above. Can you please clarify what you mean?

[matthewbauer]

Yeah pkgsMusl is not actually cross-compiled, so it is not included in cross-trunk. We could add pkgsMusl.stdenv to release.nix I think which would at least give you the toolchain to start with.

[LBensman]

@matthewbauer, Ah, if I understand correctly what you're saying, it would get pre-built if it were a cross-compilation definition (run in one architecture and target a different one for output), but b/c its same architecture, it's not part of that cross-trunk job?

Yes, I think just building pkgsMusl.stdenv should get most of those long-compiling dependent derivations taken care of where one could just build their final target(s) directly.

The only other thing I'd suggest including with pkgsMusl.stdenv is pkgsMusl.llvmPackages_latest to include not just gcc-based toolchain, but clang/llvm toolchain as well so that environment is ready for both (as many use clang for C/C++ compilation). But if clang/llvm is too much, then at least pkgsMusl.stdenv would already go a long way in making that whole section useful!!!

[domenkozar]

@LBensman in the meantime you can use https://cachix.org/ for your own cache.

[domenkozar]

Fixed with https://github.com/NixOS/nixpkgs/pull/89425