Nixpkgs: libc should just be another library

[Ericson2314]

Right now, libc is special: instead of just being another library dependency, it is passed to cc-wrapper where a bunch of magic happens for dynamic linking. I rather it just be another library with the caveat that if you don't depend on it dynamic linking won't work. Worst case, some of that dynamic-linking logic would go from the wrapper to the setup hook. mkDerivation would add that dependency by default, but packages (e.g. libc itself for obvious reasons) would opt out.

Relatedly, when bootstrapping (a new stdenv from a tarball or a cross stdenv), we current build "static" versions of the compiler that's aren't wrapped with libc, and then build libc, and then build the compiler again. At the very last, I'd like to just re-wrap the old compiler with libc and avoid the rebuild, but if we're not wrapping libc too, then let's just build and wrap the compiler once.

I assume there's a reason for all the double wrapping and double building we do now, but I don't know what it is.

CC @copumpkin @vcunat @DavidEGrayson @elitak @dmjio @eternaleye

[DavidEGrayson]

I have some knowledge of this from my experience building a cross-compiler to target Windows in Nix. If you can figure out how to build a GCC toolchain just once instead of building it twice, that would be cool, but I'm not sure it's supported by GCC.

When compiling a mingw-w64 cross-compiler, I know libstdc++ needs to see libwinpthreads when it is configuring itself so that it can use that library to implement things like std::future. That makes me believe that more generally, libstdc++ needs to see your libc and other basic system libraries when it is being configured. Since libstdc++ is a component of GCC and I am not sure if you can compile it on its own, that makes me think you have to get those libraries compiled with some sort of compiler before compiling the final GCC, so you need to build two different GCCs.

It's certainly possible I'm missing something here. If you want to make libc just be another library, you should make sure you can also have libstdc++ just be another library.

[Ericson2314]

@DavidEGrayson thanks for that info! Sounds like to start, we'll still need to build GCC twice, but that shouldn't impede the cc-wrapper side of things--i.e. maybe the second GCC will build fine if it has libc as a normal dep instead of from the first gcc's wrapper script?

[eternaleye]

@DavidEGrayson

Since libstdc++ is a component of GCC and I am not sure if you can compile it on its own

It's worth noting here that it _is_ possible to split the build process of GCC considerably further than even that. In particular, Exherbo has managed to split out all of the following into distinct packages:

• libatomic
• libcilkrts
• libgcc (for libgcc_s.so)
• libgfortran
• libgomp
• libquadmath
• libstdc++

All of these can be found in this directory of the arbor repository; gcc itself (with libgcov still bundled, as well as crt*.o and libgcc.a) can be found here.

Exherbo _does_ still require rebuilding, however - the threads option on GCC requires the ability for the build process of the GCC package (in libgcc.a, I think) to link against the target libc, so one builds first without threads, then the target libc, then with threads. Building libgcc.a has not yet been factored out.

[Ericson2314]

Oooo that's so tantalizingly close.

[eternaleye]

Yup. At very least, it _does_ allow the second GCC to treat libc as a normal dep, right out of the gate.

[vcunat]

The (main) motivation is reducing the amount of compiling when bootstrapping?

[Ericson2314]

@vcunat Yes that, because when cross compiling there's no cache to rely on. But also simplicity for simplicity's sake---I'm convinced that with enough blood sweat and tears, bootstrapping can become dramatically simpler.

[vcunat]

There are some cross tools in cache, I believe, mainly from http://hydra.nixos.org/jobset/nixpkgs/cross-trunk. If there are more cross-scenarios that are "considered supported/maintained", I think it would be good to put a few basic packages in there for them.

It's not just for having binaries, but also for easier keeping an eye on breakages (and it simplifies finding when they started).

[Ericson2314]

Oh, I'm thinking of soon proposing we try to cross build the whole world for some platform, now that the infrastructure is getting good enough for that to be a realistic goal.

But for embedded stuff and other things, I still suspect people will be forced to fine-tune crossSystem breaking the cache.

[Ericson2314]

https://github.com/NixOS/nixpkgs/issues/38451 would be easier to write if libc and the C compiler's runtime libs were propagatedBuildInputs, which is implied by this.

[Ericson2314]

https://github.com/richfelker/musl-cross-make is some very interesting prior art.

[DavidEGrayson]

Yes, I referred to that project when making my own cross-compilers for Linux, though I just stripped it down to the bare essentials and I was only concerned with static linking so I didn't try to get dynamic linking working.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Ericson2314]

Still do

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/gcc-separation-of-language-frontends-and-the-print-prog-name-option/9116/3