Nixpkgs: glibc: Strip bonus libraries

[Ericson2314]

Glibc currently includes libiconv and libiintl from gettext. This is a constant thorn in our side because (all?) other libcs do not. This creates conditional soup everywhere for Darwin (and linux-musl, and other unix), and generally lures Linux software into being less portable than intended. We should build glibc without them to the extent that is permitted; I bet we could even get another distribution to follow suit!

Prerequisites:

• #36948: No libgcc thrown in with glibc
• #37012: Always depend on gettext and libiconv

---

CC @matthewbauer

[matthewbauer]

One thing that might be easily fixed:

"gettext" does not provide libintl for Linux while it does for macOS. "libiconv" provides libiconv regardless of system though.

[matthewbauer]

@Ericson2314 Is there a definitive list of these bonus libraries in Glibc? I just wonder if there could be a few other unusual libraries that might come up.

Related to this, I just found this link of "bonus libraries" in libSystem, which is the libc used in Nixpkgs/Darwin world:

https://docstore.mik.ua/orelly/unix3/mac/ch05_02.htm

I don't know if any of those aren't in Glibc?

[matthewbauer]

-nodefaultlibs might do this in GCC

https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html

[matthewbauer]

Interestingly, I haven't found a comprehensive list of what these "bonus" libraries are. So I decided to make my own:

| Bundled Library (no need for -lLIBNAME) | Glibc | libSystem | musl | MSVCRT | BSD libc | Bionic | Newlib |
|------------|-------|-----------|-------|--------|----------|--------|--------|
| libm | [ ] | [X] | [X] | [?] | [?] | [?] | [?] |
| libiconv | [X] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libintl | [X] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libpthread | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libinfo | [ ] | [X] | [ ] | [?] | [?] | [?] | [?] |
| libcurses | [ ] | [ ] | [ ] | [?] | [?] | [?] | [?] |
| libresolv | [ ] | [ ] | [ ] | [?] | [?] | [?] | [?] |
| librt | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libcrypt | [ ] | [X] | [X] | [?] | [?] | [?] | [?] |
| libxnet | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libdl | [ ] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libutil | [ ] | [X] | [X] | [?] | [?] | [?] | [?] |

The program used to generate is:

#include <sys/types.h>
#include <stdint.h>
#include <time.h>
#include <iconv.h>

#define bool int

double acos (double); // m
iconv_t iconv_open(const char *, const char *); // iconv
char *gettext (const char *); // intl
pthread_t pthread_self(void); // pthread
void si_search_module_set_flags(const char *, uint32_t); // info
bool has_colors(void); // curses
void res_close (void); // resolv
int clock_getres(clockid_t, struct timespec *); // rt
char *crypt (const char *, const char *); // crypt
int gethostname(char *, size_t len); // xnet
int dlinfo(void *, int, void *); // dl
int login_tty(int); // util

int main() {
  (void)acos(15);
  iconv_open("", "");
  gettext("HI");
  pthread_self();
  si_search_module_set_flags("", 0);
  has_colors();
  res_close();
  clock_getres(CLOCK_MONOTONIC, NULL);
  crypt("", "");
  gethostname((char *) "", 0);
  dlinfo(NULL, 0, NULL);
  login_tty(0);
}

If anyone has musl, MSVCRT, BSD libc, Bionic, or Newlib setup, please try compiling the above program (with no library flags) and report the results to me!

nix-shell -p stdenv.cc --run 'cc test.c'

Also report which are missing when you run this in a Nix stdenv:

nix-shell -p stdenv.cc --run 'cc -lm -liconv -lintl -lpthread -linfo -lcurses -lresolv -lrt -lcrypt -lxnet -ldl -lutil'

| Available Library (succeeds with -lLIBNAME) | Glibc | libSystem | musl | MSVCRT | BSD libc | Bionic | Newlib |
|------------|-------|-----------|-------|--------|----------|--------|--------|
| libm | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libiconv | [ ] | [X] | [ ] | [?] | [?] | [?] | [?] |
| libintl | [ ] | [ ] | [ ] | [?] | [?] | [?] | [?] |
| libpthread | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libinfo | [X] | [X] | [ ] | [?] | [?] | [?] | [?] |
| libcurses | [X] | [ ] | [ ] | [?] | [?] | [?] | [?] |
| libresolv | [X] | [X] | [ ] | [?] | [?] | [?] | [?] |
| librt | [X] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libcrypt | [X] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libxnet | [X] | [ ] | [X] | [?] | [?] | [?] | [?] |
| libdl | [X] | [X] | [X] | [?] | [?] | [?] | [?] |
| libutil | [X] | [X] | [X] | [?] | [?] | [?] | [?] |

[dtzWill]

Here's recent Nixpkgs-musl:

$ nix run -f channel:nixos-unstable --arg localSystem '{config="x86_64-unknown-linux-musl";}' stdenv.cc -c cc test.c -o test
/tmp/ccbfeaJi.o: In function `main':
test.c:(.text.startup+0x3b): undefined reference to `si_search_module_set_flags'
test.c:(.text.startup+0x40): undefined reference to `has_colors'
test.c:(.text.startup+0x45): undefined reference to `res_close'
collect2: error: ld returned 1 exit status
$ nix run -f channel:nixos-unstable --arg localSystem '{config="x86_64-unknown-linux-musl";}' stdenv.cc -c cc test.c -o test -lm -liconv -lintl -lpthread -linfo -lcurses -lresolv -lrt -lcrypt -lxnet -ldl -lutil
/nix/store/nkqpf3p6hl255dnx459ap27lwn407jiz-binutils-2.28.1/bin/ld: cannot find -liconv
/nix/store/nkqpf3p6hl255dnx459ap27lwn407jiz-binutils-2.28.1/bin/ld: cannot find -lintl
/nix/store/nkqpf3p6hl255dnx459ap27lwn407jiz-binutils-2.28.1/bin/ld: cannot find -linfo
/nix/store/nkqpf3p6hl255dnx459ap27lwn407jiz-binutils-2.28.1/bin/ld: cannot find -lcurses
collect2: error: ld returned 1 exit status

LMK if there's anything more I can do to help!

[matthewbauer]

@Ericson2314

Have you looked into filtering out the symbols from all Libcs? This is basically what is done on macOS & libSystem in Nixpkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_c_symbols

Having a restricted list of symbols for all Libcs would help with the "fail one place, fail everywhere" philosophy. I'm not sure how big a breakage would be for this, but as long as there is a flag to disable it in places that need the hidden symbols, it might be useful.

Also, I was looking to see if there is a way in either LLVM or GCC to disable compiler specific features? Basically if we could break everything that uses specific features (and have packages that rely on them explicitly set a flag to enable them).

[dezgeg]

Why not just do it the reverse way and have Darwin include whatever glibc includes?

[Ericson2314]

@matthewbauer I haven't looked into either of those, but sounds fruitful. I'd have hoped there might just be preexisting obscure configure flags for glibc though.

@dezgeg Do you think that would be easier? I would have assumed those are equally difficult.

[Ericson2314]

@matthewbauer I haven't looked into either of those, but sounds fruitful. I'd have hoped there might just be preexisting obscure configure flags for glibc though.

@dezgeg Do you think that would be easier? I would have assumed those are equally difficult.

[dezgeg]

Isn't it just enough to have the respective stdenvs but libiconv et al to buildInputs? Possibly with some -Wl,--as-needed -iconv etc. in the default linker flags.

[matthewbauer]

Isn't it just enough to have the respective stdenvs but libiconv et al to buildInputs? Possibly with some -Wl,--as-needed -iconv etc. in the default linker flags.

I guess we could do that. For something like Musl it might break the "spirit" of having a stripped down Libc but the -as-needed would hopefully take care of that issue. The only case I can think of where it might fail is if there is some sort of feature detection going on where iconv is only used when it is available. That's probably not that common though.

Since libiconv is not used that much, it doesn't seem to bad to have to declare it as a build input everywhere it is needed. For some of the others, it may get more annoying & just always adding them to stdenv might make more sense.

[Ericson2314]

I still rather have more cc-wrapper extraPackages than a fatter libc derivation.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.