Nixpkgs: glibc 2.27 breaks locale support

[peti]

I recently updated to the latest version of master @ 546696e367fe94159aa7765313d93caffd11fb37, and now my perl binary emits the following errors when I run non-trivial scripts (like mosh or mr):

~~~
$ mosh example.org
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

The locale requested by LC_CTYPE=en_US.UTF-8 isn't available here.
Running `locale-gen en_US.UTF-8' may be necessary.

mosh-client needs a UTF-8 native locale to run.

Unfortunately, the client's environment (LC_CTYPE=en_US.UTF-8) specifies
the character set "US-ASCII".

locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
~~~

I have

~
$ ls -Ll $LOCALE_ARCHIVE
-r--r--r-- 4 root root 114228912 Jan 1 1970 /run/current-system/sw/lib/locale/locale-archive
~

properly configured (as per nixos-unstable @ ea145b68a01).

[jtojnar]

I am getting the following warning when I run any GTK application from master.

(process:9154): Gtk-WARNING **: 12:00:47.472: Locale not supported by C library.
    Using the fallback 'C' locale.

Possibly something to do with recent glibc bump?

[dezgeg]

Are the failing programs also compiled against glibc 2.27 or are they using the old glibc? I wonder if glibc made an incompatible change to the locale file format again?

[peti]

According to git bisect, the commit that causes these issues is 273e58ebd9f8ef04948a89d496c7cb23dab8cbe8:

~~~
273e58ebd9f8ef04948a89d496c7cb23dab8cbe8 is the first bad commit
commit 273e58ebd9f8ef04948a89d496c7cb23dab8cbe8
Author: Shea Levy [e-mail]
Date: Sun Feb 18 16:52:44 2018 -0500

glibc: Make 2.27 the default.

:040000 040000 3eedda0d7ed484b1be7d3336999f28dba8c3c89f c3824531b2620a536e18f8749f3dd76fa0b7a686 M pkgs
bisect run success
~~~

Ping @shlevy.

[shlevy]

https://sourceware.org/glibc/wiki/Release/2.27#Statically_compiled_applications_using_locales seems relevant...

[shlevy]

5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_US/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
5130 openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

[shlevy]

Bah... Looks like localedef just creates broken archives.

[shlevy]

Ah! Figured it out. If you have LOCALE_ARCHIVE pointing to a pre-2.27 archive, you get this issue. We do have version-specific variants of LOCALE_ARCHIVE, so there are a few options I can see

• Do nothing. glibc 2.27 broke locale backwards compat and we just need users to run in a consistent environment
• Switch NixOS on release-18.03 to set LOCALE_ARCHIVE_2_11 (which is the version-specific env var from 2.11 through 2.26)
• Have NixOS on release-18.03 set LOCALE_ARCHIVE_2_27, which will require building glibc 2.27 locales there.

[shlevy]

@peti thoughts?

[coreyoconnor]

Trying to work this through. Adding my observations.

I have a machine using 2.27. When connecting via mosh the errors in the bug report are printed. I can reproduce that part.

If you have LOCALE_ARCHIVE pointing to a pre-2.27 archive, you get this issue

When ssh'ing to a machine using 2.27 I see:

[coconnor@grr:~]$ echo $LOCALE_ARCHIVE
/run/current-system/sw/lib/locale/locale-archive

[coconnor@grr:~]$ file -d /run/current-system/sw/lib/locale
/run/current-system/sw/lib/locale: symbolic link to /nix/store/8kw71xqs5bz6s68dylv6y13082zn0023-glibc-locales-2.27/lib/locale

Which, in that case, has a LOCALE_ARCHIVE not pointing to pre-2.27.

Does mosh reference a different LOCALE_ARCHIVE than the ssh session? Why is there different behavior here?

Figuring it might be perl specific I tried

[coconnor@grr:~]$ perl
print "hmm..."
hmm...

I also recall seeing messages during a nixos-rebuild of a system related to locale but don't have them on hand.

[shlevy]

@coreyoconnor Hmm, do you get the bug with ssh or just mosh?

[shlevy]

(e.g. if you run perl after connecting over ssh)

[coreyoconnor]

@shlevy

I see the error messages using just mosh. I do not see the error messages running perl after connecting over ssh. (The above session was all over ssh)

[shlevy]

Strange... Are you connecting to an old screen session or something? Or maybe mosh in your PATH is linked against glibc 2.27 or something?

[coreyoconnor]

Indeed! mosh is in .nix-profile and references pre-2.27:

uninstalling and including in system environment worked as expected.

[mguentner]

Ref: https://github.com/NixOS/nixpkgs/pull/38738

[shlevy]

OK, my current thoughts are:

1. NixOS should use the version specific env vars always
2. We should only bump the version specific env vars when there is a locale archive format change (which seems to be pretty rare...)
3. (the controversial piece) In the release before we bump to an incompatible glibc, we should also set the next version-specific env var, requiring us to build the next version's glibcLocales.

So, concretely this would mean:

1. Change master to set only LOCALE_ARCHIVE_2_27
2. Change 18.03 to set LOCALE_ARCHIVE_2_11
3. Change 18.03 to set LOCALE_ARCHIVE_2_27 to point to a glibc 2.27 locale archive

Since this is a somewhat sweeping change, I'd appreciate some additional thoughts on this. @peti @edolstra thoughts?

[peti]

My understanding is that this issue will disappear when nixos-unstable catches up with the master branch to contain glibc-2.27. For me personally, that is good enough, i.e. I'd be happy to "just wait".

As far as the versioned variable suggestion goes, I think it sounds good, but I haven't really thought about it and therefore don't feel like I have a qualified opinion on the matter.

[shlevy]

@peti It will still exist for anyone running nixos-18.03 and developing on master, but yes, you're generally correct.

[coreyoconnor]

One aspect of this will resolve once unstable catches up with the master branch.

The other aspect of this is similar to the "qt version mismatch" issue: A users profile may contain software incompatible with the services provided by the OS.

[dezgeg]

Changing the environment variable name sounds a bit annoying. What about changing the locale archive patch to instead try loading from "$LOCALE_ARCHIVE.$ABIVER" (where ABIVER is some compile-time constant we pick and change each time the incompatible change happens) first if such file exists. Then there is no need to do any preparation like "(the controversial piece) In the release before we bump to an incompatible glibc, we should also set the next version-specific env var, requiring us to build the next version's glibcLocales."

[shlevy]

@dezgeg Why wouldn't the controversial piece be needed on that model? We still wouldn't have a new locale archive version available to programs, so someone running current stable NixOS but testing master after a bump will see locale issues.

[shlevy]

(it's also not clear to me why a separate extension is better or worse than a separtae env var)

[dezgeg]

We still wouldn't have a new locale archive version available to programs, so someone running current stable NixOS but testing master after a bump will see locale issues.

Right, I didn't think it through completely. Yes that needs to be done.

(it's also not clear to me why a separate extension is better or worse than a separtae env var)

Because there are several places in Nixpkgs that for some reason or another do stuff like:
nixos/modules/installer/tools/nixos-enter.sh:LOCALE_ARCHIVE=$system/sw/lib/locale/locale-archive chroot "$mountPoint" "$system/activate" >&2 || true nixos/modules/services/misc/dictd.nix: environment = { LOCALE_ARCHIVE = "/run/current-system/sw/lib/locale/locale-archive"; }; n pkgs/development/haskell-modules/generic-builder.nix: LOCALE_ARCHIVE = optionalString stdenv.isLinux "${glibcLocales}/lib/locale/locale-archive";

If you got rid of the plain LOCALE_ARCHIVE variable and kept only the version-specific ones you'd be constantly updating all such usages.

[shlevy]

There'd be no need to get rid of the old one, and we already have the version specific one.

[dezgeg]

I interpreted your "Change master to set only LOCALE_ARCHIVE_2_27" as removing plain LOCAL_ARCHIVE. If you keep both then the first example:

nixos/modules/installer/tools/nixos-enter.sh:LOCALE_ARCHIVE=$system/sw/lib/locale/locale-archive chroot "$mountPoint" "$system/activate" >&2 || true
doesn't work because now there will be both LOCALE_ARCHIVE_2_27 and LOCALE_ARCHIVE in the environment and the former takes precedence so the plain LOCALE_ARCHIVE is effectively ignored.

[vcunat]

This really reminds me of our /etc/fonts/2.11; see https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/fontconfig/default.nix#L7

[peti]

Here is one more thing to consider: people who use Nixpkgs on a host system other than NixOS may have different versions of the glibc locale installed in their system, so they'll probably end up having to set those version-specific environment variables to make everything work. This is not exactly obvious, though, and we probably need to document how to do that.

[vcunat]

That's one of the things we considered in the fontconfig case IIRC.

[peti]

A simple NixOS module that works around this issue by providing version-specific LOCALE_ARCHIVE environment variables is available at https://gist.github.com/peti/2c818d6cb49b0b0f2fd7c300f8386bc3.

[timbertson]

people who use Nixpkgs on a host system other than NixOS may have different versions of the glibc locale installed in their system, so they'll probably end up having to set those version-specific environment variables to make everything work

nixpkgs ships a (seemingly) valid locale archive in glibcLocales which will always match the glibc version in nixpkgs. Couldn't we make that the primary locale by default, with a fallback for the user's system locale?

[rkoe]

This also breaks other programs, e.g. gnome3-aisleriot:

$ sol
Non UTF-8 locale (ANSI_X3.4-1968) is not supported!

$ locale
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_DK.UTF-8
LC_CTYPE="en_DK.UTF-8"
LC_NUMERIC="en_DK.UTF-8"
LC_TIME="en_DK.UTF-8"
LC_COLLATE="en_DK.UTF-8"
LC_MONETARY="en_DK.UTF-8"
LC_MESSAGES="en_DK.UTF-8"
LC_PAPER="en_DK.UTF-8"
LC_NAME="en_DK.UTF-8"
LC_ADDRESS="en_DK.UTF-8"
LC_TELEPHONE="en_DK.UTF-8"
LC_MEASUREMENT="en_DK.UTF-8"
LC_IDENTIFICATION="en_DK.UTF-8"
LC_ALL=

$ strace sol
...
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_DK.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_DK.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en_DK/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/locale/en/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/azxcljz7prb2qm36andkfc7zqzccb4ks-glib-2.56.0/lib/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = 3
...

[SFrijters]

I ran into this issue after updating to emacs 26: it uses glibc-2.27 but it doesn't install the matching glibc-locales. After manually installing that package and invoking emacs with LOCALE_ARCHIVE set to the matching path the error goes away.
Is this something that will be fixed on a general level (I'm not using NixOS by the way, just nixpkgs), or is this something that should be fixed by changing the emacs derivation?

[matthew-piziak]

I'm getting a bunch of these on rebuild:

/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)

And Intero is failing with

can't load .so/.DLL for: /nix/store/wdph2s4xzz4dkrw31cgby2l53892il0c-ghc-8.2.2/lib/ghc-8.2.2/ghc-prim-0.5.1.1/libHSghc-prim-0.5.1.1-ghc8.2.2.so (/nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libm.so.6: version `GLIBC_2.27' not found (required by /nix/store/wdph2s4xzz4dkrw31cgby2l53892il0c-ghc-8.2.2/lib/ghc-8.2.2/ghc-prim-0.5.1.1/libHSghc-prim-0.5.1.1-ghc8.2.2.so))

[grahamc]

I wonder if we should revert the change for now, given the pain this is causing non-NixOS deployments.

[utdemir]

I am using NixOS, and even for me this is painful.

I think the reason is that I use stable channel for my system, however I still use nixpkgs git repository instead of NIX_PATH when managing individual default.nix files I manage to have a consistent set of packages independent of the system.

It would be cool to have either of those:

• Revert the change
• Apply the change to NixOS stable
• Add both versions to nixpkgs and allow overriding it. I think this is a good option user-experience wise, but I don't know how feasible it is.

[7c6f434c]

Adding both versions is probably fine, building full package set with both glibc versions is a costlier proposition with less chances of acceptance. Without it the user experience would still be quite suboptimal.

On the other hand, having to have stable and master branches with the same glibc is also a bit unfortunate…

[grahamc]

Is it possible to fix glibc's bc-breaking change?

[peti]

I wonder if we should revert the change for now, given the pain this is causing non-NixOS deployments.

That feels like a slight over-reaction considering how trivial the fix is. It's just a matter of setting two environment variables -- possibly even only one -- and then everything works fine.

[matthew-piziak]

@peti Can you please point me to that trivial fix? I'm assuming the environment variable is LOCALE_ARCHIVE. To what should it be set?

[peti]

Configure your environment as follows:

• Set $LOCALE_ARCHIVE_2_27 to the path "${glibcLocales}/lib/locale/locale-archive". You can find out what glibcLocales is by running:

  $ nix-build --no-out-link "" -A glibcLocales
  /nix/store/m53mq2077pfxhqf37gdbj7fkkdc1c8hc-glibc-locales-2.27

• Set $LOCALE_ARCHIVE_2_11 to the path of your system's locale.

You can also set both variables to paths into the Nix store, if you prefer. See https://github.com/NixOS/nixpkgs/issues/38991#issuecomment-384238296 for an example.

[matthew-piziak]

Thank you!!

[michalrus]

Breaking Hubstaff, as well: https://github.com/NixOS/nixpkgs/issues/38988#issuecomment-404415795.

Can I somehow fix that derivation?

[domenkozar]

This seems to affect 18.09 due to glibc bump to 2.27

[vcunat]

AFAIK it only affects _mixing_ 18.09 packages/OS with 18.03 or older (or a different OS with older glibc version).

[vcunat]

Space for improvement I can see:

• [ ] documentation: entry in release notes, section "backward incompatibilities"
• [ ] (maybe) a nixos option to provide locale archives for both versions, possibly implemented _both_ in 18.09 and 18.03

Suggestions? PRs?

[timbertson]

Is there no plan to fix this in nixpkgs? The nixos pain seems temporary
while people upgrade, but many people will be running nixpkgs on an OS with
a different libc for a long time to come. Other OSes have their own, often
slower, upgrade path, and the beauty of nix is that it's supposed to allow
that kind of thing :(

On Thu., 6 Sep. 2018, 11:12 pm Vladimír Čunát, [e-mail]
wrote:

Space for improvement I can see:

• documentation: entry in release notes, section "backward
  incompatibilities"
• nixos option to provide locale archives for both versions

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/38991#issuecomment-419087642,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA3XCObb_AT3CbZ2v7GjkoPkg-7njo1ks5uYR8ugaJpZM4TWNpP
.

[vcunat]

I have no idea what kind of "fix in nixpkgs" you have in mind. Locale archives have been traditionally provided _impurely_ in nixpkgs. Yes, you could wrap everything with the archives for all languages, but most people probably wouldn't want that. AFAIK the only consequence of this issue is not having locales, which doesn't sound too critical if it happens. EDIT: I'm sorry if the comment sounds aggressive; feel free to suggest a fix (or even better, implement and test it).

[dckc]

I'm an occasional nix user, and I run into this issue every time I download new nar packages. One 5-line error message for each package. Mostly I just ignore it, but when I am feeling energetic, I look up the work-around from Oct 2016 in https://github.com/NixOS/nixpkgs/issues/8398#issuecomment-251287741 and apply it. It seems to work fine.

Help me understand why, two years later, nixpkgs (or: at least the part of the installer that calls perl, which then gripes) doesn't act like that work-around is always in place? Other work was considered higher priority? The work-around is a bad idea? The work-around would take a lot of work to integrate properly? Something else?

For me, the issue is not as broad as locale support in general. It's just the UX for installing / updating packages.

[bobvanderlinden]

@vcunat

AFAIK the only consequence of this issue is not having locales, which doesn't sound too critical if it happens

Unfortunately there are applications that will just not work. I ran into this issue because I wanted to get kitty from nixpkgs-unstable working on a system based on nixpkgs-stable. That resulted in an error. Apparently there are more python applications that will not be able to start due to this 'bug'/'annoyance'.

I'd think that people who have applications installed using nix-env (like the manual still prescribes) and they upgrade their system, some applications will break. In my case my graphical terminal would break until I update the channels of my local user.

You can see here that the same error appears for other applications as well (ktile, khard): https://github.com/NixOS/nixpkgs/issues/41784

What would be a good workaround for this? It seems glibcLocale upgrades is a recurring problem. Should applications statically link to glibcLocale? Or should we set LOCALE_ARCHIVE_2_27 and LOCALE_ARCHIVE_2_11 by default to the right versions of the glibcLocale package?

[vcunat]

The full locale archive-set has 109 MiB. That seems quite a hit to include it everywhere (in glibc closure).

[bobvanderlinden]

Alright, so I guess linking every derivation is out of the question. We could have an NixOS option that is on by default, which adds the compatibility environment variables. It's nasty, but breaking applications is also nasty I think.

We could also decide that this doesn't hit as many people and those people just need to upgrade their nix-env.

[timbertson]

Is it just about the closure size? Or are there other reasons to prefer system locales?

Currently on non-nixos systems, there's no good workaround. The options are:
(a) overriding individual packages which are particularly broken, or
(b) manually setting LOCALE_ARCHIVE_2_27 _somewhere_ in your system configuration

The second option is fragile - if you set it to the result of pkgs.globcLocales, that won't always be for version 2.27. It's also not obvious how to integrate this with nix (you can't reference the store path directly, you need to get the current path from nix-build or similar). I also don't even know of a good way to set this globally - if I set it in my shellrc, it won't take effect in applications launched from gnome, for example. If someone knows a general fix for non-nixos systems which isn't "upgrade glibc system wide", that would be appreciated.

But let's say I'm happy to pay whatever closure size is required. Would it be possible to implement a nixpkgs flag (in ~/.nixpkgs/config.nix) to do the wrapping in the glibc derivation or somewhere similar? Or is it still unknown how you'd actually implement that? Would it take out the option of binary caching, making me build everything myself?

[7c6f434c]

You could try config.glibc.installLocales = true but that means full rebuild. You could also install (nix-env -i or nix-build with -o) glibcLocales and put the resulting profile or symlink path into the variable. The variable is Nix-only anyway. I would hope that glibc locale format will stay the same longer than it would take the underlying distribution to catch up…

[bobvanderlinden]

Hmm, that might be an option for me and anyone reading this issue, but there will be people affected by the upgrade.

That said, I don't think people who are on non-nixos are affected, as they do not have 2 channels: one for root using nixos-rebuild , one for user using nix-env. When everything upgrades to the new glibc there shouldn't be a problem.

config.glibc.installLocales = true is a good suggestion. If that was on by default people wouldn't be affected by the upgrade. It does seem in line with Nix's promise to be reproducible (not rely on runtime for this case).

[vcunat]

Wrapping the most problematic packages might help in future, and the closure price might be OK if they have a large closure anyway. The hard-coding in glibc is possible, but (1) if you do it locally, you will essentially use no binaries from cache.nixos.org and have to rebuild everything, or (2) we might choose that everyone will always get it into the closure. (Having every package twice on Hydra because of this isn't realistic.)

[vcunat]

BTW, the size price can be cut to a fraction by choosing a small locale-set, e.g. only with en_US.UTF-8.

[timbertson]

FWIW adding this to an attrset in ~/.nixpkgs/config.nix worked: glibc = { locales = true; };
As mentioned though, you end up forking the world and so having to rebuild everything. Which yeah, it isn't very plausible to double the hydra builds just for this bug.

It _does_ seem like the perfect use case for https://github.com/NixOS/nixpkgs/pull/10851 or a similar system for substituting security updates though, so maybe one day...

[vcunat]

Yes, you can use such a scheme, e.g. just replace the glibc via pkgs.replaceDependency. You will end up copying your whole closure and search-replacing its contents, but that's surely way cheaper than recompiling.

[timbertson]

That still requires me to make a modified derivation for each program I care about though, unless I'm misunderstanding something - I can't just substitute the glibc derivation system-wide.

[vcunat]

Yes, it probably requires you to apply a function to every "final" derivation. One could hack a nix function that applies it the the whole nixpkgs tree, but I don't expect that to be simple to do very reliably.

[7c6f434c]

I wonder if using an overlay to replace stdenv.mkDerivation is a good idea.

[dezgeg]

I wonder if we could avoid most of the problems by compiling in locales for en_US.UTF-8 (and perhaps few related popular variants) to the glibc derivation?

[vcunat]

Just en_US.UTF-8 is only 1.7 MiB, so that approach really seems appealing to me over long term (glibc has 26 MiB), but:

• combining with env-var locale archives might be tricky, and
• this in principle can't help all those binaries that people already have on their systems.

[vandenoever]

I'm on a fresh 18.09 NixOS system with a 122M locale-archive for glibc-locales-2.27 and yet applications like Firefox and gkrellm claim that 'locale not supported by C library' even after setting LOCALE_ARCHIVE_2_27 in a bash session.

[vcunat]

@vandenoever: on 18.09 you have 2.27 locale. If you want to support older ones on there, you need _2_11 with glibc version e.g. from 18.03 (2.26).

[bfortz]

Is it just about the closure size? Or are there other reasons to prefer system locales?
Currently on non-nixos systems, there's no good workaround. The options are:
(a) overriding individual packages which are particularly broken, or
(b) manually setting LOCALE_ARCHIVE_2_27 _somewhere_ in your system configuration

I have been scratching my head on this all day. I have nothing against setting an env variable (but it is not clear where to do it properly, and it makes things ugly for non-specialist user), but this is not the only problem: you are not even sure the right version of glibc-locale is even installed in nix (if you are on a non-nixos box). IMO, this goes against the philosophy of nix where dependencies should be automatically fetched. I know the wrapper solution is not ideal because it increases the closure size, but at least it is safe and works out-of-the-box without users having to manually tweak their environment. Otherwise, if e.g. I want to use perl, I must manually install glibc-locale and set the env var. Too much trouble. I agree with @timbertson that I would prefer a larger closure size to all this mess.

[bfortz]

And as a side note, while upgrading mixxx (#a847dcc), I also messed up my database (happy I had a backup...) because of the mismatch between nixos-18.03 and master. Again, the wrapper was the cleanest way to solve the problem.

[coretemp]

Shouldn't this have been a release critical error?

Module `/etc/nixos/configuration.nix' has an unsupported attribute `boot'. This is caused by assignments to the top-level attributes `config' or `options'.

when I set config.glibc.installLocales = true;

When you suggest a work around can you please be specific as to where you apply the workaround?

[coretemp]

Why are two different versions of glibc in the output below on NixOS-18.09?

root # ldd `which htop`
        linux-vdso.so.1 (0x00007ffde8ff5000)
        libncursesw.so.5 => /nix/store/pk8fj5jcwvl6mb7rm34yhhga1vagh15v-ncurses-5.9/lib/libncursesw.so.5 (0x00007f96923cd000)
        libm.so.6 => /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/libm.so.6 (0x00007f96920cc000)
        libc.so.6 => /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/libc.so.6 (0x00007f9691d1f000)
        /nix/store/ikc9iziqc2rldacnbb2cdh7bdc1b2c3n-glibc-2.19/lib/ld-linux-x86-64.so.2 => /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f969262f000)

[coretemp]

I figured it out myself:

The solution is to remove old programs that use older glibc versions that have been installed via nix-env from your profile (via nix-env -e).

I do think it's a bug that previously installed programs break regardless of what the glibc maintainers did or did not do.

[peti]

I would like to close this chapter, er, bug. I realize some people still have open questions about the stae of affairs, but from a Nixpkgs development POV there's nothing left to do. A grand sweeping solution that solves all problems does clearly not exist and even if it did then I am unaware of anyone actively pursuing it. Therefore, I don't consider this problem an "open issue" that's actively being worked on and am happy to lay it at rest. Obviously, this ticket remains open for people to comment and/or ask questions.

[justinwoo]

For those coming to this thread later:

One workaround for non-NixOS is described here: https://github.com/NixOS/nixpkgs/issues/38991#issuecomment-400657551 (and a gist here: https://gist.github.com/peti/2c818d6cb49b0b0f2fd7c300f8386bc3)

So if you want to fix your Travis CI setup:

sudo: required
dist: trusty
language: nix
script:
  - glibcLocales=$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)
  - echo $glibcLocales
  - export LOCALE_ARCHIVE_2_27="${glibcLocales}/lib/locale/locale-archive"
  - # other things you do in your script

[tbenst]

For ubuntu, I added the following to my .bashrc:

export LOCALE_ARCHIVE_2_11="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"
export LOCALE_ARCHIVE_2_27="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"
export LOCALE_ARCHIVE="/usr/bin/locale"

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/conflict-between-glibc-and-glibclocales-workaround-inside/7608/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/bad-locale-support-perl-warning-setting-locale-failed/8261/1