Nixpkgs: Missing a lot of translation in plasma5-based system.

[cx405]

Issue description

I have the following in configuration.nix:

environment.systemPackages = with pkgs; [
...
kdeFrameworks.ki18n
qt5.qttranslations
];
...
services.xserver.displayManager.sddm.enable = true;
services.xserver.desktopManager.plasma5.enable = true;
i18n.supportedLocales = [ "en_US.UTF-8/UTF-8" "ru_RU.UTF-8/UTF-8" "de_DE.UTF-8/UTF-8" ];
i18n.defaultLocale = "ru_RU.UTF-8";

sddm is not localized, the log out/suspend/reboot/shutdown window is also not localized.
Many plasma5 programs use mixed English+Russian elements, like strings in menu "File, Edit".
In systemsettings5, header "All settings, Help, Configure and field [Search]" are not translated, but all elements are.
In systemsettings5 -> Localization, only English is listed, but this configuration element is correctly localized.

I have a feeling that I miss some package or setting.

Steps to reproduce

Technical details

(updated)

• system: "x86_64-linux"
• host os: Linux 4.15.15, NixOS, 18.03.131807.489a14add9a (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.0
• channels(root): "nixos-18.03.131807.489a14add9a, nixos-master-18.09pre133640.ea145b68a01"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[cx405]

sddm translation files are apparently in the system, but are not used:

/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/ar.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/ca.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/cs.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/da.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/de.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/es.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/et.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/fi.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/fr.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/hi_IN.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/hu.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/it.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/ja.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/kk.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/ko.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/lt.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/lv.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/nb.qm
/nix/store/qc3gvbczd9d6wm6j0ps6w3ww360n7wbv-sddm-0.17.0/share/sddm/translations/nl.qm
...

When logged into the GUI and tty as user or root:
$ echo $LANG
ru_RU.UTF-8

[cx405]

Pinging everyone, who contributed to sddm for awareness:
@ttuegel
@vcunat
@bennofs
@wkennington
@peterhoeg
@heydojo
@dezgeg
@rnhmjoj
@lethalman
@jokogr
@fpletz
@edolstra
@primeos
@aske
@abbradar

[jtojnar]

Not sure how QT handles translations but you can try adding something like the following to SSDM package expression:

  postFixup = ''
    wrapProgram "$out/bin/sddm" \
      --prefix XDG_DATA_DIRS : "$out/share"
  '';

You will also need makeWrapper in nativeBuildInputs.

[ttuegel]

sddm translation files are apparently in the system

Those files are present, but not installed. They should find their way into /run/current-system/sw/share to be effective.

[cx405]

@ttuegel Hello sir, this is what I have:

$ ls /run/current-system/sw/share/sddm/translations/
ar.qm  es.qm     hu.qm  lt.qm  pl.qm     sk.qm                 sv.qm
ca.qm  et.qm     it.qm  lv.qm  pt_BR.qm  [email protected]  tr.qm
cs.qm  fi.qm     ja.qm  nb.qm  pt_PT.qm  [email protected]       uk.qm
da.qm  fr.qm     kk.qm  nl.qm  ro.qm     [email protected]           zh_CN.qm
de.qm  hi_IN.qm  ko.qm  nn.qm  ru.qm     sr.qm                 zh_TW.qm

[ttuegel]

I will have to check this out, but I think the most likely issue is that systemd is interfering with the XDG_DATA_DIRS setting; systemd is very picky about service process environments.

[ghost]

33987

[lheckemann]

Many plasma5 programs use mixed English+Russian elements, like strings in menu "File, Edit".

I've dug into this a bit with the example of konsole.

It's trying to find the locale files for all the kde frameworks it
uses, which aren't necessarily in the system path. As a naive
workaround, I tried adding all the relevant packages (in
kdeApplications) to environment.systemPackages, but this didn't
help since some of them are multiple-output and the outputs installed
aren't the ones containing the locale information.

So essentially the solution seems to be to get all the different KDE frameworks' /share dirs into XDG_DATA_DIRS somehow. I'm not sure what the best approach for this is though — should they all just be in systemPackages or should there be a separate ~profile~ buildEnv containing all the translations?

Expression built up during testing

{ pkgs ? import <nixpkgs> {}
, lib ? pkgs.lib
, stdenv ? pkgs.stdenv
, kdeFrameworks ? pkgs.kdeFrameworks
, buildEnv ? pkgs.buildEnv
, writeScript ? pkgs.writeScript
, runCommandNoCC ? pkgs.runCommandNoCC
, vim ? pkgs.vim
, konsole ? pkgs.konsole
}:
let makeEnv = name: pkgs: buildEnv {
    inherit name;
    paths = pkgs;
    extraOutputsToInstall = ["out"];
    pathsToLink = ["/share/locale"];
  };
  test = env: writeScript "konsole-test-${env.name}" ''
    #!${stdenv.shell}
    TMP=$(mktemp -d --tmpdir locale-debugging-XXXXXXX)
    pushd "$TMP" >/dev/null
    export LOCALE=de_DE.UTF-8 \
           LC_ALL=de_DE.UTF-8 \
           XDG_DATA_DIRS="${env}/share:$XDG_DATA_DIRS"
    strace -e file -o >(grep LC_MESSAGES >log) konsole

    grep -o "LC_MESSAGES/.*\"" log | sort | uniq > attempted
    grep -v ENOENT log | grep -o "LC_MESSAGES/.*\"" | sort | uniq > found
    comm -23 attempted found
    popd >/dev/null; rm -r "$TMP"
  '';
in rec {
  frameworks = with kdeFrameworks; [ konsole kbookmarks kcodecs kcompletion kconfig kcoreaddons kdbusaddons kglobalaccel kitemviews kjobwidgets knotifications sonnet ki18n kxmlgui ];
  env-all = makeEnv "kde-locales-all" ([ konsole ] ++ (builtins.filter lib.isDerivation (builtins.attrValues kdeFrameworks)));
  env-some = makeEnv "kde-locales" frameworks;
  tests = rec {
    test-with-all-libs = test env-all;
    test-with-select-libs = test env-some;
    compare = writeScript "compare" ''
      #!${stdenv.shell}
      ${vim}/bin/vim -d <(${test-with-all-libs}) <(${test-with-select-libs})
    '';
  };
  tests-dir = runCommandNoCC "tests" {} ''
    mkdir -p $out
    cd $out
    ${lib.concatMapStringsSep "\n" (name: "ln -s ${tests.${name}} ${name}") (builtins.attrNames tests)}

  '';
}

[florianjacob]

I threw some strace at the problem as well, I can confirm the findings of @lheckemann that the partial translation comes from the applications having access to their own .mo files, but can't find the KDE Frameworks libraries' .mo files in /run/current-system/sw/share/locale.

Sadly, I couldn't find a viable solution to get those on path either. :disappointed:

As a concrete example, strace shows dolphin requires the translation files from kxmlgui, kconfigwidgets and kio, and adding their shared directories to the XDG_DATA_DIRS path is enough to get a fully translated dolphin:

env XDG_DATA_DIRS=$XDG_DATA_DIRS:/nix/store/glrflj9nghjh20700jaa656zlg2ib8sv-kxmlgui-5.49.0/share:/nix/store/hh9pca9r34xaqrlnd2zy2zn0vmxzi2fx-kconfigwidgets-5.49.0/share:/nix/store/4zidp8vdiy3yy07jbamk35d82rx6x5sl-kio-5.49.0/share dolphin

[lheckemann]

@florianjacob just putting them in systemPackages should do the trick, but I don't think it's the right approach.

[florianjacob]

@lheckemann does that still work for you? This did not do the trick for me, even after rebooting, am I missing something?

environment.systemPackages = [
…
  pkgs.kdeFrameworks.kio
  pkgs.kdeFrameworks.kxmlgui
  pkgs.kdeFrameworks.kconfigwidgets
…
];
# even this does not make the relevant files appear in /run/current-system/sw/share/locale
environment.pathsToLink = [ "/share/locale" ];

[lheckemann]

You also need environment.extraOutputsToInstall = ["out"]; since a lot of these frameworks have their locale stuff in the out output but bin as their default output.

[florianjacob]

oooh. @lheckemann out and bin both being outputs that are handled differently was the missing piece for me to understand the problem. Thank you!

Seems like it works to directly put the out outpu in system packages, avoiding to put the out of every other package on system path as well:

environment.systemPackages = with pkgs; [
   …
    # workaround to get localization
    # dolphin
    kdeFrameworks.kxmlgui.out
    kdeFrameworks.kio.out
    kdeFrameworks.kconfigwidgets.out
    # +konsole
    kdeFrameworks.ki18n.out
    # +kate
    kdeFrameworks.ktexteditor.out
]

Strangely, this even works without pathsToLink. :thinking:

I'm now in the process of finding all affected libraries, hoping that we can then bundle them together in a buildEnv and give them to each KDE application.

[michaelpj]

I think it works without pathsToLink because of https://github.com/NixOS/nixpkgs/issues/47173. You should probably include it, though.

[ttuegel]

You also need environment.extraOutputsToInstall = ["out"]; since a lot of these frameworks have their locale stuff in the out output but bin as their default output.

That seems like a bug; the localizations should probably go in bin.

[jtojnar]

That seems like a bug; the localizations should probably go in bin.

When package contains both library and a executables, I would expect the localizations to be in the library output (out). The other way around would cause a cyclic dependency.

[schmittlauch]

@florianjacob

but can't find the KDE Frameworks libraries' .mo files in /run/current-system/sw/share/locale.

Does that mean that only translation files for systemPackages are taken into account anyways? Because the *.mo files for user profile installed software goes to ~/.nix-profile/share/locale/ only.

[florianjacob]

@schmittlauch I'm not sure whether I understood what you're asking about, but the issue is not in systemPackages system profile vs user profile installed packages, but in the derivation outputs that are put in those profiles, i.e. the key thing is the systemPackages = [ kdeFrameworks.kxmlgui.out ] .out part, not the systemPackages.

[cx405]

This is still issue. This is my own personal opinion, but users expect software output within systemPackages to contain expected output, so ".out" as extra option definitely goes against that common paradigm. Maybe those packages should "out" by default instead?

[ttuegel]

Maybe those packages should "out" by default instead?

If including the localizations in .bin would create a cyclic dependency, then there is no reason to have distinct .bin and .out outputs in the first place. So: yes.

[florianjacob]

Could it be that this has actually somehow, miraculously, been fixed with 19.09? :open_mouth: Can't find things that aren't translated anymore.

[worldofpeace]

@florianjacob I believe when the qt wrappers were merged @ttuegel cleaned-up/simplified the multiple outputs.

[ttuegel]

Could it be that this has actually somehow, miraculously, been fixed with 19.09? open_mouth Can't find things that aren't translated anymore.

I'm thrilled to hear that! :grinning:

I will go ahead and close this issue. There may still be isolated translation problems, but we can deal with them individually.