Dietpi: perl: warning: Setting locale failed | DietPi 6 | Odroid-C2

[JasonSwindle]

Creating a bug report/issue:

Required Information:

• DietPi Version | cat /DietPi/dietpi/.version?

6
0

• SBC Device (EG: RPi 3)?
  Odroid-C2

• Power supply used (EG: 5v 1A RAVpower)?
  PowerPort+ 1 Port

• SD card used (EG: Sandisk ultra)?
  SanDisk Ultra 16GB Ultra Micro SDHC UHS-I/Class 10

• Distro (EG: Jessie) | uname -a?
  Debian Jessie | Linux golem 3.14.79+ #1 SMP PREEMPT Sun Jul 23 00:04:44 CEST 2017 aarch64 GNU/Linux

Expected behaviour:

I don't get a warning about LC_ALL

Actual behaviour:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "0:0",
    LC_ALL = "0",
    LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Steps to reproduce:

apt install <something>

Did you submit a dietpi-bugreport?

No

[k-plan]

@Fourdee

Checked on RPi Zero with Dietpi 6.0:

Generating locales (this might take a while)...
  0.UTF-8...cannot open locale definition file `0': No such file or directory
 done
  en_GB.UTF-8... done
  de_DE.UTF-8... done
Generation complete.

 DietPi-Set_Software
─────────────────────────────────────────────────────
 Mode: locale (en_GB.UTF-8)

[    ..] Running command: dpkg-reconfigure -f noninteractive locales

Yes something goes wrong after first boot (with dietpi.txt defaults) => auto update to 6.0 => inital setup, while first run Running command: dpkg-reconfigure -f noninteractive locales during setup a different NTP Server.

[Fourdee]

Test 1 RPi Stretch:

• Fresh image, defaults
• Go through all of 1st run setup, no changes. Reboot
• dietpi-config, add de.DE-UTF8 , set as default. reboot
  -- No errors
• 🈯️ apt-get install mc

root@DietPi:~# apt-get install mc
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig

Test 2 RPi Stretch:

• Fresh image, defaults
• During 1st run of dietpi-software, use dietpi-config, add de.DE-UTF8 , set as default.
• Continue with dietpi-software default install, reboot.
  -- No errors
• 🈯️ apt-get install mc

Test 3 Odroid C2 Jessie:

• Fresh image, defaults
• During 1st run of dietpi-software, use dietpi-config, add de.DE-UTF8 , set as default.
• Continue with dietpi-software default install, reboot.
  -- No errors
• 🈯️ apt-get install mc

[Fourdee]

@JasonSwindle
@k-plan

I'am unable to replicate in local testing, we'll need more info:
https://github.com/Fourdee/DietPi/issues/1430#issuecomment-362306553

Please can you provide the exact steps you completed (after image write), to cause the 0.UTF-8 error?

[k-plan]

@Fourdee

RPi Zero with RPi Stretch (DietPi v6.0, no auto update to v6.1 because not available at this time):

• During 1st run of dietpi-software, use dietpi-config, add de.DE-UTF8 , set en_GB.UTF-8 as default.
• change NTP server to de.pool.ntp.org
• HitOKand watch messages in terminal:
• after now automated running G_AGI ( apt-get update ,apt-upgrade , apt-get install .... )

....

Generating locales (this might take a while)...
  0.UTF-8...cannot open locale definition file `0': No such file or directory
 done
  en_GB.UTF-8... done
  de_DE.UTF-8... done
Generation complete.

 DietPi-Set_Software
─────────────────────────────────────────────────────
 Mode: locale (en_GB.UTF-8)

[    ..] Running command: dpkg-reconfigure -f noninteractive locales

...

But I have later to check again.

Edit:

• before dietpi banner appears on SSH terminal you will shortly see:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
-bash: warning: setlocale: LC_ALL: cannot change locale (0)
-bash: warning: setlocale: LC_ALL: cannot change locale (0)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (0)
[  OK  ] Root access verified.

[Fourdee]

@k-plan

Legend, thank you, i'll try to replicate tomorrow 👍

[MichaIng]

I am not sure if it is related, but as far as I can follow the code that late hour, if en_GB is chosen for default, then it is the only one available. dietpi-set_software locale takes the chosen default language and adds en_GB.UFT-8 to be the only generated ones. We really need to change this behaviour and leave all user selected languages in place, just adding en_GB if missing. I review possibilities also to just generate a single "additional" language after dpkg --configure locales.

[JasonSwindle]

Sorry about the delay, here are the steps I did from my bash histroy

Steps to reproduce:

1. Install DietPi (Odroid C2)
2. Leave LANG at en_GB.UTF-8 UTF-8
3. Install Docker
   i.
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 | grep [email protected] || exit 1
   ii.
   bash # cat /etc/apt/sources.list.d/docker.list deb [arch=arm64] https://download.docker.com/linux/debian jessie stable
4. apt-get update
5. Get error

I also found when I installed knowit ( pip install knowit ) I get error:

Traceback (most recent call last):
  File "/home/dietpi/docker-compose/env/bin/knowit", line 11, in <module>
    sys.exit(main())
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/__main__.py", line 129, in main
    knowit(videopath, options, context)
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/__main__.py", line 83, in knowit
    info = api.know(video_path, context)
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/api.py", line 49, in know
    return provider.describe(video_path, context)
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/providers/mediainfo.py", line 330, in describe
    data = self.executor.extract_info(video_path).to_data()
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/providers/mediainfo.py", line 89, in extract_info
    xml = self._execute(filename)
  File "/home/dietpi/docker-compose/env/local/lib/python2.7/site-packages/knowit/providers/mediainfo.py", line 165, in _execute
    locale.setlocale(locale.LC_CTYPE, locale.getdefaultlocale())
  File "/home/dietpi/docker-compose/env/lib/python2.7/locale.py", line 543, in getdefaultlocale
    return _parse_localename(localename)
  File "/home/dietpi/docker-compose/env/lib/python2.7/locale.py", line 475, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: 0

and I am seeing Perl errors like this:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "0:0",
    LC_ALL = "0",
    LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (0)

[JasonSwindle]

I also see, for a flash, on logging in:

-bash: warning: setlocale: LC_ALL: cannot change locale (0)
-bash: warning: setlocale: LC_ALL: cannot change locale (0)
-bash: warning: setlocale: LC_ALL: cannot change locale (0)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (0)
[  OK  ] Root access verified.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (0)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (0)

[Fourdee]

🈯️ C2 fresh install test, as per @JasonSwindle (https://github.com/Fourdee/DietPi/issues/1430#issuecomment-363175613)

root@DietPi:~#  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
root@DietPi:~#       sudo apt-key fingerprint 0EBFCD88 | grep [email protected] || exit 1
uid                  Docker Release (CE deb) <[email protected]>
root@DietPi:~# cat /etc/apt/sources.list.d/docker.list
cat: /etc/apt/sources.list.d/docker.list: No such file or directory
root@DietPi:~# apt-get update
............
Fetched 1,287 kB in 7s (165 kB/s)
Reading package lists... Done

---

RPi Zero @k-plan test (https://github.com/Fourdee/DietPi/issues/1430#issuecomment-362325454):

• Outside scope of test: auto updated to v6.1

~🈯️ Unable to replicate, test fine.~
mmm locale (0) being set after NTP mirror change, need to confirm.
🈴 Confirmed 0 after reboot.

Yay! Now we can investigate 😄

Test:

• Gen locales +de_DE +en_GB (default)
• Missing root@DietPi:~# cat /etc/locale.gen | grep -v '#' en_GB.UTF-8 UTF-8

---

Notes, the previous solution resolved this (prior to v6.0 changes):
https://github.com/Fourdee/DietPi/issues/825#issuecomment-288093527

---

• Is this a random issue with automated locale generation? Failing on some installs?
• Is PSU/SD card instability a causing factor during locale generation?

[MichaIng]

@Fourdee
Did you check for errors/invalid entries inside /etc/locale.gen and/or /etc/profile.d/99-dietpi-force-locale.sh? At least on @k-plan's test, the error occurred already on interactive locale generation, not our non-interactive, so there seem to be something wrong already before.

I was reading a bid deeper and had a look into our scripts, where and how we touch locales:

• https://wiki.debian.org/Locale: First of all, the LANG variable appears to be a fallback, not overwriting existing other locale settings, e.g. LC_NUMERIC.
• I played around a bid, and indeed, if e.g. LC_ALL is exported outside of script (due to our /etc/profile.d/99-dietpi-force-locale.sh), executing locale inside script shows that all LC_* values are set to the defined LC_ALL value, not the LANG value from script. Running e.g. ifconfig inside script shows, that setting LANG=en_GB.UTF-8 is without effect. We need to switch to LC_ALL if we wan't to prevent parsing errors.
• The reason for adding /etc/profile.d/99-dietpi-force-locale.sh was actually, that LANG was without effect and made whiptail look ugly: https://github.com/Fourdee/DietPi/issues/1367. With this LANG still is without effect, but /etc/profile.d/99-dietpi-force-locale.sh with all by us tested languages made whiptail look normal. Adding POSIX there verifies that.
• Another thing is, that without exporting that value, all subscripts will use system default again. At the moment /etc/profile.d/99-dietpi-force-locale.sh exports them and (just) adjusting them, even without export inside the script, makes the adjusted values still available within subscripts. But actually this is not what we necessarily wanted to.
• What we actually need, is export LC_ALL=en_GB.UTF-8 in all our scripts instead of for whole terminal/session. This correctly forces it for all kind of output, especially also for sub functions/scripts. Can be verified by calling locales and e.g. ifconfig within any script.
• Putting them into globals bears the problem, that they GB will be also exported for terminal. I don't know if there is a more elegant way, to make globals check, if it was called from within any of our scripts (thus should export LC_ALL) or directly from terminal/login script etc (where it should not export it), but I just used the new $HIERARCHY variable. It then needs to be created before calling globals and then within all our scripts, where parsing errors could occur.
• In the end we then don't need /etc/profile.d/99-dietpi-force-locale.sh anymore.

---

Then just to prevent possible errors we should adjust some other stuff:

• https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-software#L13846-L13861
• https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-config#L2274-L2296
• Instead of just checking for empty string here, we should better check, if this locale is actually available for generation on the system: cat /usr/share/i18n/SUPPORTED | grep '$locale_string'
• But actually I would, besides first run setup, remove all our own reading/writing stuff. Let user decide which languages he wan't to have available, which he wan't to be default etc. just check for grep -q '^[[:blank:]]*en_GB.UTF-8 UTF-8' /etc/locale.gen and otherwise add this string and just regenerate (not changing anything about defaults): locale-gen Sadly on Debian there is no way to just generate a single additional language.

---

I hope as we then do not touching any locale variables with not 100% known string (user input), this errors will be history.

[Fourdee]

But actually I would, besides first run setup, remove all our own reading/writing stuff. Let user decide which languages he wan't to have available, which he wan't to be default etc. just check for grep -q '^[[:blank:]]*en_GB.UTF-8 UTF-8' /etc/locale.gen and otherwise add this string and just regenerate (not changing anything about defaults): locale-gen Sadly on Debian there is no way to just generate a single additional language.

What we actually need, is export LC_ALL=en_GB.UTF-8 in all our scripts instead of for whole terminal/session. This correctly forces it for all kind of output, especially also for sub functions/scripts. Can be verified by calling locales and e.g. ifconfig within any script.

Agree 👍

The only reasons for our locale scripts are for first run, and, to ensure en_GB is regenerated if removed.

Ok, lets clean this up + full reset of locale during patch:

• Revert dietpi-config to interactive dpkg-reconfigure locales only. Check for grep -ci -m1 '^en_GB.UTF-8' /etc/locale.gen, prompt user to re-select if not found.
• Remove /etc/profile.d/99-dietpi-force-locale.sh
• LC_ALL=en_GB.UTF-8 in all our scripts (we shouldn't need to export it?)

---

test

rm /etc/profile.d/99-dietpi-force-locale.sh
rm /etc/environment
rm /etc/default/locale #This was /locales in patch (wrong)

• Remove all LANG= LC_ALL= from ALL DietPi scripts
• dpkg-reconfigure locales
  -- de_DE default with en_GB still installed
• reboot

oroot@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

#####
root@DietPi:~# date
So 11. Feb 19:09:33 GMT 2018
root@DietPi:~# LC_ALL=en_GB.UTF-8 date
Sun 11 Feb 19:09:43 GMT 2018

[Fourdee]

We dont need export for LC_ALL, and, it only needs to be in dietpi-globals:
LC_ALL=en_GB.UTF-8 in dietpi-globals is all that is required.

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
root@DietPi:~# dietpi-config
[  OK  ] Root access verified.
[ INFO ] DietPi-Drive_Manager | Checking RootFS R/W access. Please wait...
[  OK  ] DietPi-Drive_Manager | RootFS R/W access.

LC_ALL en_GB.UTF-8
LANG de_DE.UTF-8
So 11. Feb 20:06:12 GMT 2018


root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

🈯️ I believe we still need to define LANG= in scripts, else scrapes will be incorrect:

root@DietPi:~# dietpi-config
[  OK  ] Root access verified.
[ INFO ] DietPi-Drive_Manager | Checking RootFS R/W access. Please wait...
[  OK  ] DietPi-Drive_Manager | RootFS R/W access.

LC_ALL en_GB.UTF-8
LANG de_DE.UTF-8
So 11. Feb 20:06:12 GMT 2018

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
root@DietPi:~# dietpi-config
[  OK  ] Root access verified.
[ INFO ] DietPi-Drive_Manager | Checking RootFS R/W access. Please wait...
[  OK  ] DietPi-Drive_Manager | RootFS R/W access.

LC_ALL en_GB.UTF-8
LANG en_GB.UTF-8
Sun 11 Feb 20:09:59 GMT 2018
root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

[Fourdee]

Whiptail is fine with new system on OpenSSH, need to test dropbear.

• 🈴 Dropbear POSIX, we still need a solution for this.

What we actually need, is export LC_ALL=en_GB.UTF-8 in all our scripts instead of for whole terminal/session. This correctly forces it for all kind of output, especially also for sub functions/scripts. Can be verified by calling locales and e.g. ifconfig within any script.

Yep, this will fix dropbear, but override issue.

As this is a confirmed bug/issue with dropbear, unless we can find a viable solution, we should mark is as non-dietpi issue?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=533525

[MichaIng]

@Fourdee
How did you test the correct "localed" output, when calling scripts from within our scripts? During my tests it was necessary to export LC_ALL within globals. Somehow this also makes sense, as exporting is explicitly the way to give variables to subscripts?

LC_ALL=en_GB.UTF-8 date of course works, as this is another way to explicitly give variables to subscript.
But, as long as LC_ALL was not exported anywhere (/etc/profile.d/99-dietpi-force-locale.sh removed, unset LC_ALL LANG LANGUAGE):

root@DietPi:~# LC_ALL=de_DE.UTF-8
root@DietPi:~# date
Sun Feb 11 20:52:11 GMT 2018

As can be seen beneath, LANG is indeed just fallback, if the separate LC_... are not set explicitly. LC_ALL overwrites everything.

root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
...
LC_IDENTIFICATION="POSIX"
LC_ALL=
root@DietPi:~# export LANG=en_GB.UTF-8
root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
...
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
root@DietPi:~# export LC_TIME=C
root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
...
LC_TIME=C
...
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
root@DietPi:~# export LC_ALL=de_DE.UTF-8
root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
...
LC_TIME="de_DE.UTF-8"
...
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8
root@DietPi:~# unset LANG
root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
...
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8

The issue with whiptail on dropbear seems for me just a prove, that locales are not changed without exporting. Doesn't matter if done within terminal/profile or later in our script, but needs to be exported. Then for our scripts, dropbear is no issue anymore.

If we really export LC_ALL just inside the scripts, or within globals just, if we can verify (e.g. $HIERARCHY) that it was not sourced from terminal/.bashrc/profile, then users will still have POSIX locales within terminal. This is not a DietPi issue (and does then not affect our script results), but we could still fix it for users by using /etc/profile.d/99-dietpi-force-locale.sh, but just export LANG there exactly as it was set in /etc/default/locale by interactive dpkg-reconfigure locales.

---

About the actual topic of this thread:

• I could verify that on 1st run dietpi-software, directly changing ntpd mirror results in finally locale error.
• It has to do something with the fact, that changing ntpd mirror is calling dietpi-software (again) to install NTP, which then, as it is has still 1st run flag, results in all the other 1st run adjustments. I didn't check further, but there something seems to break the locales then.
• I faced the error btw. without changing anything about the locales first. Just 1st run dietpi-software > dietpi-config > adjust ntpd mirror > error (can be already seen there for short) > reboot > still error.
• Reconfiguring via dietpi-config solves it then.

[Fourdee]

@MichaIng

• After v6.2 patch
• Set locale to de_DE.UTF8, and default
• Remove LANG= entry from dietpi-globals
• in dietpi-config change following:

    #Import DietPi-Globals ---------------------------------------------------------------
    . /DietPi/dietpi/func/dietpi-globals
    G_CHECK_ROOT_USER
    G_CHECK_ROOTFS_RW
    export G_PROGRAM_NAME='DietPi-Config'
    G_INIT_HIERARCHY
date
echo $LANG
echo $LC_ALL
exit
    #Import DietPi-Globals ---------------------------------------------------------------

• run, check output
• add LANG= back into dietpi-globals
• run, check output

You'll get same results as last test on this page( https://github.com/Fourdee/DietPi/issues/1430#issuecomment-364779497)

Essentially, all programs run inside DietPi scripts require LANG, else scrape of (date in this case) will be in de_DE (not what we want).

[Fourdee]

Note to self:
G_WHIP_MSG needs work.

[  OK  ] DietPi-Set_software | dpkg-reconfigure -f noninteractive locales
[  OK  ] locale en_GB.UTF-8 | Completed
[FAILED] DietPi-Patchfile | Notice (locale):

[MichaIng]

@Fourdee
On my Stretch VM the notice worked just fine. I am just not too sure, if it makes sense that way, as we adjust the dietpi.txt setting on dietpi-config locale adjustment, at least if error does not completely break the script 🤔. We need to check, if the value is actually a valid locale (as mentioned above) and otherwise reset to en_GB.UTF-8, or what you think?

And now I got it why it works for you without exporting 😄:

• If you chose a default (instead of None), then the resulting entry from /etc/default/locale seems to be exported. Thus we can adjust it in our scripts and it will be given forward to subscripts. This does not work for LC_ALL now.
• But try to set None as default, then subscripts will not take our adjusted LANG value. Btw: Without default, POSIX is used here, interestingly on direct VM terminal, whiptails still looks nice, just on SSH terminal it is ugly 🤔.
• Worse: If you export LC_TIME=de_DE.UTF-8 in terminal, or add it to /etc/default/locale, which can be also done via update-locale LC_TIME=de_DE.UTF-8 + reboot, then our LANG=en_GB.UTF-8 will be overwritten for that type, thus date will be parsed wrong.

[Fourdee]

@MichaIng

I am just not too sure, if it makes sense that way, as we adjust the dietpi.txt setting on dietpi-config locale adjustment,

In case of end user has used dietpi-automation, we'll set that, else the default (en_GB).
Validation would be better, but I dont have the time at the moment to implement, and v6.2 needs to be out this week.

Sending commit, where we init locale during script start (outside of globals). Not as "clean" as i'd like, but, I see this is the only solution.
All global functions add locale init aswell. Not needed for majority, but add all regardless in case of future changes that may require it.

[Fourdee]

cat << _EOF_ > test
#!/bin/bash
echo -e "current"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

. /DietPi/dietpi/func/dietpi-globals
echo -e "current - post globals"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

G_INIT
echo -e "current - post G_INIT"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

_EOF_
chmod +x test
./test

🈯️ Result:

root@DietPi:~# ./test
current
Mo 12. Feb 20:23:53 GMT 2018
de_DE.UTF-8


current - post globals
Mo 12. Feb 20:23:53 GMT 2018
de_DE.UTF-8


current - post G_INIT_LOCALE
Mon 12 Feb 20:23:53 GMT 2018
en_GB.UTF-8
en_GB.UTF-8

[theAkito]

How does one apply this fix now?

[MichaIng]

@Akito13
At first, could you verify my suggestion and post output of cat /etc/profile.d/99-dietpi-force-locale.sh here, please?
I guess during 1st run setup (dietpi-software>dietpi-config), something I couldn't find yet, breaks the entries in this file.

To fix now:
If you don't use Dropbear to access via SSH, then it should be enough to:
rm /etc/profile.d/99-dietpi-force-locale.sh and reboot, or alternatively choose you locales again via dietpi-config. We will remove the file during next patch this week and deal with things differently.

If you access via Dropbear, then you need to export LANG=<whatever you want> somewhere, e.g. within /etc/profile.d/99-dietpi-force-locale.sh.

@Fourdee
I am not yet satisfied with current solution 😛:

• On Dropbear, whiptail is still broken, as /etc/default/locale seems to be ignored there, thus our scripts do not forward it to subscripts. At least for our scripts it seems not important, as POSIX seems to be mostly the same than en_GB.UTF-8.
• Everywhere else also, it can be even worse:

root@DietPi:~# cat /DietPi/dietpi/dietpi-config
...
        #Import DietPi-Globals ---------------------------------------------------------------
        . /DietPi/dietpi/func/dietpi-globals
        G_CHECK_ROOT_USER
        G_CHECK_ROOTFS_RW
        export G_PROGRAM_NAME='DietPi-Config'
        G_INIT_HIERARCHY
        G_INIT_LOCALE
        #Import DietPi-Globals ---------------------------------------------------------------
        date
        exit
...
root@DietPi:~# update-locale LC_TIME=de_DE.UTF-8
root@DietPi:~# cat /etc/default/locale
#  File generated by update-locale
LANG=en_GB.UTF-8
LC_TIME=de_DE.UTF-8
root@DietPi:~# dietpi-config
[  OK  ] Root access verified.
[ INFO ] DietPi-Drive_Manager | Checking RootFS R/W access. Please wait...
[  OK  ] DietPi-Drive_Manager | RootFS R/W access.

Di 13. Feb 10:02:14 GMT 2018

• Also here again a prove, that LC_ALL will be ignored/not forwarded to subscripts, if we don't export.
• So my suggestion: As the INIT functions within globals do future adjustments easier: just export LC_ALL=en_GB.UTF-8 there within INIT locales. It has btw. no influence on locales in terminal and fixes Dropbear issue as well:

root@DietPi:~# cat /DietPi/dietpi/func/dietpi-globals
...
        export LC_ALL=en_GB.UTF-8
...
root@DietPi:~# dietpi-config
[  OK  ] Root access verified.
[ INFO ] DietPi-Drive_Manager | Checking RootFS R/W access. Please wait...
[  OK  ] DietPi-Drive_Manager | RootFS R/W access.

Tue 13 Feb 10:09:33 GMT 2018
root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME=de_DE.UTF-8
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

• Then to clean up, we could merge this INIT functions actually, as both do not hurt or are necessary in all our scripts. Maybe even put G_PROGRAM_NAME as parameter inside 🙂.

[Fourdee]

@MichaIng

I am not yet satisfied with current solution

Good, me either 😆

Also here again a prove, that LC_ALL will be ignored/not forwarded to subscripts, if we don't export.
So my suggestion: As the INIT functions within globals do future adjustments easier: just export LC_ALL=en_GB.UTF-8 there within INIT locales. It has btw. no influence on locales in terminal and fixes Dropbear issue as well:

👍

test

cat << _EOF_ > test
#!/bin/bash
echo -e "current"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

. /DietPi/dietpi/func/dietpi-globals
echo -e "current - post globals"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

G_INIT
echo -e "current - post G_INIT"
date
echo \$LANG
echo \$LC_ALL
echo \$LC_TIME

_EOF_
chmod +x test
./test
locale

🈯️

root@DietPi:~# ./test
current
Di 13. Feb 14:57:12 GMT 2018
de_DE.UTF-8


current - post globals
Di 13. Feb 14:57:12 GMT 2018
de_DE.UTF-8


current - post G_INIT
Tue 13 Feb 14:57:12 GMT 2018
en_GB.UTF-8
en_GB.UTF-8

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

Test 2:
🈯️ same result

update-locale LC_TIME=de_DE.UTF-8
#rerun test above

---

🈴 We cant use G_INIT inside any function for globals, as it overrides current bash session locale. So expect non en bug reports where G_ commands used from local bash session.

G_AGI mooo
Log file contents:
E: Unable to locate package mooo

root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

[Fourdee]

@MichaIng

Thanks for the solutions on this one, appreciate it. Haven't been myself last few days, apologies for that.

I believe this now resolves all outstanding Global/Locale issues?

[MichaIng]

@Fourdee
Jep, variables inside functions are set for the environment, where they are called. Thus export them, as well as just set them, will do this for terminal. But just setting them will have not always some effect, because subscripts (date) just use exported ones (or the ones, set in /etc/default/locale.

So as we source globals and expect users to run global functions, we cannot set or export any locale variable there. Just unsetting them at end of function/globals does not work either, because then we unset it for whole environment, forcing reset to POSIX.

I was experimenting with local export LC_ALL and export local LC_ALL within global functions, but it seems just the first declaration is used: local export LC_ALL does not work with date as test, and export local LC_ALL does export the variable to terminal again...
This combination, export variable for subscript, but handle it still local, just valid for inside the G_function would be nice, but didn't find any way to achieve this.

But it should be not too hard, to take care of this "just" within dietpi-globals. As long as we don't rely on the correct English output of any 3rd party commands there, that create possibly translated output, we stay save.

---

Just had a look:

• I found du, ps and whiptail where just the last one is within Dropbear affected by locales.
• G_WHIP_MSG test looks ugly there, if called directly within SSH terminal. What solves it, is LC_ALL=en_GB.UTF-8 whiptail --... to set locale just for subscript within G_WHIP_MSG.

[Fourdee]

@MichaIng

Best we can do now is define a local var:

G_AGI(){
        local LANG=en_GB.UTF-8
        local LC_ALL=en_GB.UTF-8
...

---

Log file contents:
E: Unable to locate package mightymoocow

[FAILED] Unable to continue, the program will now terminate.

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

But still no effect on update-locale LC_TIME=de_DE.UTF-8 and dropbear I believe.

---

Tried this, but obviously, only local to G_INIT, we'd need to add manually local Lxxx to each function in globals:

    G_INIT(){

        # - Originating script only
        if [ -n "$G_PROGRAM_NAME" ]; then

            # - Set locale for scripts, prevents incorrect scraping
            export LANG=en_GB.UTF-8
            export LC_ALL=en_GB.UTF-8

            # - HIERARCHY system for G_DIETPI-NOTIFY 3
            [ -n "$HIERARCHY" ] && export HIERARCHY=$((HIERARCHY+1)) || export HIERARCHY=0

            # - Auto print init header for all scripts
            # G_DIETPI-NOTIFY 3 "$G_PROGRAM_NAME" 'Init'

        # - Functions within this script
        else

            local LANG=en_GB.UTF-8
            local LC_ALL=en_GB.UTF-8

        fi

    }

[Fourdee]

@MichaIng
I think for now, leave current bash session with globals functions as is?

Having local Lxxx inside each global function isn't ideal, and, still wouldn't fix dropbear + update-locale LC_TIME=de_DE.UTF-8 I believe, when used with G_ commands in current bash session.

If we start receiving reports from users, with non-en_GB default locale, causing a negative effect by using G_ in current bash sessions with issues, we'll need to reassess.

I'll mark this as closed (for now). Please reopen if required.

[Fourdee]

mmm locale (0) being set after NTP mirror change, need to confirm.

Now unable to replicate.

[MichaIng]

@Fourdee
Jep, works only, if LC_ALL is exported somewhere in terminal environment. This is actually the (maybe only) way to combine export and local: Export for terminal environment and then adjust as local within function. The local adjusted value is given to subscripts then, but does not affect terminal environment:
With just local LC_ALL=en_GB.UTF-8

root@DietPi:~# export LC_ALL=de_DE.UTF-8
root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8
root@DietPi:~# G_AGI test
[ INFO ] APT installation for: test, please wait...
E: Unable to locate package test

root@DietPi:~# unset LC_ALL
root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME=de_DE.UTF-8
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
root@DietPi:~# export LC_MESSAGES=de_DE.UTF-8
root@DietPi:~# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME=de_DE.UTF-8
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES=de_DE.UTF-8
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
root@DietPi:~# G_AGI test
[ INFO ] APT installation for: test, please wait...
E: Paket test kann nicht gefunden werden.

Okay so we have two possibilities:

1. Leave export LC_ALL within /etc/profile.d/ with user chosen default (/etc/default/locale) or en_GB.UTF-8 if no default was selected. But then the flexibility of setting different locales for different things get's lost.
2. Leave out locales in globals + global functions and just take care, that we not rely on specifically translated 3rd party commands there, or give directly LC_ALL=... command.
3. If we cannot be sure, that local LC_ALL has any effect, then I would also not set it at all, as this might give a sense of security that we actually can't give. Then better go with 2.

[Fourdee]

Bugs with patch, investigating.

[Fourdee]

Ok, found the correct method needed for automated setting of locales, implemented into dietpi-config.
Resolves ALL outstanding locale issues for ALL systems (patch tested also).

• en_GB automatically installed, if its not the default
• Only list UTF-8 in dietpi-config locale (thats all we support). Based on /usr/share/i18n/SUPPORTED @MichaIng legend 👍
• automation + dietpi-config use both /DietPi/dietpi/func/dietpi-set_software locale
• User selected option is applied as default locale to "common" LANG/LC_xxx

---

Running some final tests to verify.

root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

/DietPi/dietpi/func/dietpi-set_software locale de_DE.UTF-8

Re-login

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8

/DietPi/dietpi/func/dietpi-set_software locale en_GB.UTF-8

re-login

root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

🈯️

[Fourdee]

Patch test:

root@DietPi:~# locale
LANG=de_DE.UTF-8
LANGUAGE=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8

###

root@DietPi:~# G_DEV_1

Some general warnings are expected, as we cant activate new locales, until re-login. After which, everything will be fine:

root@DietPi:~# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=de_DE.UTF-8
LANGUAGE=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=de_DE.UTF-8

re-login:

root@DietPi:~# locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

fixed 🈯️
Many thanks @MichaIng for the notes 👍 Was finally able to do them justice :)

[Fourdee]

Completed.

[MichaIng]

@Fourdee
Nice one, jep this will definitely fix it for everybody 😆.
Still some hints:

• LANG does really not need to be set, if LC_ALL is exported.
• same as update-locale LC_ALL overwrites all other choices anyway, so no need to do it for anything else then.
• /etc/environment should not be removed, as I saw some raw warnings that is is missing. Just echo '' > /etc/environment instead and only once, so users can use it for other stuff. Maybe also do a backup of it to /mnt/dietpi_userdata/environment?
• The new menu indeed looks elegant. But it takes the change to use multiple locales for different things (also maybe chosen by 3rd party software via gui, who knows).

Sorry to make that suggestion after you had all the effort realizing the new handling 😅:

• I would stay with the interactive dpkg-reconfigure locales and let users choose what/how much locales they want and let them choose their default, if wanted.
• Afterwards, we check for $(cat /etc/locale.gen | grep '^[[:blank:]]*en_GB.UTF-8 UTF-8'), otherwise echo en_GB.UTF-8 UTF-8 >> /etc/locale.gen; locale-gen
• locale-gen will just regenerate the locales (make them available), but does not change something else about defaults etc. So we then can be sure, that en_GB.UTF-8 is available for our scripts, while users have full control/flexibility.

I can do PR about this 😉.

[Fourdee]

@MichaIng

No worries, all good points 👍

The new menu indeed looks elegant. But it takes the change to use multiple locales for different things (also maybe chosen by 3rd party software via gui, who knows).

We'll wait for user feedback after this release. This is how I've wanted it from the very start of dietpi-config (simple experience, as per the rest of DietPi. With minimal "please do this" prompts etc, eg: en_GB required now automatic).
Pending user feedback we can revert/change.

echo '' > /etc/environment instead and only once, so users can use it for other stuff. Maybe also do a backup of it to /mnt/dietpi_userdata/environment?

Good idea, i'll send commit 👍

update-locale LC_ALL overwrites all other choices anyway, so no need to do it for anything else then.

Yep, although unnecessary now, for the sake of "linux always changing stuff (evolving)", i'd prefer we define all items (that we know can effect locale) to prevent an issue occurring in the future.
From experience, its better to "do more than is needed" now and define everything that is required in the first instance.