Singularity: /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

[kforner]

Hi,

The wrapper that starts my app tries to perform some setlocale commands, and I get these warnings:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Warning message:
In Sys.setlocale("LC_COLLATE", "en_US.UTF-8") :
OS reports request to set locale to "en_US.UTF-8" cannot be honored
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_COLLATE failed, using "C"
3: Setting LC_TIME failed, using "C"
4: Setting LC_MESSAGES failed, using "C"
5: Setting LC_MONETARY failed, using "C"
6: Setting LC_PAPER failed, using "C"
7: Setting LC_MEASUREMENT failed, using "C"

I tried to add the following files (from a strace log):
/usr/share/locale/locale.alias
/usr/lib/locale/locale-archive
/usr/share/locale-langpack/en/LC_MESSAGES

But I still can'get rid of this error:
In Sys.setlocale("LC_COLLATE", "en_US.UTF-8") :
OS reports request to set locale to "en_US.UTF-8" cannot be honored

[gmkurtzer]

Did you do the strace from within singularity (e.g. singularity strace [container name]), or on the program outside of the container? It might be interesting to compare the two. I bet it is just missing a locale (or few), or a config.

What operating system are you running this on?

I will try to debug later today (if I have a chance) or early next week.

[jmstover]

What about adding in the following (is there an easy way to add in a
directory)?

/usr/share/X11/locale/en_US.UTF-8/*
/usr/share/i18n/locales/en_US
/usr/share/locale/en_US/*

And, is the UTF8 locale required? I usually kill UTF8 as I run into many...
many, many programs I run just can't handle the UTF-8 character set so you
get messed up characters displayed that it can't display correctly (I'm
looking at you "man"!).

-J

On Fri, Mar 11, 2016 at 11:38 AM, Gregory M. Kurtzer <
[email protected]> wrote:

Did you do the strace from within singularity (e.g. singularity strace
[container name]), or on the program outside of the container? It might be
interesting to compare the two. I bet it is just missing a locale (or few),
or a config.

What operating system are you running this on?

I will try to debug later today (if I have a chance) or early next week.

—
Reply to this email directly or view it on GitHub
https://github.com/gmkurtzer/singularity/issues/11#issuecomment-195471763
.

[kforner]

@jmstover
I tried adding:

/usr/share/X11/locale/en_US.UTF-8/
/usr/share/i18n/locales/en_US
/usr/share/locale/en/

with no success

And, is the UTF8 locale required? I usually kill UTF8 as I run into many... many, many programs I run just can't handle the UTF-8 character set so you get messed up characters displayed that it can't display correctly (I'm looking at you "man"!).

The point is not if it is required, but to try to wrap using singularity existing apps. This app uses UTF8.

[gmkurtzer]

Is this happening on all containers you have made, or just the R container?

What distribution are you building on?

How can I try and replicate the problem?

What is your environment usually configured to with respect to LC*, LANG, C, etc...?

Thanks!

[kforner]

Is this happening on all containers you have made, or just the R container?

only the R container for now.

What distribution are you building on?

ubuntu 14.04
Linux 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

How can I try and replicate the problem? What is your environment usually configured to with respect > to LC*, LANG, C, etc...?

%locale
LANG=en_US.UTF-8
LANGUAGE=en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

To reproduce it on a Debian/ubuntu

install base R
%sudo apt-get install r-base

r.spec

Name: R302
Exec: /usr/bin/R
%files
/usr/lib/R
/bin/uname
/usr/share/locale/locale.alias
/usr/lib/locale/locale-archive
/usr/share/locale-langpack/en/LC_MESSAGES

build and run

[gmkurtzer]

Thanks, I'll play with this as soon as I get to my desktop!

[gmkurtzer]

Hi Karl,

The problem was that I had a TODO/FIXME in the code related to clobbering the locale-archive. Until I decide how to deal with that longer term I disabled that clobbering. With the latest commit, the following now works for me:

Name: R
Exec: /usr/bin/R

%files
/bin/rm
/bin/uname
/usr/lib/R
/usr/lib/locale/locale-archive

[kforner]

indeed, that fixes it. Thanks !

[gmkurtzer]

Excellent! I think I am going to create a specfile key to disable locales (for people that want very small containers), but by default it will include.

Thank you for your assistance and patience debugging this!

[gmkurtzer]

Hi Karl,

Can you please it one more time? I have it now such that you shouldn't have to specify any additional %files when R is included in the container. Can you try it again from scratch with only this in your specfile:

Name: R
Exec: /usr/bin/R

Thank you!

[kforner]

It works great !
I'm impressed, how does it manage to include the installed R packages too ?
Does it take the whole /usr/lib/R ?

[gmkurtzer]

Check out this file:

https://github.com/gmkurtzer/singularity/blob/master/libexec/mods/r.dep

I'm only grabbing the files it needs for proper operation (e.g. not including docs and other extraneous bits - I hope, I'm not an R expert).

Also if you Exec a script that uses Rscript as the interpreter it will also work with nothing else listed.

Please let me know if there are any other things I can/should catch for proper R inclusion.

Greg

[kforner]

Hmm I see.
Is there some way to disable this automatic process, if you want to remove some files, or control exactly what is happening ?

Other question : how do the processes running in the container distinguish between the contain filesystem and the user/host filesystem. e.g for R how to use a R package installed in the user home dir, or how to install a package inside the container ?

[gmkurtzer]

I have a plan to create a flag to disable the dependency check(s) so the only files which get automatically included into the container are the ones specifically expressed. Keep in mind this would become very very difficult for someone to manage on their own.... But possible.

The files that get used for containers Vs. what is in our home directory is a good question. The R module stack included in the container will all be executed as you would expect and if you have custom installed modules in your home directory I would think these would also get used. But again I'm not familiar enough with R to know for sure.

In theory (at least in other dynamic loading scripting interfaces) there is a potential for incompatibility here. Because the modules installed in the users home directory were built against the root operating system they may not be compatible with the contents of the container. Hopefully there is a way to disable that if it becomes necessary.

Now that the bug is fixed we should probably move this discussion to the Google Group so others can benefit and join in the discussion.

Greg

[gmkurtzer]

BTW, you can now disable the internal dependency checks with the specfile option: "NoDeps: yes".

[kforner]

woot

[raphaunix]

This worked for me:

echo "LC_ALL=en_US.UTF-8" >> /etc/environment
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
locale-gen en_US.UTF-8

Enjoy.

[NataliaDiaz]

Running 'ssh-add' command in the terminal (this command adds the ssh key to the ssh agent) and export LC_ALL=fr_FR.UTF-8 in the bashrc made the trick!

[SuperShinyEyes]

I ran into this issue as I tried to run tensorboard.
I built a simg using this Dockerfile:

FROM tensorflow/tensorflow:1.7.0-gpu-py3

# This layer is supposed to solve but didn't.
RUN apt-get clean && apt-get update && apt-get install -y locales && \
    export LANGUAGE=en_US.UTF-8 && \
    export LANG=en_US.UTF-8 && \
    export LC_ALL=en_US.UTF-8 && \
    locale-gen en_US.UTF-8 && \
    dpkg-reconfigure locales

# This doesn't solve either
RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment     && \
    echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen     && \
    echo "LANG=en_US.UTF-8" > /etc/locale.conf     && \
    locale-gen en_US.UTF-8

# Solve Jupyter permission issue
CMD unset XDG_RUNTIME_DIR && \
    jupyter notebook --port=12220 --no-browser

When I run the following in the image

locale -a

I get

locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.UTF-8
POSIX
en_US.utf8

[SuperShinyEyes]

I fixed with the following Dockerfile:

FROM tensorflow/tensorflow:1.7.0-gpu-py3

# Solve locale issues when running Tensorboard.
# My host has language-pack-fi but I want to use
# language-pack-en in the container
RUN apt-get clean && apt-get update && apt-get install -y \
    locales \
    language-pack-fi  \
    language-pack-en && \
    export LANGUAGE=en_US.UTF-8 && \
    export LANG=en_US.UTF-8 && \
    export LC_ALL=en_US.UTF-8 && \
    locale-gen en_US.UTF-8 && \
    dpkg-reconfigure locales

# Solve Jupyter permission issue
CMD unset XDG_RUNTIME_DIR && \
    jupyter notebook --port=12220 --no-browser

[ecogest]

A way very similar to raphaunix's (but a bit shorter), on debian 9, was to :

• uncomment #en_US.UTF-8 UTF-8 in /etc/locale.gen
• and then:
  sudo locale-gen en_US.UTF-8

I didn't need the other two commands.

[bsafwen]

@raphaunix

This worked for me:

echo "LC_ALL=en_US.UTF-8" >> /etc/environment
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
locale-gen en_US.UTF-8

Enjoy.

Man I cannot thank you enough.

[sethiele]

I've tried all your ways, but it didn't helped.

I got this error:

rake aborted! ArgumentError: invalid byte sequence in US-ASCII /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/utils.rb:34:in `gsub' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/utils.rb:34:in `clean_content' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/utils.rb:18:in `create_nokogiri' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:117:in `check_path' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:89:in `block in process_files' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:89:in `map' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:89:in `process_files' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:70:in `check_files' /usr/local/bundle/gems/html-proofer-3.11.0/lib/html-proofer/runner.rb:42:in `run' /builds/sethiele/geekparents/Rakefile:51:in `block in <top (required)>' /usr/local/bundle/gems/rake-12.3.2/exe/rake:27:in `<top (required)>' Tasks: TOP => test (See full trace by running task with --trace)

[jscook2345]

@sethiele

Can you open a new issue and reference this one as something you tried?

What does locale -a print for you in your container?

[forskamse]

Under CentOS 6/7, use the following command to solve it:
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8

[marioroy]

The following works for Debian 9 and Ubuntu 18.04.

See Using UTF-8 in Debian
https://fruit.je/utf-8

apt-get update
apt-get install -y locales

echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen
locale-gen && update-locale LANG=en_US.UTF-8

[soichih]

I've added this to the end of Dockerfile so I can run the container via singularity

RUN apt-get install -y locales
RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && echo "LANG=en_US.UTF-8" > /etc/locale.conf && locale-gen en_US.UTF-8
ENV LC_CTYPE="en_US.UTF-8"