Salt: Running python scripts w/ `cmd.run` results in bizarre default encoding

[eykd]

Description of Issue/Question

When running a python script (v2.x or v3.x) via cmd.run state, I'm getting a bizarre default encoding:

$ sudo salt 'myserver' cmd.run "python3 -c 'import locale; print(locale.getpreferredencoding())'"
myserver:
    ANSI_X3.4-1968

$ sudo salt 'myserver' cmd.run "python2 -c 'import locale; print(locale.getpreferredencoding())'"
myserver:
    ANSI_X3.4-1968

However, running this w/ salt-call gives me the expected default of UTF-8:

$ sudo salt-call cmd.run "python3 -c 'import locale; print(locale.getpreferredencoding())'"
local:
    UTF-8

Versions Report

$ salt --versions-report                                                                                                                                  [3/384]
Salt Version:
           Salt: 2016.11.5

Dependency Versions:
           cffi: 1.10.0
       cherrypy: Not Installed
       dateutil: 2.4.0
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-83-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[eykd]

Still an issue in 2017.7.2 (yeah, I know, we're behind the curve). I'm curious if anyone can confirm the bug in the latest?

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[AstraLuma]

I just stumbled on this issue.

The default cmd.run (and friends) configuration looses the system encoding, setting it to C. (Hardcoded in the source.)

This causes explosive problems for software based on click but is a potential lurking problem for anything based on Python3 (because it detects the system encoding as ascii, causing problems if eg it reads a utf-8 file or path), as well for anything that intelligently processes encodings.

The code causing problems is https://github.com/saltstack/salt/blob/b6a4394002574696011f67232ef36536c48bef5d/salt/modules/cmdmod.py#L553-L565 . This has existed since at least 2017.7.

The work around is to set reset_system_locale: false in all invocations of cmd.run (and set the system encoding to C.___ on all minions).

The fix is to either:

• Look at the current locale setting a preserve encoding information when setting environments
• Instead of setting LC_* directly, defer to the locale machinery (assuming it'll set envvars on salt's behalf)

(Click is special because it proactively detects when Python is running with ascii stdio set and will bomb out.)

[AstraLuma]

Oh, and I believe this is the same mechanism by which the filesystem encoding is set, so Salt's erasing that information.

[AstraLuma]

The suggested fix is:

lang, encoding = locale.getlocale()
new_locale = 'C' if encoding is None else "C" + encoding

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[AstraLuma]

still a bug.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.