Salt: Strange pip/virtualenv installation issue

[frnsys]

I've been struggling with an issue for the past couple days, and I'm at a loss at how to fix it.

I am using Salt to setup a Python virtualenv (virtualenv-3.3) on an Ubuntu 13.04 server (Minion) and then install pip packages through a requirements.txt file.

The highstate command keeps failing at the installation of python-dateutil (version 2.1), where I get the error:

File "<string>", line 16, in <module>
File "/var/app/digester/dev-env/build/python-dateutil.setup.py", line 16, in <module>
open(TOPDIR + "/dateutil/__init__.py").read()).group(1)
File "/var/app/digester/dev-env/lib/python3.3/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 193: ordinal not in range(128)

The relevant part of the state file I'm using looks like:

pip-pkgs:
    pip.installed:
        - names:
            - virtualenv
            - cython
        - bin_env: pip-3.3
        - require:
            - pkg: app-pkgs

app-venv:
    virtualenv.managed:
        - name: /var/app/digester/dev-env
        - cwd: /var/app/digester/
        - venv_bin: virtualenv-3.3
        - requirements: /var/app/digester/requirements.txt
        - no_site_packages: true
        - require:
            - pkg: app-pkgs
            - pip: pip-pkgs
            - git: digester

I have also tried using cmd.run and executing pip install -r requirements.txt through a shell script (after activating the virtualenv).

But the strange thing is, if I go in to the directory, activate the virtualenv, and then manually install python-dateutil, the installation goes through without any issue.

So I am unable to reproduce this issue in any other way except through Salt.

I'm new to Salt, so I'm not sure where I should be looking for a solution. Is there some configuration option for the virtualenv state that I'm missing? I would greatly appreciate any help!

[basepi]

Hrm, I'm not sure what's going on here. It seems like it might be an encoding error in the virtualenv module with regards to requirements.txt.

Have you tried doing the pip install -r requirements.txt directly, not through cmd.run? If it still errors then, then I think it's a bug in pip or virtualenv, not in salt.

[frnsys]

If I manually run the command, it works without a problem.

I've got a workaround which just checks out the latest version of python-dateutil, and it has been working fine with that.

I'm still curious about what the issue could be, I was having a lot of trouble figuring it out!

[basepi]

Well it's very strange, because the traceback is coming from python-dateutils.setup.py, not salt! So it boggles my mind that it is failing over cmd.run but not when you run it directly.

I'll put it on the current milestone, and hopefully someone can shed some light on it.

[cvrebert]

This is most likely locale-related, but IMO definitely a bug in dateutil's setup.py.
You probably set a locale in your .bashrc or whatever, whereas the user Salt runs under probably just uses the system default.

python-dateutil-2.1/setup.py lines 15-16:

VERSION = re.search('__version__ = "([^"]+)"',
                    open(TOPDIR + "/dateutil/__init__.py").read()).group(1)

So it's reading a text file _without explicitly specifying which encoding to use_.

python-dateutil-2.1/dateutil/__init__.py line 8:

line 1: # -*- coding: utf-8 -*-
...
line 8: __author__ = "Tomi Pieviläinen <[email protected]>"

Note the Unicode a-with-umlaut, which means ASCII won't work; actual UTF-8 needs to be used.

From open()'s docs in Python 3.3:

In text mode, if encoding is not specified the encoding used is platform dependent:
locale.getpreferredencoding(False) is called to get the current locale encoding.

Evidently, when run by Salt, Python is detecting/guessing ASCII as this encoding, thus causing the file read to fail.

dateutil's setup.py ought to explicitly use UTF-8 when reading __init__.py.

[basepi]

Wow, thanks for doing all the code diving, @cvrebert!

@ftzeng I would recommend filing a bug with the dateutil guys: https://launchpad.net/dateutil

[kahrens]

I have this same configuration and am having this very same issue occur when the virtualenv state uses a requirements file to install packages. In my case, I get the UnicodeDecodeError when it is trying to install Pillow.

As was mentioned in this thread, it is a locale problem, but I disagree with saying it is not a Salt issue. The problem is not with all the individual packages having illegal characters in them. If the individual packages are pip installed (outside of Salt) with a locale that supports the characters (like en_US.UTF-8), the problem does not occur. The problem appears to be that the virtualenv.manage state is calling pip via the pip module, which in turn uses cmdmod module run_all() function to do the work. cmd.run_all() will call the _run() function, but does not specify a value for the argument 'reset_system_locale,' which results in it being defaulted to True. When reset_system_locale is True, the _run() function will execute this code:

if reset_system_locale is True:
    if not salt.utils.is_windows():
        # Default to C!
        # Salt only knows how to parse English words
        # Don't override if the user has passed LC_ALL
        env.setdefault('LC_ALL', 'C')
    else:
        # On Windows set the codepage to US English.
        cmd = 'chcp 437 > nul & ' + cmd

The env.setdefault() call will unconditionally override the system locale to be 'C.' This code provides the option to override the system locale when using cmd.run (via a command line salt command for example), but doesn't appear to have a way to override this when it is used via a state like virtualenv.

Is there a way to set reset_system_locale to False for the virtualenv state?

As ftzeng stated in his original post, if you run the pip install manually or outside of virtualenv.manage, things work fine. One would expect this since the cmd._run() call is not overriding the system default locale to 'C' in those cases.

I'm currently using Salt 0.17.4 on Ubuntu 13.04.

[basepi]

Yes, we could definitely add an argument to virtualenv.managed that would allow us to pass through a value for reset_system_locale.

Any chance you'd be willing to submit a pull req?

[kahrens]

Sure, I'll submit a request. Thanks for the reply!

[agateblue]

I'm facing this exact issue, with the last packaged version of salt on Debian : running pip install directly on the minion works, but it fails with UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position xxx when executed from cmd.run or pip.install state.

As I'm totally new to salt, I don't feel confident to write a PR for this, unfortunately.

[vinta]

A workaround to this issue is editing /etc/default/locale:

LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_ALL=en_US.UTF-8