Salt: virtualenv 20 changed version output from stdout to stderr

[gabeio]

Description of Issue

It seems that virtualenv v20 changed their version output from stdout to stderr.

https://github.com/saltstack/salt/blob/b4dd2352f92a3d67fcfafefadd95dfd39a8e8c2e/salt/modules/virtualenv_mod.py#L173

It correctly returns a 0 status code but because stdout is empty not ret['stdout'].strip() throws us into the exception.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

/project/.venv:
  virtualenv.managed:
    - user: ubuntu
    - python: python2.7

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Attempt to create a virtualenv with the latest virtualenv installed (v20+)

(this is being run withing packer which is why it has the ==> amazon-ebs:)

==> amazon-ebs: [INFO    ] Running state [/project/.venv] at time 22:43:04.915533
==> amazon-ebs: [INFO    ] Executing state virtualenv.managed for [/project/.venv]
==> amazon-ebs: [INFO    ] Executing command ['/usr/local/bin/virtualenv', '--version'] as user 'ubuntu' in directory '/home/ubuntu'
==> amazon-ebs: [DEBUG   ] env command: ('su', '-s', '/bin/bash', '-', 'ubuntu', '-c', '/usr/bin/python3')
==> amazon-ebs: [DEBUG   ] stderr: virtualenv 20.0.4 from /usr/local/lib/python2.7/dist-packages/virtualenv/__init__.pyc
==> amazon-ebs: [DEBUG   ] An exception occurred in this state: Unable to get the virtualenv version output using '['/usr/local/bin/virtualenv', '--version']'. Returned data: {'pid': 13344, 'retcode': 0, 'stdout': '', 'stderr': 'virtualenv 20.0.4 from /usr/local/lib/python2.7/dist-packages/virtualenv/__init__.pyc'}
==> amazon-ebs: Traceback (most recent call last):
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/modules/virtualenv_mod.py", line 161, in create
==> amazon-ebs:     import virtualenv
==> amazon-ebs: ModuleNotFoundError: No module named 'virtualenv'
==> amazon-ebs:
==> amazon-ebs: During handling of the above exception, another exception occurred:
==> amazon-ebs:
==> amazon-ebs: Traceback (most recent call last):
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/state.py", line 1933, in call
==> amazon-ebs:     **cdata['kwargs'])
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/loader.py", line 1951, in wrapper
==> amazon-ebs:     return f(*args, **kwargs)
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/states/virtualenv_mod.py", line 209, in managed
==> amazon-ebs:     **kwargs
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/modules/virtualenv_mod.py", line 176, in create
==> amazon-ebs:     'Returned data: {1}'.format(version_cmd, ret)
==> amazon-ebs: salt.exceptions.CommandExecutionError: Unable to get the virtualenv version output using '['/usr/local/bin/virtualenv', '--version']'. Returned data: {'pid': 13344, 'retcode': 0, 'stdout': '', 'stderr': 'virtualenv 20.0.4 from /usr/local/lib/python2.7/dist-packages/virtualenv/__init__.pyc'}
==> amazon-ebs: [ERROR   ] An exception occurred in this state: Traceback (most recent call last):
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/modules/virtualenv_mod.py", line 161, in create
==> amazon-ebs:     import virtualenv
==> amazon-ebs: ModuleNotFoundError: No module named 'virtualenv'
==> amazon-ebs:
==> amazon-ebs: During handling of the above exception, another exception occurred:
==> amazon-ebs:
==> amazon-ebs: Traceback (most recent call last):
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/state.py", line 1933, in call
==> amazon-ebs:     **cdata['kwargs'])
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/loader.py", line 1951, in wrapper
==> amazon-ebs:     return f(*args, **kwargs)
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/states/virtualenv_mod.py", line 209, in managed
==> amazon-ebs:     **kwargs
==> amazon-ebs:   File "/usr/lib/python3/dist-packages/salt/modules/virtualenv_mod.py", line 176, in create
==> amazon-ebs:     'Returned data: {1}'.format(version_cmd, ret)
==> amazon-ebs: salt.exceptions.CommandExecutionError: Unable to get the virtualenv version output using '['/usr/local/bin/virtualenv', '--version']'. Returned data: {'pid': 13344, 'retcode': 0, 'stdout': '', 'stderr': 'virtualenv 20.0.4 from /usr/local/lib/python2.7/dist-packages/virtualenv/__init__.pyc'}
==> amazon-ebs:
==> amazon-ebs: [INFO    ] Completed state [/project/.venv] at time 22:43:05.357418 (duration_in_ms=441.883)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2019.2.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Nov  7 2019, 10:44:02)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1057-aws
         system: Linux
        version: Ubuntu 18.04 bionic

[redbaron4]

Another potential issue here is that the format of version display has changed too

For older systems:

$ virtualenv --version 2> /dev/null
15.1.0

For 20.0+:

# virtualenv --version > /dev/null
virtualenv 20.0.5 from /usr/lib/python2.7/site-packages/virtualenv/__init__.pyc

[Ch3LL]

im not able to replicate the issue where virtualenv's stdout is empty on virtualenv versions 20.0.0, 20.0.5, etc, but the new output from virtualenv is going to cause a valuerror. i'll look to fix that. I actually have a PR here https://github.com/saltstack/salt/pull/56360 thats working on getting it to work with 20.0.10. If you try 20.0.10 do you still see an empty stdout?

[Ch3LL]

fix here for the version output: https://github.com/saltstack/salt/pull/56364

like previously stated I cant replicate the behavior your seeing where the output is added to sderr instead. any additional information to help replicate that? It might be an upstream bug if it is indeed logging to stderr

[redbaron4]

@Ch3LL

Apparently this is a Python2vs3 issue.

Running the command on console still shows that output is going to stderr (This is using Python2)

# virtualenv --version 2> /dev/null
# virtualenv --version > /dev/null
virtualenv 20.0.10 from /usr/lib/python2.7/site-packages/virtualenv/__init__.pyc

With any virtualenv prior to 20

# virtualenv --version 2> /dev/null
16.7.10
# virtualenv --version > /dev/null
# 

Looking at the virtualenv code they have made major changes to v20.0. Before v20, they used optparse (which logged to stdout). See code on legacy branch here. Now they have switched to using argparse and action=version (See master branch here) and this issue makes it look like action version logs to stderr for Python2 (they have fixed it in Python3)

When I use Python3

# python --version
Python 3.6.3
# python -m virtualenv --version 2> /dev/null
virtualenv 20.0.10 from /opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/virtualenv/__init__.py
# python -m virtualenv --version > /dev/null
#

For cross python support, maybe both stdout and stderr can be checked or docs can be updated to ask user to use virtualenv<20.0 for Python2 installations.

[OrangeDog]

IMO salt should be using the built-in venv module (#51742), not an external tool.