Salt: virtualenv.managed does not use env_vars when creating the virtualenv

Created on 19 Jul 2018  路  15Comments  路  Source: saltstack/salt

Description of Issue/Question

We use Salt to manage internal hosts that don't have external access to sites like PyPI, but we so however run an internal PyPI mirror. This means when we run virtualenv to create a new virtualenv on one of these internal hosts they need to point to the internal PyPI mirror to fetch any needed python packages (ie. setuptools, pip). Usually we do this by setting the PIP_INDEX_URL environment variable before running virtualenv.

I have tried passing Salt the correct PIP_INDEX_URL environment variable using env_vars but Salt doesn't seem to use these env_vars when invoking virtualenv.

Setup

virtualenv:
  pkg.installed:
    - name: python-virtualenv

salt-cloud-venv:
  virtualenv.managed:
    - name: /opt/salt-cloud/
    - system_site_packages: True
    - index_url: https://<internal mirror>
    - pip_pkgs:
      - pyvmomi >= 6.0.0.2016.6
    - env_vars:
        PIP_INDEX_URL: https://<internal mirror>
    - require:
      - pkg: virtualenv

Steps to Reproduce Issue

salt-call state.apply --output-diff

[ERROR   ] Command '[u'/usr/bin/virtualenv', u'--system-site-packages', u'/opt/salt-cloud/']' failed with return code: 1
[ERROR   ] stdout: New python executable in /opt/salt-cloud/bin/python2
Also creating executable in /opt/salt-cloud/bin/python
Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel:
  Collecting setuptools
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/opt/salt-cloud/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[ERROR   ] stderr: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
    main()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel failed with error code 2
[ERROR   ] retcode: 1
[ERROR   ] New python executable in /opt/salt-cloud/bin/python2
Also creating executable in /opt/salt-cloud/bin/python
Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel:
  Collecting setuptools
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/opt/salt-cloud/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
    main()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel failed with error code 2
local:
----------
          ID: salt-cloud-venv
    Function: virtualenv.managed
        Name: /opt/salt-cloud/
      Result: False
     Comment: New python executable in /opt/salt-cloud/bin/python2
              Also creating executable in /opt/salt-cloud/bin/python                                                                
              Installing setuptools, pkg_resources, pip, wheel...                                                                   
                Complete output from command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel:                      
                Collecting setuptools                                                                                               
              Exception:                                                                                                            
              Traceback (most recent call last):                                                                                    
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main                
                  status = self.run(options, args)                                                                                  
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 353, in run            
                  wb.build(autobuilding=True)                                                                                       
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 749, in build                     
                  self.requirement_set.prepare_files(self.finder)                                                                   
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files       
                  ignore_dependencies=self.ignore_dependencies))                                                                    
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file       
                  require_hashes                                                                                                    
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link   
                  self.link = finder.find_requirement(self, upgrade)                                                                
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement          
                  all_candidates = self.find_all_candidates(req.name)                                                               
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates       
                  for page in self._get_pages(url_locations, project_name):                                                         
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages                
                  page = self._get_page(location)                                                                                   
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page                 
                  return HTMLPage.get_page(link, session=self.session)                                                              
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 792, in get_page                  
                  "Cache-Control": "max-age=600",                                                                                   
                File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 501, in get                                                                                                                                  
                  return self.request('GET', url, **kwargs)                                                                         
                File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 386, in request                
                  return super(PipSession, self).request(method, url, *args, **kwargs)                                              
                File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 488, in request                                                                                                                              
                  resp = self.send(prep, **send_kwargs)                                                                             
                File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 609, in send                                                                                                                                 
                  r = adapter.send(request, **kwargs)                                                                               
                File "/opt/salt-cloud/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 47, in send                                                                                                                           
                  resp = super(CacheControlAdapter, self).send(request, **kw)                                                       
                File "/opt/salt-cloud/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/adapters.py", line 423, in send                                                                                                                                 
                  timeout=timeout                                                                                                   
                File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 643, in urlopen                                                                                                                          
                  _stacktrace=sys.exc_info()[2])                                                                                    
                File "/opt/salt-cloud/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment                                                                                                                            
                  total -= 1                                                                                                        
              TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'                                                      
              ----------------------------------------                                                                              
              ...Installing setuptools, pkg_resources, pip, wheel...done.                                                           
              Running virtualenv with interpreter /usr/bin/python2Traceback (most recent call last):                                
                File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
                  main()
                File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
                  symlink=options.symlink)
                File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
                  download=download,
                File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
                  call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
                File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
                  % (cmd_desc, proc.returncode))
              OSError: Command /opt/salt-cloud/bin/python2 - setuptools pkg_resources pip wheel failed with error code 2
     Started: 12:41:12.314248
    Duration: 2210.273 ms
     Changes:   

Summary for local
-------------
Succeeded: 1
Failed:     1
-------------
Total states run:     2
Total run time:    1.920 s

The error would appear to be due to the fact that the internal host can't connect to the PyPI to download the setuptools package. If virtualenv where to have the env_vars variables set before it was invoked then the issue would not occur.

Versions Report

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Nov 24 2017, 17:33:09)
   python-gnupg: 0.3.9
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.5 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-6-amd64
         system: Linux
        version: debian 9.5
info-needed
Source
picture jerrykan
👍2

All 15 comments

@jerrykan Thanks for the report. Unfortunately I have not been able to reproduce this one. The env_vars are used by pip is called. Using the state above I see it attempt to connect to a local URL to install packages, it fails because I didn't have one running, but still attempted. In additional to the information that you provided above, can you also provide what version of virtualenv you're using and also the version of urllib3 that you have installed?

garethgreenaway picture garethgreenaway on 19 Jul 2018

@garethgreenaway Thanks for looking at this. The part where Salt calls pip directly seems to work fine. The issue is when Salt calls virtualenv to create the new virtualenv. As best as I can tell virtualenv invokes the system pip to install setuptools and pip within the virtualenv when it is first created. However because the environment variables aren't set before Salt invokes virtualenv, when virtualenv invokes the system pip it doesn't know to use the internal PyPI mirror.

If I manually create the virtualenv first using something like cmd.run and then use virtualenv.managed to mange the virtualenv, then everything seems to work OK. ie:

virtualenv:
  pkg.installed:
    - name: python-virtualenv

salt-cloud-venv:
  cmd.run:
    - name: virtualenv --system-site-packages /opt/salt-cloud
    - unless: test -d /opt/salt-cloud
    - env:
      - PIP_INDEX_URL: https://<internal mirror>
    - require:
      - pkg: virtualenv

  virtualenv.managed:
    - name: /opt/salt-cloud/
    - system_site_packages: True
    - index_url: https://<internal mirror>
    - pip_pkgs:
      - pyvmomi >= 6.0.0.2016.6
    - env_vars:
        PIP_INDEX_URL: https://<internal mirror>
    - require:
      - cmd: salt-cloud-venv
jerrykan picture jerrykan on 20 Jul 2018
👍1

Have the same issue. It seems like since the virtualenv command gets passed down to "cmd.run_all" it should be allowed to use the "env" kwarg in "virtualenv.managed" and "virtualenv.create". That might be confusing to have "env" and "env_var" though.

eoswald picture eoswald on 13 Sep 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] picture stale[bot] on 9 Jan 2020
😕1

I can only assume this is still a valid issue

jerrykan picture jerrykan on 10 Jan 2020

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] picture stale[bot] on 10 Jan 2020

I am having the same issue.. trying to pass command line args to virtualenv during the create phase.

I tried setting VIRTUALENV_ via - env_vars: which will be inspected by virturalenv. but they are not set when the venv cmd is run.

I was able to work around it by setting env: which is passed along via kwargs to cmd.run_all in the modules

    - env:
        VIRTUALENV_DOWNLOAD: 'true'
philipmattingly picture philipmattingly on 7 Feb 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] picture stale[bot] on 8 Mar 2020

This wasn't stale 2 months ago, it still isn't stale now.

jerrykan picture jerrykan on 9 Mar 2020

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] picture stale[bot] on 9 Mar 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] picture stale[bot] on 8 Apr 2020

Is there any way to stop issues being marked as stale every single month... it seems a little bit aggressive.

jerrykan picture jerrykan on 8 Apr 2020
👍1

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] picture stale[bot] on 8 Apr 2020

the stale bot has been disabled, but we do still need some automation in place for follow up, clearly we missed it here. @jerrykan can you confirm the salt version you are still seeing this, please? For Salt v2018.3.x we are in Extended Life Support so only if something is critical is identified, are you still on v2018.3? Are you able to upgrade?

sagetherage picture sagetherage on 19 Jun 2020

I've tested this with Salt v3000.2 and not seeing the problem any more. So I'll close the issue.

jerrykan picture jerrykan on 25 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mooperd picture mooperd  路  3Comments
sfozz picture sfozz  路  3Comments
Inveracity picture Inveracity  路  3Comments
twangboy picture twangboy  路  3Comments
sagetherage picture sagetherage  路  3Comments