Salt: pip_state: An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.

[silenius]

Description of Issue/Question

pip.installed fails to run:

----------
          ID: devpi_venv_pkgs
    Function: pip.installed
      Result: False
     Comment: An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
              An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
     Started: 16:46:18.340387
    Duration: 1.242 ms
     Changes:   

Setup

Relevant parts:

  devpi_pip_pkg:
    __env__: base
    __sls__: venvs.webapps.devpi
    pkg:
    - name: py35-pip
    - require:
      - pkg: devpi_python_pkg
    - installed
    - order: 10007

  devpi_python_pkg:
    __env__: base
    __sls__: venvs.webapps.devpi
    pkg:
    - name: python35
    - installed
    - order: 10005

  devpi_venv:
    __env__: base
    __sls__: venvs.webapps.devpi
    virtualenv:
    - name: /usr/local/venvs/devpi
    - venv_bin: /usr/local/bin/pyvenv-3.5
    - system_site_packages: false
    - user: www-devpi
    - clear: false
    - pip_upgrade: true
    - pip_pkgs:
      - pip
      - setuptools
      - certifi
    - require:
      - file: /usr/local/venvs/devpi
      - user: www-devpi
      - pkg: devpi_python_pkg
      - pkg: postgresql93-client
    - managed
    - order: 10006

  devpi_venv_pkgs:
    __env__: base
    __sls__: venvs.webapps.devpi
    pip:
    - pkgs:
      - devpi-server
      - devpi-web
    - user: www-devpi
    - cwd: /usr/local/venvs/devpi
    - bin_env: /usr/local/venvs/devpi/bin/pip
    - reload_modules: true
    - require:
      - virtualenv: devpi_venv
      - pkg: devpi_pip_pkg
    - installed
    - order: 10008

This is my complete sls:

{% if cfg.lang == 'python' %}

##########
# PYTHON #
##########

# Python packages
{{ venv }}_python_pkg:
  pkg.installed:
    - name: python{{ cfg.python_version|string|replace('.', '') }}

# Python virtualenv (only needed for Python 2)
{% if cfg.python_version == 2.7 %}
{{ venv }}_virtualenv_pkg:
  pkg.installed:
    - name: py27-virtualenv
{% endif %}

# Install the virtualenv
{{ venv }}_venv:
  virtualenv.managed:
    - name: {{ cfg.full_path }}
    - venv_bin: {{ cfg.venv_builder }}
    - system_site_packages: False
    - user: {{ cfg.user }}
    - clear: False
    - pip_upgrade: True
    - pip_pkgs:
      - pip
      - setuptools
      - certifi
    - require:
      - file: {{ cfg.full_path }}
      - user: {{ cfg.user }}
      - pkg: {{ venv }}_python_pkg
      - pkg: {{ salt['pillar.get']('postgresql:lookup:pkg_client') }}

{% if cfg.pip_pkgs is defined %}
{{ venv }}_pip_pkg:
  pkg.installed:
    - name: py{{ cfg.python_version|string|replace('.', '') }}-pip
    - require:
      - pkg: {{ venv }}_python_pkg

{{ venv }}_venv_pkgs:
  pip.installed:
    - pkgs:
        {{ cfg.pip_pkgs|yaml }}
    - user: {{ cfg.user }}
    - cwd: {{ cfg.full_path }}
    - bin_env: {{ cfg.pip }}
    - reload_modules: True
    - require:
      - virtualenv: {{ venv }}_venv
      - pkg: {{ venv }}_pip_pkg
{% endif %}

# Upgrade pip,setuptools,certifi to latest version (once the virtualenv is installed)
#{{ venv }}_upgrade_pip:
#  cmd.run:
#    - name: {{ cfg.pip }} --no-cache-dir install -U pip setuptools certifi
#    - runas: {{ cfg.user }}
#    - reload_modules: True
#    - require:
#      - virtualenv: {{ venv }}_venv

Steps to Reproduce Issue

Any pip.installed with Python 3 (haven't tested with Python 2) should fail, even when the pip package is installed globally (which should not be needed anyway).

This is because of https://github.com/saltstack/salt/blob/develop/salt/states/pip_state.py#L112-L119

Versions Report

salt% salt --versions-report
Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: 1.11.2
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: 1.5.0
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: 1.5.2
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.14 (default, Jan  8 2018, 23:05:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 16.0.3
           RAET: 0.6.5
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:   
         locale: UTF-8
        machine: amd64
        release: 10.3-RELEASE
         system: FreeBSD
        version: Not Installed

[silenius]

Mmh I think I found the issue, Salt runs 2.7 and installed pip module was 3.5, it works when the py27-pip package is installed. Now I wonder why the pip package must be installed globally as it is already available in the virtualenv. I haven't yet looked in depth in the code, but I think that if bin_env is provided it should not be necessary to install pip globally .. or I missed something?

[gtmanfred]

The pip module for the version of python that salt is running with needs to be installed, so that it can be imported and used to compare versions of packages.

[ketzacoatl]

I am seeing this on 2018.3.0, and I believe this ticket should be re-opened (unless of course, I'm misunderstanding and there is known solution I'm not following) .

Here is my versions report:

# salt --versions-report                                                                                                                        [2/9642]
Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-112-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Pip is installed:

# pip --version
pip 10.0.1 from /usr/local/lib/python2.7/dist-packages/pip-10.0.1-py2.7.egg/pip (python 2.7)

and it works:

# pip install django
Collecting django
  Downloading https://files.pythonhosted.org/packages/92/56/1f30c1e6a58b0c97c492461148edcbece2c6e43dcc3529695165744349ee/Django-1.11.12-py2.py3-none-any.whl (6.9MB)
    4% |█▍                              | 296kB 103kB/s eta 0:01:04^C
Operation cancelled by user
...

But the same operation fails in salt's pip state or module:

# salt-call --local state.low '{"state": "pip", "fun": "installed", "name": "django"}'                                                                  
...
[ERROR   ] An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
local:
    ----------   
    __id__:
        django   
    __run_num__:     
        0
    __sls__:
        None
    changes:
        ----------
    comment:
        An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
    duration:    
        21.694       
    name:
        django       
    result:
        False        
    start_time:  
        10:10:55.258097

[ketzacoatl]

With 2017.7.3, and pip 9.0.3, I can get this to work:

    default: Reading package lists...
    default: Building dependency tree...
    default: Reading state information...
    default: python-pip is already the newest version (8.1.1-2ubuntu0.4).
    default: 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    default: Requirement already up-to-date: pip==9.0.3 in /usr/local/lib/python2.7/dist-packages
    default: You are using pip version 9.0.3, however version 10.0.1 is available.
    default: You should consider upgrading via the 'pip install --upgrade pip' command.
    default: local:
    default:     ----------
    default:     __id__:
    default:         django
    default:     __run_num__:
    default:         0
    default:     __sls__:
    default:         None
    default:     changes:
    default:         ----------
    default:     comment:
    default:         Python package django was already installed
    default:         All packages were successfully installed
    default:     duration:
    default:         4436.845
    default:     name:
    default:         django
    default:     result:
    default:         True

That was tested with salt-call --local state.low '{"state": "pip", "fun": "installed", "name": "django", "__id__": "django"}'. I will test other combinations of versions of saltstack and pip to see if I can pinpoint the issue more specifically. I think it might have to do with changes in pip 10.x

[ketzacoatl]

It seems related to pip 10.x:

Here I downgrade to pip 9.0.3:

root@ubuntu-xenial:~# pip install --upgrade pip==9.0.3
Collecting pip==9.0.3
  Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 422kB/s 
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-9.0.3
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Then install django with pip.installed and state.low:

root@ubuntu-xenial:~# salt-call --local state.low '{"state": "pip", "fun": "installed", "name": "django", "__id__": "django"}'
local:
    ----------
    __id__:
        django
    __run_num__:
        0
    __sls__:
        None
    changes:
        ----------
        Django==1.11.12:
            Installed
    comment:
        All packages were successfully installed
    duration:
        36384.096
    name:
        django
    result:
        True
    start_time:
        10:12:27.912056

Upgrade to pip 10.0.1:

root@ubuntu-xenial:~# pip install --upgrade pip
Cache entry deserialization failed, entry ignored
Collecting pip
  Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 180kB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.1

Re-try using pip.installed:

root@ubuntu-xenial:~# salt-call --local state.low '{"state": "pip", "fun": "installed", "name": "django", "__id__": "django"}'
[ERROR   ] An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
local:
    ----------
    __id__:
        django
    __run_num__:
        0
    __sls__:
        None
    changes:
        ----------
    comment:
        An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
    duration:
        6.134
    name:
        django
    result:
        False
    start_time:
        10:14:57.890680

[ketzacoatl]

Where is saltstack trying to use pip's internals? Most places I look in the code I see saltstack using the pip executable, rather than importing and running from within salt. Is salt using pip in some way to introspect before calling the executable?

[gtmanfred]

It is used in the pip state for comparing versions.

[ketzacoatl]

Ah yes, your previous comment about that is now more clear to me. Using pip internals means salt breaks when those internals changes. Is there a way to do the version comparison using the executable instead of pip's internals?

[rosscdh]

Suffering the same in Salt: 2018.3.0

[ketzacoatl]

@rosscdh to confirm, the short-term workaround is to not use pip 10.x, eg pin pip to install 9.x.

[rosscdh]

thanks yeah got that :) appreciate the feedback on sunday mate! keep up the amazing work

[gtmanfred]

fixes for pip10 will be in 2017.7.6 and 2018.3.1 releases.

[Tanoti]

@gtmanfred Do you know when the Centos and Red Hat rpms for these will be available? This has broken our salt/pip systems.

[gtmanfred]

Both releases have been branched. Once the 2018.3.1 tests pass then there will be releases

https://jenkinsci.saltstack.com/job/2018.3.1/

Also, there is one last thing for 2017.7.6 and tornado 5.0 on python3 >=3.5

[Reiner030]

mmh... bug still exists / something else broken?

          ID: awscli-pip
    Function: pip.installed
        Name: awscli
      Result: False
     Comment: An importable Python 2 pip module is required but could not be found on your system. This usually means that the system's pip package is not installed properly.
     Started: 17:54:46.282569
    Duration: 709.242 ms
     Changes:

tested with:

# pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

# salt-call --version
salt-call 2017.7.8 (Nitrogen)
# salt-call --versions-report
Salt Version:
           Salt: 2017.7.8

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.5
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-8-amd64
         system: Linux
        version: debian 9.5

and its already setup ???

# pip install awscli
Requirement already satisfied: awscli in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: colorama<=0.3.9,>=0.2.5 in /usr/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: rsa<=3.5.0,>=3.1.2 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.12 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: PyYAML<=3.13,>=3.10 in /usr/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: botocore==1.12.39 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python2.7/dist-packages (from rsa<=3.5.0,>=3.1.2->awscli)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /usr/lib/python2.7/dist-packages (from s3transfer<0.2.0,>=0.1.12->awscli)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python2.7/dist-packages (from botocore==1.12.39->awscli)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /usr/lib/python2.7/dist-packages (from botocore==1.12.39->awscli)
Requirement already satisfied: urllib3<1.25,>=1.20; python_version == "2.7" in /usr/local/lib/python2.7/dist-packages (from botocore==1.12.39->awscli)

[MadsRC]

@ReinReiner030 I realize that I'm pretty late to the game, but I just fixed an identical problem for a client of mine (Hotfix, so I haven't digged into the actual source), but essentially it was due to an incompatible urlllib3 module being installed on the system.

Running "pip uninstall urllib3" removed version 1.24.1 from /usr/local/lib/python2.7/dist-packages, after which the Salt-Minion process was able to find version 1.19.1 which was installed in /usr/lib/python2.7/dist-packages (or run pip install urllib3==1.19.1)

A bit of salt hacking lead me to the actual exception happening when importing pip (pip_state.py). The exception was simply discarded, instead of logged - If I can find the time later, I'll see what I can do about making a PR.

[Reiner030]

Thanks @MadsRC - it seems now already fixed somehow by package update of Debian or perhaps saltstack.

compared to previous salt-call --versions-report I have now also setup:
gitdb: 2.0.0 gitpython: 2.1.1
and aws cli was already setup in some run before.

Testing it now I get also an update like this - and there is shown that I have only urllib3 version 1.19.1 installed and now got 1.24.1 downloaded (and installed?):
````

pip install awscli

Collecting awscli
Downloading https://files.pythonhosted.org/packages/0d/21/684544fb0fc52f86149740ecc8bcfef6e705d0ce3799424c9c76cf95de4f/awscli-1.16.95-py2.py3-none-any.whl (1.4MB)
100% |████████████████████████████████| 1.4MB 638kB/s
Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |████████████████████████████████| 51kB 10.8MB/s
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli)
Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
100% |████████████████████████████████| 61kB 11.3MB/s
Requirement already satisfied: PyYAML<=3.13,>=3.10 in /usr/lib/python2.7/dist-packages (from awscli)
Collecting botocore==1.12.85 (from awscli)
Downloading https://files.pythonhosted.org/packages/0b/66/38c903912870df4cdead76841041c1107252be129c75dbb533ce469572ec/botocore-1.12.85-py2.py3-none-any.whl (5.2MB)
100% |████████████████████████████████| 5.2MB 260kB/s
Requirement already satisfied: docutils>=0.10 in /usr/lib/python2.7/dist-packages (from awscli)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python2.7/dist-packages (from rsa<=3.5.0,>=3.1.2->awscli)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /usr/lib/python2.7/dist-packages (from s3transfer<0.2.0,>=0.1.12->awscli)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/lib/python2.7/dist-packages (from botocore==1.12.85->awscli)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /usr/lib/python2.7/dist-packages (from botocore==1.12.85->awscli)
Collecting urllib3<1.25,>=1.20; python_version == "2.7" (from botocore==1.12.85->awscli)
Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
100% |████████████████████████████████| 122kB 9.7MB/s
Installing collected packages: colorama, rsa, urllib3, botocore, s3transfer, awscli
Found existing installation: urllib3 1.19.1
Not uninstalling urllib3 at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: botocore 1.4.70
Not uninstalling botocore at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: s3transfer 0.1.9
Not uninstalling s3transfer at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed awscli-1.16.95 botocore-1.12.85 colorama-0.3.9 rsa-3.4.2 s3transfer-0.1.13 urllib3-1.24.1
````

And it's still running fine.
````

aws --version

aws-cli/1.16.95 Python/2.7.13 Linux/4.9.0-8-amd64 botocore/1.12.85
````

so it must be something else which depends of/to urllib or is meanwhile fixed.

[Reiner030]

Update: as @MadsRC wrote before (thx 😄 ) I had now also to uninstall urllib3 version 1.24.1 - in my case salt-cloud wasn't working:
````

salt-cloud -d test-instance

[INFO ] salt-cloud starting
[WARNING ] Missing dependency: 'requests'. The ec2 driver requires 'requests' to be installed.
[WARNING ] The cloud driver, 'ec2', configured under the 'aws_eu_central_1' cloud provider alias, could not be loaded. Please check your provider configuration files and ensure all required dependencies are installed for the 'ec2' driver.
In rare cases, this could indicate the 'ec2.get_configured_provider()' function could not be found.
Removing 'ec2' from the available providers list
No machines were found to be destroyed
````

[MichaelSaah]

I'd like to suggest that the version incompatibilities noted here (namely, that pip2 >= 10.* is incompatible w/ 2018.3.0) be added to the docs: https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.pip.html

EDIT: I see that the docs are hosted publicly; I'll make a PR.

[dhs-rec]

Which Salt version contains the fix? I'm still seeing this (on Linux) runnig Salt Minion 2018.3.4 when I try to install modules into the systems Python 2 (no virtualenv) when the Minion itself is Python 3. Windows Python 3 minons (same version) don't have the problem. Path to pip is specified on both platforms using bin_env.

[dhs-rec]

Quick test with 2018.3.5 (Py3) still triggers the error.

[dhs-rec]

Still there in 3000.3(!). However, this time I even want to install Python 3 modules on a minion running Python3 Salt after explicitely installing python3-pip AND specifying bin_env: /usr/bin/pip3.
Please reopen.