Salt: 'pip.installed' is not available when running highstate

[angeloudy]

Description of Issue/Question

state file:

  pip.installed:
    - name: haproxyctl >= 0.5
    - find_links: https://github.com/neurogeek/haproxyctl/archive/
    - reload_modules: True

This runs OK if I do salt minion1 state.apply haproxy
But if I run salt minion1 state.highstate, I got an error

Function: pip.installed
        Name: haproxyctl >= 0.5
      Result: False
     Comment: State 'pip.installed' was not found in SLS 'haproxy'
              Reason: 'pip.installed' is not available.
     Changes:

there's also some errors in /var/log/salt/minion:

2017-12-14 11:50:25,478 [salt.state       ][ERROR   ][58945] Error encountered during module reload. Modules were not reloaded.
2017-12-14 11:50:41,427 [salt.loader      ][ERROR   ][58945] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/salt/loader.py", line 1405, in _load_module
    mod = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/salt/states/pip_state.py", line 36, in <module>
    import pip
  File "/usr/local/lib/python3.6/site-packages/pip/__init__.py", line 26, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/usr/local/lib/python3.6/site-packages/pip/utils/__init__.py", line 23, in <module>
    from pip.locations import (
  File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 103, in <module>
    bin_user = os.path.join(user_site, 'bin')
  File "/usr/local/lib/python3.6/posixpath.py", line 78, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Setup

OS: FreeBSD 11.1
python: 3.6

Steps to Reproduce Issue

upgrade salt from py27-salt to py36-salt

Versions Report

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: 1.7.0
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.3 (default, Dec  8 2017, 18:14:50)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 16.0.3
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:
         locale: US-ASCII
        machine: amd64
        release: 11.1-RELEASE-p1
         system: FreeBSD
        version: Not Installed

[garethgreenaway]

@angeloudy Thanks for the report. Are you able to confirm if the issues you're seeing are happening only on FreeBSD or happen on other systems too.

[angeloudy]

@garethgreenaway we are only using FreeBSD.

[Ch3LL]

I was able to replicate this on Arch. One thing I noticed was I could only replicate it when the state was a part of a highstate. If I call the state directly with salt-call --local state.sls <sls file> it worked just fine.

To note this is also broken on oxygen.

[angeloudy]

I think this issue is part of pip.
check this out https://github.com/pypa/pip/pull/4442

[Ch3LL]

I can verify that when applying this patch I can now run the state just fine :) Seems its an upstream issue.

@angeloudy are you seeing the same ?

[d-shi]

This just started happening to us as well:

2018-03-17 04:39:24,454 [salt.loader      ][ERROR   ][22855] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1411, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 36, in <module>
    import pip
  File "/usr/local/lib/python2.7/dist-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.packages.ssl_match_hostname.ssl'

Tried with both 2017.7.1 and 2017.7.4. Happens both in highstate and when I run salt minion state.apply pip-state.

Version report:

Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: 2.5.1
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-109-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[tokiwinter]

We're seeing similar issues, which seemingly started occurring overnight:

2018-03-17 15:27:56,237 [salt.loader      ][ERROR   ][28869] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/salt/loader.py", line 1412, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/opt/virtualenvs/salt/lib/python2.7/site-packages/salt/states/pip_state.py", line 36, in <module>
    import pip
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/opt/virtualenvs/salt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'

I have managed to workaround this in our particularly simple use case by refactoring our use of pip.installed to use virtualenv.managed instead.

[syphernl]

I'm seeing the same as @d-shi on our Ubuntu 16.04/Salt 2017.7.4 nodes.
It started occurring after pip was upgraded to 9.0.2. Seems to work fine with 9.0.1 though.

[diginc]

Also encountered this on pip 9.0.2, stphrnl's downgrade work around resolved it.

[Ch3LL]

can any of you confirm this patch fixes the issue as well? https://github.com/pypa/pip/pull/4442

this appears to be an upstream issue .

[diginc]

My error seems to be slightly different to the locations.py error the original poster had. The patch didn't seem to help which makes sense since there is no locations.py in my stacktrace.

Toki and my issue probably belong in #46611 - I put a comment there with my stacktrace and this information.

[Ch3LL]

ahh yes you are right seems your issue is more like that one. As I confirmed that the upstream fix fixes the original issue I will go ahead and close this and it seems the issue in #46611 is the right one for the rest of you. Thanks!