Salt: pip.installed not available with pip 9.0.2

[autophagy]

Description of Issue/Question

Attempting to run highstate against a minion with multiple pip.installed definitions results in an exception:

State 'pip.installed' was not found in SLS 'k8s'
Reason: 'pip.installed' is not available.
````

However, this only occurs if the pip version is **9.0.2**, and not if the pip version **9.0.1**. When the pip version is 9.0.1, the highstate functions normally. This occurs regardless of whether I refresh the modules.

### Setup
(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

kubernetes:
pip.installed:
- name: kubernetes==3.0
- reload_modules: True

### Steps to Reproduce Issue
Define the pip.installed definition as above, install pip 9.0.2 on the minion. Results in:

      ID: kubernetes
Function: pip.installed
    Name: kubernetes==3.0
  Result: False
 Comment: State 'pip.installed' was not found in SLS 'k8s'
          Reason: 'pip.installed' is not available.

If run on the minion:

[root@node ~]$ sudo salt-call state.highstate
[ERROR ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1412, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/site-packages/salt/states/pip_state.py", line 37, in
import pip
File "/usr/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg/pip/__init__.py", line 45, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg/pip/vcs/mercurial.py", line 9, in
from pip.download import path_to_url
File "/usr/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg/pip/download.py", line 40, in
from pip._vendor import requests, six
File "/usr/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg/pip/_vendor/requests/__init__.py", line 98, in
from . import packages
File "/usr/lib/python2.7/site-packages/pip-9.0.2-py2.7.egg/pip/_vendor/requests/packages.py", line 12, in
sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.urllib'

Downgrade pip to 9.0.1 (with ``pip install pip==9.0.1``) and the highstate is correctly applied. 

### Versions Report

Salt Version:
Salt: 2017.7.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.0
docker-py: 1.10.6
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.21.1.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core
```

No difference between master and minion versions.

[autophagy]

I believe this might be the same issue mentioned here: https://github.com/saltstack/salt/issues/44980#issuecomment-374125632

[diginc]

Attempted the locations.py patch from #44980 without luck. Downgrading to pip 9.0.1 did work though per this comment.

[ERROR   ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1411, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 36, in <module>
    import pip
  File "/usr/local/lib/python2.7/dist-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.urllib'

pip 9.0.2 from /usr/local/lib/python2.7/dist-packages (python 2.7), Ubuntu 14.04

[garethgreenaway]

Seems similar to #44980. The pip module isn't available because something isn't loading right, which definitely seems like an upstream issue.

[vdog]

https://github.com/pypa/pip/pull/5101 actually should be the right fix.

[vdog]

This looks like it may not get fixed, or it will surely get broken in pip 10, which is going to be released soon @garethgreenaway

https://github.com/pypa/pip/issues/5081

The pip maintainers are stressing that import pip should not be used by other python programs.

https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

[diginc]

Thankfully the fixed it with 9.0.3 - which is the right thing considering this wasn't an advertised breaking change or semantic versioned correctly.

I'm surprised by the best practice/recommendation to never use import pip...I'd expect the top python package provider to provide a stable python native (not subprocess shell calls) API interface.

[bryceml]

I seem to have this issue running on version 9.0.1 on Ubuntu 18.04. It also still happens if I do pip install --upgrade pip and specify the bin_env to /usr/local/bin/pip and therefore attempt to use 18.0.

Relevant state:

# makes sure python-pip and python3-pip are installed
python-pip:
  pkg.installed:
    - pkgs:
      - python-pip
      - python3-pip
# installs python2 version of tensorflow
tensorflow:
  pip.installed:
    - name: tensorflow
    - require:
      - pkg: python-pip

Some Log Lines:

[DEBUG   ] Could not LazyLoad pip.installed: 'pip.installed' is not available.
[INFO    ] Running state [tensorflow] at time 10:32:31.413069
[DEBUG   ] Could not LazyLoad pip.installed: 'pip.installed' is not available.
[ERROR   ] State 'pip.installed' was not found in SLS 'research.seppi' 
Reason: 'pip.installed' is not available.

It works fine when I run the state explicitly, but not when running from highstate.

[ribx]

@bryceml I have this problem too, with 9.0.1 on Gentoo Linux. I just updated to 2019.2. How did you solve your problem?

EDIT: it disappeared after a restart, maybe there where problems when I updated the python version from 2.7 to 3.6.

[unclesamwk]

+1