Mbed-cli: pip 10 support - no attribute get_installed_distributions

[alzix]

it seems that pip.get_installed_distributions() is no longer available in pip 10.

get_installed_distributions is reference from https://github.com/ARMmbed/mbed-cli/blob/53f8b113c9b00ecffbf6031cc4b34333b8f52cf7/mbed/mbed.py#L1452
mbed compile fails with the following error:

$ mbed compile
[mbed] ERROR: Unknown Error: 'module' object has no attribute 'get_installed_distributions'

[hug-dev]

(python -m pip install pip==9.0.3 to go back to the old version of pip that works with mbed-cli, safer in a virtual environment, until this gets fixed)

[fkjagodzinski]

I've also run into this bug today and found one thing worth sharing before finding this GitHub issue: https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program -- pip's internal API was never considered stable and should not be used here. The fix should use other Python module or call pip via the terminal.

[theotherjimmy]

I was looking for that text the other day and could not find it. Thanks @fkjagodzinski

[theotherjimmy]

Based on the link, importing pip is a bug.

[JanneKiiskila]

This needs to be fixed ASAP, because pip will nag about this! I did it yesterday (as it was nagging), but now stuff is broken and I can't compile anymore. However, downgrading pip did not work, either.

jankii01@ubuntu:~/mbed/mbed-os-example-blinky$ python -m pip install pip==9.0.3
Collecting pip==9.0.3
From cffi callback <function _verify_callback at 0x7f949f914668>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f949a705488>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f949a705d70>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f949a705ed8>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f949a6d30c8>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f949a6d3230>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)) - skipping
  Could not find a version that satisfies the requirement pip==9.0.3 (from versions: )
No matching distribution found for pip==9.0.3
jankii01@ubuntu:~/mbed/mbed-os-example-blinky$ python -m pip install pip==9.1.0
Collecting pip==9.1.0
From cffi callback <function _verify_callback at 0x7f235d140668>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f2357f31488>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f2357f31d70>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f2357f31ed8>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f2357eff0c8>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)': /simple/pip/
From cffi callback <function _verify_callback at 0x7f2357eff230>:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
  Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)) - skipping
  Could not find a version that satisfies the requirement pip==9.1.0 (from versions: )
No matching distribution found for pip==9.1.0

[JanneKiiskila]

Seems I have another package mismatch then as well... Which others seem to have as well, https://github.com/pyca/pyopenssl/issues/728.

Plus then this issue as well; https://askubuntu.com/questions/561377/pip-wont-run-throws-errors-instead

And it's not that easy fix, even though I now managed to get pip itself back to version 9.0.3, it's still failing. So, some Python lib is now somehow not right (indeed, pip --version gave me 9.0.3, but I still had some pip 10.0.0 files elsewhere and for some reason mbed cli picked those up, instead of 9.0.3 files). Plus it's NOT that easy to figure out what and exactly where.

But, in the end;

sudo su
find / -name pip
rm <anything pip related you can find>

plus clean re-install via sudo apt-get install python-pip did it, Ubuntu 16.04 seems to pull in pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7).

[jeromecoutant]

Hi
Any news about this pip issue ? :-(
Thx

[screamerbg]

Just put up a PR for this, which will fix future compatibility as well. A new version of Mbed CLI will be released to addresses this by the end of today (25th of April)

[JanneKiiskila]

And you mean this PR; https://github.com/ARMmbed/mbed-cli/pull/661

[screamerbg]

The Mbed CLI 1.5.1 is now live and addresses this issue. @alzix @JanneKiiskila @hug-dev thanks for the bug report

[JanneKiiskila]

Thanks for quick fix!

[WonderMr]

For pip 10 use from pip._internal.utils.misc import get_installed_distributions

[theotherjimmy]

@WonderMr That's pretty internal sounding. We don't want to rely on internals.

[xvart]

So had this awesome error today on ubuntu 18, and one of the many reasons I will never use python in production the thing is a never ending moving target of crazy

[richdevboston]

yes, this is pretty frustrating. Never ending moving target seems right with many of the system-level utilities like pip and conda. Pip especially.