cryptography 2.1.4 installs on Amazon Linux but isn't available
This is reproducible with a stock Amazon Linux 2017.09 instance after having installed the system-rpm-config rpm (which is installed as part of running yum groupinstall 'Development Tools'). Install cryptography, it shows no error on installation. Do a pip list and it doesn't show up. Attempt to use it with a import cryptography and you get an import error.
Steps to reproduce
[ec2-user@ip-172-31-4-199 ~]$ rpm -qi system-release
Name : system-release
Version : 2017.09
Release : 0.1
Architecture: noarch
Install Date: Mon 15 Jan 2018 06:42:56 PM UTC
Group : System Environment/Base
Size : 27776
License : GPLv2
Signature : RSA/SHA256, Thu 02 Nov 2017 08:35:33 PM UTC, Key ID bcb4a85b21c0f39f
Source RPM : system-release-2017.09-0.1.src.rpm
Build Date : Wed 01 Nov 2017 07:45:55 PM UTC
Build Host : build-64011.build
Relocations : (not relocatable)
Packager : Amazon.com, Inc. <http://aws.amazon.com>
Vendor : Amazon.com
Summary : Amazon Linux AMI release file
Description :
Amazon Linux AMI release files
[ec2-user@ip-172-31-4-199 ~]$ sudo yum -y install system-rpm-config
Logout of the instance and log back in. This will cause the environment variable setting added to /etc/profile.d/python-install-layout.sh by the system-rpm-config RPM to be loaded in the sessions environment.
[ec2-user@ip-172-31-4-199 ~]$ printenv PYTHON_INSTALL_LAYOUT
amzn
[ec2-user@ip-172-31-4-199 ~]$ virtualenv test
New python executable in /home/ec2-user/test/bin/python2.7
Also creating executable in /home/ec2-user/test/bin/python
Installing setuptools, pip, wheel...done.
[ec2-user@ip-172-31-4-199 ~]$ test/bin/pip -V
pip 9.0.1 from /home/ec2-user/test/local/lib/python2.7/dist-packages (python 2.7)
[ec2-user@ip-172-31-4-199 ~]$ test/bin/python -V
Python 2.7.12
[ec2-user@ip-172-31-4-199 ~]$ test/bin/pip install cryptography
Collecting cryptography
Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
100% |████████████████████████████████| 2.2MB 525kB/s
Collecting asn1crypto>=0.21.0 (from cryptography)
Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |████████████████████████████████| 102kB 7.6MB/s
Collecting six>=1.4.1 (from cryptography)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting enum34; python_version < "3" (from cryptography)
Downloading enum34-1.1.6-py2-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography)
Downloading cffi-1.11.4-cp27-cp27mu-manylinux1_x86_64.whl (406kB)
100% |████████████████████████████████| 409kB 2.8MB/s
Collecting ipaddress; python_version < "3" (from cryptography)
Downloading ipaddress-1.0.19.tar.gz
Collecting idna>=2.1 (from cryptography)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 10.1MB/s
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography)
Downloading pycparser-2.18.tar.gz (245kB)
100% |████████████████████████████████| 256kB 4.0MB/s
Building wheels for collected packages: ipaddress, pycparser
Running setup.py bdist_wheel for ipaddress ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
Running setup.py bdist_wheel for pycparser ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built ipaddress pycparser
Installing collected packages: asn1crypto, six, enum34, pycparser, cffi, ipaddress, idna, cryptography
Successfully installed asn1crypto-0.24.0 cffi-1.11.4 cryptography-2.1.4 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 pycparser-2.18 six-1.11.0
[ec2-user@ip-172-31-4-199 ~]$ test/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
enum34 (1.1.6)
idna (2.6)
ipaddress (1.0.19)
pip (9.0.1)
pycparser (2.18)
setuptools (38.4.0)
six (1.11.0)
wheel (0.30.0)
[ec2-user@ip-172-31-4-199 ~]$ test/bin/python -c "import cryptography"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cryptography
gene1wood
All 17 comments
Can you say more about how you're reproducing this, I couldn't using the docker amazonlinux image:
root@jenkins-docker-01:~# docker run --rm -it amazonlinux
bash-4.2# rpm -qi system-release
Name : system-release
Version : 2017.09
Release : 0.0
Architecture: noarch
Install Date: Sun Oct 1 17:57:22 2017
Group : System Environment/Base
Size : 27776
License : GPLv2
Signature : RSA/SHA256, Thu Aug 17 00:48:33 2017, Key ID bcb4a85b21c0f39f
Source RPM : system-release-2017.09-0.0.src.rpm
Build Date : Wed Aug 16 21:05:34 2017
Build Host : build-60008.build
Relocations : (not relocatable)
Packager : Amazon.com, Inc. <http://aws.amazon.com>
Vendor : Amazon.com
Summary : Amazon Linux AMI release file
Description :
Amazon Linux AMI release files
bash-4.2# yum install python27-virtualenv
Loaded plugins: ovl, priorities
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
(1/5): amzn-main/latest/group_gz | 4.4 kB 00:00:00
(2/5): amzn-updates/latest/group_gz | 4.4 kB 00:00:00
(3/5): amzn-updates/latest/updateinfo | 452 kB 00:00:00
(4/5): amzn-updates/latest/primary_db | 686 kB 00:00:01
(5/5): amzn-main/latest/primary_db | 3.8 MB 00:00:01
Resolving Dependencies
--> Running transaction check
---> Package python27-virtualenv.noarch 0:15.1.0-1.14.amzn1 will be installed
--> Processing Dependency: python27-setuptools for package: python27-virtualenv-15.1.0-1.14.amzn1.noarch
--> Processing Dependency: python27-pip for package: python27-virtualenv-15.1.0-1.14.amzn1.noarch
--> Processing Dependency: python27-devel for package: python27-virtualenv-15.1.0-1.14.amzn1.noarch
--> Running transaction check
---> Package python27-devel.x86_64 0:2.7.12-2.121.amzn1 will be installed
--> Processing Dependency: python27(x86-64) = 2.7.12-2.121.amzn1 for package: python27-devel-2.7.12-2.121.amzn1.x86_64
--> Processing Dependency: python27(x86-64) = 2.7.12-2.121.amzn1 for package: python27-devel-2.7.12-2.121.amzn1.x86_64
--> Processing Dependency: python27(x86-64) = 2.7.12-2.121.amzn1 for package: python27-devel-2.7.12-2.121.amzn1.x86_64
---> Package python27-pip.noarch 0:9.0.1-1.24.amzn1 will be installed
---> Package python27-setuptools.noarch 0:36.2.7-1.33.amzn1 will be installed
--> Processing Dependency: python27-backports-ssl_match_hostname for package: python27-setuptools-36.2.7-1.33.amzn1.noarch
--> Running transaction check
---> Package python27.x86_64 0:2.7.12-2.120.amzn1 will be updated
---> Package python27.x86_64 0:2.7.12-2.121.amzn1 will be an update
--> Processing Dependency: python27-libs(x86-64) = 2.7.12-2.121.amzn1 for package: python27-2.7.12-2.121.amzn1.x86_64
---> Package python27-backports-ssl_match_hostname.noarch 0:3.4.0.2-1.12.amzn1 will be installed
--> Processing Dependency: python27-backports for package: python27-backports-ssl_match_hostname-3.4.0.2-1.12.amzn1.noarch
--> Running transaction check
---> Package python27-backports.x86_64 0:1.0-3.14.amzn1 will be installed
---> Package python27-libs.x86_64 0:2.7.12-2.120.amzn1 will be updated
---> Package python27-libs.x86_64 0:2.7.12-2.121.amzn1 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================
Installing:
python27-virtualenv noarch 15.1.0-1.14.amzn1 amzn-main 2.1 M
Installing for dependencies:
python27-backports x86_64 1.0-3.14.amzn1 amzn-main 5.0 k
python27-backports-ssl_match_hostname noarch 3.4.0.2-1.12.amzn1 amzn-main 12 k
python27-devel x86_64 2.7.12-2.121.amzn1 amzn-updates 524 k
python27-pip noarch 9.0.1-1.24.amzn1 amzn-main 2.1 M
python27-setuptools noarch 36.2.7-1.33.amzn1 amzn-main 672 k
Updating for dependencies:
python27 x86_64 2.7.12-2.121.amzn1 amzn-updates 102 k
python27-libs x86_64 2.7.12-2.121.amzn1 amzn-updates 6.8 M
Transaction Summary
==================================================================================================================================
Install 1 Package (+5 Dependent packages)
Upgrade ( 2 Dependent packages)
Total download size: 12 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): python27-backports-1.0-3.14.amzn1.x86_64.rpm | 5.0 kB 00:00:00
(2/8): python27-backports-ssl_match_hostname-3.4.0.2-1.12.amzn1.noarch.rpm | 12 kB 00:00:00
(3/8): python27-2.7.12-2.121.amzn1.x86_64.rpm | 102 kB 00:00:00
(4/8): python27-devel-2.7.12-2.121.amzn1.x86_64.rpm | 524 kB 00:00:00
(5/8): python27-setuptools-36.2.7-1.33.amzn1.noarch.rpm | 672 kB 00:00:00
(6/8): python27-pip-9.0.1-1.24.amzn1.noarch.rpm | 2.1 MB 00:00:01
(7/8): python27-virtualenv-15.1.0-1.14.amzn1.noarch.rpm | 2.1 MB 00:00:00
(8/8): python27-libs-2.7.12-2.121.amzn1.x86_64.rpm | 6.8 MB 00:00:01
----------------------------------------------------------------------------------------------------------------------------------
Total 6.7 MB/s | 12 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : python27-2.7.12-2.121.amzn1.x86_64 1/10
Updating : python27-libs-2.7.12-2.121.amzn1.x86_64 2/10
Installing : python27-devel-2.7.12-2.121.amzn1.x86_64 3/10
Installing : python27-backports-1.0-3.14.amzn1.x86_64 4/10
Installing : python27-backports-ssl_match_hostname-3.4.0.2-1.12.amzn1.noarch 5/10
Installing : python27-setuptools-36.2.7-1.33.amzn1.noarch 6/10
Installing : python27-pip-9.0.1-1.24.amzn1.noarch 7/10
Installing : python27-virtualenv-15.1.0-1.14.amzn1.noarch 8/10
Cleanup : python27-2.7.12-2.120.amzn1.x86_64 9/10
Cleanup : python27-libs-2.7.12-2.120.amzn1.x86_64 10/10
Verifying : python27-devel-2.7.12-2.121.amzn1.x86_64 1/10
Verifying : python27-libs-2.7.12-2.121.amzn1.x86_64 2/10
Verifying : python27-backports-ssl_match_hostname-3.4.0.2-1.12.amzn1.noarch 3/10
Verifying : python27-setuptools-36.2.7-1.33.amzn1.noarch 4/10
Verifying : python27-virtualenv-15.1.0-1.14.amzn1.noarch 5/10
Verifying : python27-backports-1.0-3.14.amzn1.x86_64 6/10
Verifying : python27-pip-9.0.1-1.24.amzn1.noarch 7/10
Verifying : python27-2.7.12-2.121.amzn1.x86_64 8/10
Verifying : python27-2.7.12-2.120.amzn1.x86_64 9/10
Verifying : python27-libs-2.7.12-2.120.amzn1.x86_64 10/10
Installed:
python27-virtualenv.noarch 0:15.1.0-1.14.amzn1
Dependency Installed:
python27-backports.x86_64 0:1.0-3.14.amzn1 python27-backports-ssl_match_hostname.noarch 0:3.4.0.2-1.12.amzn1
python27-devel.x86_64 0:2.7.12-2.121.amzn1 python27-pip.noarch 0:9.0.1-1.24.amzn1
python27-setuptools.noarch 0:36.2.7-1.33.amzn1
Dependency Updated:
python27.x86_64 0:2.7.12-2.121.amzn1 python27-libs.x86_64 0:2.7.12-2.121.amzn1
Complete!
bash-4.2# virtualenv t
New python executable in /t/bin/python2.7
Also creating executable in /t/bin/python
Installing setuptools, pip, wheel...done.
bash-4.2# ./t/bin/pip install cryptography
Collecting cryptography
Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
100% |################################| 2.2MB 461kB/s
Collecting asn1crypto>=0.21.0 (from cryptography)
Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |################################| 102kB 9.3MB/s
Collecting six>=1.4.1 (from cryptography)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting enum34; python_version < "3" (from cryptography)
Downloading enum34-1.1.6-py2-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography)
Downloading cffi-1.11.4-cp27-cp27mu-manylinux1_x86_64.whl (406kB)
100% |################################| 409kB 2.7MB/s
Collecting ipaddress; python_version < "3" (from cryptography)
Downloading ipaddress-1.0.19.tar.gz
Collecting idna>=2.1 (from cryptography)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |################################| 61kB 8.9MB/s
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography)
Downloading pycparser-2.18.tar.gz (245kB)
100% |################################| 256kB 4.2MB/s
Building wheels for collected packages: ipaddress, pycparser
Running setup.py bdist_wheel for ipaddress ... done
Stored in directory: /root/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
Running setup.py bdist_wheel for pycparser ... done
Stored in directory: /root/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built ipaddress pycparser
Installing collected packages: asn1crypto, six, enum34, pycparser, cffi, ipaddress, idna, cryptography
Successfully installed asn1crypto-0.24.0 cffi-1.11.4 cryptography-2.1.4 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 pycparser-2.18 six-1.11.0
bash-4.2# ./t/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
cffi (1.11.4)
cryptography (2.1.4)
enum34 (1.1.6)
idna (2.6)
ipaddress (1.0.19)
pip (9.0.1)
pycparser (2.18)
setuptools (38.4.0)
six (1.11.0)
wheel (0.30.0)
bash-4.2# ./t/bin/python -c "import cryptography"
bash-4.2#
alex
on 2 Feb 2018
It doesn't look like you ever actually activated the virtualenv, so when you run the virtualenv's python, you are not running in the virtualenv environment.
Rather than:
$ virtualenv test
$ test/bin/pip -V
$ test/bin/python -V
$ test/bin/pip install cryptography
$ test/bin/pip list
$ test/bin/python -c "import cryptography"
Try:
$ virtualenv test
$ source test/bin/activate
(test) $ pip -V
(test) $ python -V
(test) $ pip install cryptography
(test) $ pip list
(test) $ python -c "import cryptography"
(test) $ deactivate
$
mattsb42-aws
on 2 Feb 2018
Directly running the python binary from the virtualenv is equivalent to activating the virtualenv and running python from the PATH.
alex
on 2 Feb 2018
@alex Sure. I'm just spinning up an AWS instance with the current stock Amazon Linux (2017.09). Then the exact console output above is what I'm doing and what I get.
I'll spin up another stock instance and run the same commands to confirm it happens on all amazon linuxes.
gene1wood
on 2 Feb 2018
Hmmm, a freshly booted lightsail with Amazon Linux isn't reproducing this for me:
/p/t/imagemagick ❯❯❯ ssh [email protected]
The authenticity of host '54.236.37.90 (54.236.37.90)' can't be established.
ECDSA key fingerprint is SHA256:Km+WlK5rE0V1Exl3cqi176r/0XCtDjp6T/pq5j7L4g4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.236.37.90' (ECDSA) to the list of known hosts.
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
1 package(s) needed for security, out of 1 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-26-6-32 ~]$ yum ^C
[ec2-user@ip-172-26-6-32 ~]$ python
Python 2.7.12 (default, Nov 2 2017, 19:20:38)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[ec2-user@ip-172-26-6-32 ~]$ virtualenv t
New python executable in /home/ec2-user/t/bin/python2.7
Also creating executable in /home/ec2-user/t/bin/python
Installing setuptools, pip, wheel...done.
[ec2-user@ip-172-26-6-32 ~]$ ./t/bin/pip install cryptography
Collecting cryptography
Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
100% |████████████████████████████████| 2.2MB 533kB/s
Collecting asn1crypto>=0.21.0 (from cryptography)
Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |████████████████████████████████| 102kB 9.8MB/s
Collecting six>=1.4.1 (from cryptography)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting enum34; python_version < "3" (from cryptography)
Downloading enum34-1.1.6-py2-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography)
Downloading cffi-1.11.4-cp27-cp27mu-manylinux1_x86_64.whl (406kB)
100% |████████████████████████████████| 409kB 2.9MB/s
Collecting ipaddress; python_version < "3" (from cryptography)
Downloading ipaddress-1.0.19.tar.gz
Collecting idna>=2.1 (from cryptography)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 9.8MB/s
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography)
Downloading pycparser-2.18.tar.gz (245kB)
100% |████████████████████████████████| 256kB 4.5MB/s
Building wheels for collected packages: ipaddress, pycparser
Running setup.py bdist_wheel for ipaddress ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
Running setup.py bdist_wheel for pycparser ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built ipaddress pycparser
Installing collected packages: asn1crypto, six, enum34, pycparser, cffi, ipaddress, idna, cryptography
Successfully installed asn1crypto-0.24.0 cffi-1.11.4 cryptography-2.1.4 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 pycparser-2.18 six-1.11.0
[ec2-user@ip-172-26-6-32 ~]$ ./t/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
cffi (1.11.4)
cryptography (2.1.4)
enum34 (1.1.6)
idna (2.6)
ipaddress (1.0.19)
pip (9.0.1)
pycparser (2.18)
setuptools (38.4.0)
six (1.11.0)
wheel (0.30.0)
[ec2-user@ip-172-26-6-32 ~]$ ./t/bin/python -c "import cryptography"
Are there any pending yum updates or anything like that?
alex
on 2 Feb 2018
I just spun another amazon linux up and ran the same commands and it worked. However on this instance, I've tried a number of times this same sequence in different virtual envs, clearing the pip cache, and it doesn't work.
I'll see if I can find out how these two brand new instances differ.
[ec2-user@ip-172-31-6-167 ~]$ virtualenv test
New python executable in /home/ec2-user/test/bin/python2.7
Also creating executable in /home/ec2-user/test/bin/python
Installing setuptools, pip, wheel...done.
[ec2-user@ip-172-31-6-167 ~]$ test/bin/pip -V
pip 9.0.1 from /home/ec2-user/test/local/lib/python2.7/site-packages (python 2.7)
[ec2-user@ip-172-31-6-167 ~]$ test/bin/python -V
Python 2.7.12
[ec2-user@ip-172-31-6-167 ~]$ test/bin/pip install cryptography
Collecting cryptography
Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
100% |████████████████████████████████| 2.2MB 524kB/s
Collecting asn1crypto>=0.21.0 (from cryptography)
Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |████████████████████████████████| 102kB 8.7MB/s
Collecting six>=1.4.1 (from cryptography)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting enum34; python_version < "3" (from cryptography)
Downloading enum34-1.1.6-py2-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography)
Downloading cffi-1.11.4-cp27-cp27mu-manylinux1_x86_64.whl (406kB)
100% |████████████████████████████████| 409kB 2.7MB/s
Collecting ipaddress; python_version < "3" (from cryptography)
Downloading ipaddress-1.0.19.tar.gz
Collecting idna>=2.1 (from cryptography)
Downloading idna-2.6-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 10.0MB/s
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography)
Downloading pycparser-2.18.tar.gz (245kB)
100% |████████████████████████████████| 256kB 4.3MB/s
Building wheels for collected packages: ipaddress, pycparser
Running setup.py bdist_wheel for ipaddress ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
Running setup.py bdist_wheel for pycparser ... done
Stored in directory: /home/ec2-user/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built ipaddress pycparser
Installing collected packages: asn1crypto, six, enum34, pycparser, cffi, ipaddress, idna, cryptography
Successfully installed asn1crypto-0.24.0 cffi-1.11.4 cryptography-2.1.4 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 pycparser-2.18 six-1.11.0
[ec2-user@ip-172-31-6-167 ~]$ test/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
cffi (1.11.4)
cryptography (2.1.4)
enum34 (1.1.6)
idna (2.6)
ipaddress (1.0.19)
pip (9.0.1)
pycparser (2.18)
setuptools (38.4.0)
six (1.11.0)
wheel (0.30.0)
[ec2-user@ip-172-31-6-167 ~]$ test/bin/python -c "import cryptography"
[ec2-user@ip-172-31-6-167 ~]$
Are there any pending yum updates or anything like that?
On both instances I did no yum updates.
On the one that exhibits the problem I did install these packages before attempting to install cryptography
sudo yum -y groupinstall 'Development Tools'
sudo yum -y install libyaml-devel libffi-devel openssl-devel
Cool, glad it's at least working.
If you're able to figure out what the deal is, please let us know! I've never seen these symptoms before.
alex
on 2 Feb 2018
@alex I've added your keys to the instance if you want to poke around on it (the one that exhibits the problem)
ssh [email protected]
Hmm, I could have sworn virtualenv activate messed with the PYTHONPATH..
FWIW, I was able to replicate this on one of my AL instances, but not on a fresh instance.
mattsb42-aws
on 2 Feb 2018
So, what I'm seeing is that cryptography is ending up installed to t/local/lib64/python2.7/dist-packages/ (given a virtualenv at t/), however that path is not in sys.path:
>>> sys.path
['', '/home/ec2-user/t/local/lib64/python2.7/site-packages', '/home/ec2-user/t/local/lib/python2.7/site-packages', '/home/ec2-user/t/lib64/python2.7', '/home/ec2-user/t/lib/python2.7', '/home/ec2-user/t/lib64/python2.7/site-packages', '/home/ec2-user/t/lib/python2.7/site-packages', '/home/ec2-user/t/lib64/python2.7/lib-dynload', '/home/ec2-user/t/local/lib/python2.7/dist-packages', '/home/ec2-user/t/local/lib/python2.7/dist-packages', '/home/ec2-user/t/lib/python2.7/dist-packages', '/usr/lib64/python2.7', '/usr/lib/python2.7']
@dstufft do you recognize these symptoms?
alex
on 2 Feb 2018
Ok, I think this is probably the same issue as: https://github.com/pypa/pip/issues/4464
I'm still confused as to why a fresh issue doesn't have the issue though.
alex
on 2 Feb 2018
I was able to get my second instance which yesterday wasn't producing the problem to begin producing it. I spun up a third instance and was comparing how in any way the brand new working third instance which did not exhibit the problem differed from my original instance which did exhibit the problem.
I validated that all installed rpms were identical. I then compared environments.
I found on the instance that exhibited the problem there was one difference in the environment, there was an environment variable called PYTHON_INSTALL_LAYOUT set in the problematic instance but not in the working instance
PYTHON_INSTALL_LAYOUT=amzn
On the working instance, when I set this environment variable and tried again to install cryptography, the problem manifested.
I've not yet figured out what step I'm going through that causes a new instance that doesn't have this environment variable set to change to having it set.
I see reference to this problem in certbot/certbot#2823 certbot/certbot#2872 and mention that this may relate to the lib64 aspect that @alex called out.
So here is the confirmed steps to reproduce this every time in a brand new Amazon Linux instance
virtualenv test
export PYTHON_INSTALL_LAYOUT=amzn
test/bin/pip install cryptography
test/bin/pip list
test/bin/python -c 'import cryptography'
This seems pretty squarely like either a pip bug or an Amazon Linux bug, not a cryptography bug.
Luckily, @dstufft works on all three of those things so I'm hoping he might be able to give this some attention.
alex
on 2 Feb 2018
And here's the reason for somewhat fuzzy determinism in this happening
This environment variable, "PYTHON_INSTALL_LAYOUT=amzn" is set from this file
/etc/profile.d/python-install-layout.sh
Which is not part of a brand new Amazon Linux AMI, but gets added, in my case, when I run
sudo yum groupinstall 'Development Tools'
Doing this installs the rpm system-rpm-config which installs /etc/profile.d/python-install-layout.sh
The fuzzyness is that until the user logs out and back in, this new environment variable doesn't get set because the user is using the same session as they had before. This means that
Case 1
If the user launches an Amazon Linux instance, installs the system-rpm-config rpm, and does a pip install cryptography everything will work fine.
Case 2
If however the user logs out of that session and then reconnects to the system, then tries to do the exact same thing they did a moment ago, pip install cryptography, it will fail because by logging in they set this new environment variable when they run /etc/profile.d/python-install-layout.sh
gene1wood
on 2 Feb 2018
Closing since this isn't a cryptography issue (but great work on isolating this!)
reaperhulk
on 5 Mar 2018
Directly running the python binary from the virtualenv is equivalent to activating the virtualenv and running python from the PATH.
no it's not :(
I know many believe that and it would make sense if it did. See https://github.com/pypa/virtualenv/issues/906#issuecomment-244394963
muelli
on 9 Apr 2018
Related issues
chitoge
·
26Comments
glyph
·
46Comments
cromefire
·
23Comments
Naddiseo
·
45Comments
abhishek-ram
·
37Comments
Most helpful comment
On both instances I did no yum updates.
On the one that exhibits the problem I did install these packages before attempting to install cryptography