Gevent: Install Failure on Amazon Linux

[TylerMills]

One of the requirements for AWS Lambda is packaging all your dependencies in your project directory and uploading as a zip since you have no control over the execution environment. Seems simple enough and it's working for all my other dependencies.

To start I did the following:

pip install gevent -t .

Which results in:

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmp5WhDoF/lib/python'

Going to the directory that is generated the file structure is:
/tmp/tmp5WhDoF/lib64/python

Is there something basic I'm missing for why it's installing to lib64 and not lib considering the script is generating these folders itself.

Amazon Linux is CentOS based distribution. I'm running Python 2.7 and the latest version of pip.

[jamadden]

Can you post the complete output of pip install -v? If you're using pip 8.1, you should be getting the binary manylinux wheel (unless your python is built with non-default settings, I think).

[jamadden]

FWIW, I can't replicate this problem on CentOS (which, to be fair, is where I built the wheels in the first place.)

# /opt/2.7mu/bin/pip --version
pip 8.1.1 from /opt/python/2.7.11mu/lib/python2.7/site-packages (python 2.7)
# /opt/2.7mu/bin/pip install gevent -t .
Collecting gevent
  Downloading gevent-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 825kB/s
Collecting greenlet>=0.4.9 (from gevent)
  Downloading greenlet-0.4.9.tar.gz (54kB)
    100% |████████████████████████████████| 61kB 7.2MB/s
Building wheels for collected packages: greenlet
  Running setup.py bdist_wheel for greenlet ... done
  Stored in directory: /root/.cache/pip/wheels/08/9c/52/3d8ee766d654c43b131dcc954f255f8c6270bef617da0ac3e6
Successfully built greenlet
Installing collected packages: greenlet, gevent
Successfully installed gevent greenlet

[jamadden]

I also had no problems with Ubuntu 15.04 64bit, it also picked up the manylinux wheel.

[TylerMills]

Does seem to be downloading properly:

Downloading from URL https://pypi.python.org/packages/cp27/g/gevent/gevent-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl#

What's odd is that it failed when trying to do the .whl directly as well.

The tail of the verbose logging:

Installing collected packages: greenlet, gevent
  Running setup.py install for greenlet ...     Running command /usr/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f9cadk/greenlet/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-XmW4Uh-record/install-record.txt --single-version-externally-managed --compile --home=/tmp/tmpn_spum
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-x86_64-2.7/greenlet.o
    creating build/lib.linux-x86_64-2.7
    gcc -pthread -shared build/temp.linux-x86_64-2.7/greenlet.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/greenlet.so
    running install_lib
    creating /tmp/tmpn_spum/lib64
    creating /tmp/tmpn_spum/lib64/python
    copying build/lib.linux-x86_64-2.7/greenlet.so -> /tmp/tmpn_spum/lib64/python
    running install_headers
    creating /tmp/tmpn_spum/include
    creating /tmp/tmpn_spum/include/python
    creating /tmp/tmpn_spum/include/python/greenlet
    copying greenlet.h -> /tmp/tmpn_spum/include/python/greenlet
    running install_egg_info
    running egg_info
    creating greenlet.egg-info
    writing greenlet.egg-info/PKG-INFO
    writing top-level names to greenlet.egg-info/top_level.txt
    writing dependency_links to greenlet.egg-info/dependency_links.txt
    writing manifest file 'greenlet.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'greenlet.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'greenlet.egg-info/SOURCES.txt'
    Copying greenlet.egg-info to /tmp/tmpn_spum/lib64/python/greenlet-0.4.9.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-XmW4Uh-record/install-record.txt'
done
  Removing source in /tmp/pip-build-f9cadk/greenlet
  Compiling /tmp/pip-build-f9cadk/gevent/gevent/_socket3.py ...
    File "/tmp/pip-build-f9cadk/gevent/gevent/_socket3.py", line 186
      def makefile(self, mode="r", buffering=None, *,
                                                    ^
**  SyntaxError: invalid syntax**


Successfully installed gevent-1.1.0 greenlet-0.4.9
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpn_spum/lib/python'

[TylerMills]

Just to be sure I spun up and double checked on some clean Amazon Linux machines. The first command I run after being provisioned is to install gevent and it fails as well.

[TylerMills]

For sanity I ran more explicit commands.

sudo /usr/local/bin/pip install gevent-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl --upgrade -t .

And it fails.

sudo /usr/local/bin/pip install gevent-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl --upgrade

It works. It seems specifying the output directory is causing the failure somehow.

[jamadden]

If you're installing from the binary wheel and getting this problem, then that strongly suggests it's a problem with pip or wheel or manylinux wheels. A wheel is just a ZIP file, and no code of gevent's actually runs when its installed (in fact it doesn't even contain a setup.py). I suggest opening an issue, probably with pypa/pip but maybe with pypa/manylinux. I watch both those repositories so if there's information I can add to those bug reports I will.

In the meantime, you can try installing gevent from source by using the --no-use-wheel or --no-binary options.

[jamadden]

Are there any updates on this?

[Ivoz]

If you want to install gevent somewhere particular, try using the --prefix option of pip install instead.

[jamadden]

Since this seems to be a pip-type issue (and not gevent) and there haven't been any updates in a month I'm going to close it. Feel free to open a new issue if there are install problems with gevent itself.