Psycopg2: manylinux wheels

[philtay]

It would be nice to have the new manylinux wheels available on PyPI.

https://www.python.org/dev/peps/pep-0513/
https://github.com/pypa/manylinux

[dvarrazzo]

Psycopg is not distributed in binary form on Linux: it is only distributed in source form. Packaging is not our responsibility.

Furthermore, skimming the pep, I read "[A manylinux1 wheel] should depend only on an extremely limited set of external shared libraries" and this cannot be the case for psycopg which by default wraps the libpq.

[philtay]

Packaging is not our responsibility.

You are already publishing the win32 and win_amd64 wheels. This is just another wheel, but for Linux.

I read "[A manylinux1 wheel] should depend only on an extremely limited set of external shared libraries"

Read it again! :) The sentence you quoted means another thing. You can include all the libraries you want in the Linux wheels, including libpq. Have a look at what numpy has recently done. Inside their Linux wheel there is bundled the OpenBLAS library for instance. You can do the same with libpq.

[dvarrazzo]

Compiling stuff for windows requires an infrastructure and a knowledge that windows users don't have. Compiling it for Linux requires a couple of system packages. On my servers it is an extra line in an Ansible script.

And then what, because some volunteer have chosen to help windows users than I am _compelled_ to provide you whatever you want? Seriously?

Numpy and other projects have different contributors, each one providing the features they want, or the features they are paid/rewarded for their development. My personal effort is oriented to what I personally need, or what my employer may need, or I feel rewarding to have. if some volunteer decides to support other binary formats (meaning: provides the hardware or the VMs to create such images, and sets up an automated way to create and deploy them on PyPI), them I'm not against merging their contribution to the project.

Your attitude that I have some form of obligation towards what you think _it would be nice to have_, on top of what you are already given, is misplaced.

[FlogFr]

This question should be addressed to the people maintaining the referenced package. Not here.
Thanks

[philtay]

Ehi, relax. I never said you have any obligation and I don't know from where you infer that you should provide me whatever I want. Personally I can live pretty well without this feature, I just thought that skipping the compilation step could be valuable to many users (and consequently to psycopg itself). I'm not the only one with this idea given the fact that a new wheel standard format has been introduced just for this purpose. Kindly note that to create these wheels you don't need any VM or hardware, the Python community has already provided all the needed tools. Travis CI can do the rest.

[kevin-brown]

if some volunteer decides to support other binary formats (meaning: provides the hardware or the VMs to create such images, and sets up an automated way to create and deploy them on PyPI), them I'm not against merging their contribution to the project.

Does this offer still stand? We're looking into what packages are still missing Linux wheels at the moment, and we may be able to offer some resources to get a Linux package (likely using manylinux) available.

I have a hunch as to how this might be able to be done using Travis CI, so it can be automated using their infrastructure.

[dvarrazzo]

Issue re-opened. Looking at the numpy distribution I think we may start distributing psycopg/manywheel.

I will use the holiday break to work on psycopg 2.7 release and possibly deploy it this way.

[dvarrazzo]

@philtay, @aRkadeFR, @kevin-brown: test manylinux wheels available on testpypi: please test with:

pip install -i https://testpypi.python.org/pypi psycopg2

and let me know if they work ok or if there is any issue.

[FlogFr]

@dvarrazzo All good on my personal machine (debian jessie + python2.7). psycopg2 version 2.7.dev0 . Thanks

[NotSqrt]

I just tested them too with a 9.3 cluster, python 2.7 and 3.5, no particular problem.

[dvarrazzo]

I'd be interested in having the build performed on Travis and downloaded when needed, or to create a new release. Here there is some discussion about how to do that. https://bintray.com/ could be useful. If there is a way to send an encrypted pkey I'd just sftp the files to initd.org.

[dvarrazzo]

Notes to encrypt files: https://docs.travis-ci.com/user/encrypting-files/

[NotSqrt]

I see this in the numpy package:
https://github.com/numpy/numpy/blob/master/.travis.yml#L23

[NotSqrt]

The rest of their manylinux wheel building machinery seems to be in https://github.com/MacPython/numpy-wheels

[kevin-brown]

Verified that the psycopg2 wheels work under Ubuntu/Python 2.7.

Thanks for getting this set up!

[dvarrazzo]

@NotSqrt cool, so OSX has delocate, which would work like auditwheel repair to create a package with dynamic libs included.

[NotSqrt]

Looks like it !
I have never worked with these tools, so I can't really help any further.

[dvarrazzo]

Manylinux wheels are now built on Travis CI (https://github.com/psycopg/psycopg2-wheels) and uploaded to our server. Psycopg 2.7 will be available in this format.

[NotSqrt]

Great !

When can we expect the next release ?
Thanks