Psutil: Provide wheel for Linux and OSX [was: Install psutil without gcc?]

Created on 24 May 2016  Â·  69Comments  Â·  Source: giampaolo/psutil

This is a question: Is it possible to install psutil without gcc?

How much work would it be to port the code to ctypes (or other way to run without gcc)?

linux macos wheels

Most helpful comment

Re-opening. Goal: provide wheels for Linux and possibly OSX via travis CI.

All 69 comments

No, it is not possible. Converting the code base to ctypes for all platforms is a huge work. The main drawback I see is that you get rid of years of tested and working code and replace it with something brand new and not tested as much. Basically all the motivations I gave in https://github.com/giampaolo/psutil/issues/591#issuecomment-174220403 apply here as well. Why installing gcc is a problem for you?

Thank you for your feedback.

Installing without gcc is a restriction from the one providing the environment.

I don't like this restriction and I think it is useless, too.

I can work around it:

  • install it via rpm
  • install it via wheel (never tried before).

If you see a different (or easier) way, please tell me. My question was answered. Thank you. Closing this.

On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry).
What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip.

If the wheel folks want wheels to be successful, then they could provide a build server. Leaving building wheels to the developers is a waste of time and energy.

And non-repeatable builds will be the result if every body builds his wheels in a different build environment...

I asked on the distutils mailinglist what they think about it: https://mail.python.org/pipermail/distutils-sig/2016-May/028994.html

With https://github.com/pypa/manylinux being quite extensively used, can this be reopened ?

In practical terms what would it take to take advantage of manylinux? Is it something which can be integrated and automatized via travis or it would require me to set up a local environment consisting of docker + manylinux + whatever?

As of now I upload the tar.gz from my linux laptop and the windows exes/wheels are hosted on appveyor (I download them locally then I upload them on PYPI). How would this kind of workflow gonna change with manylinux?

Have a look at https://github.com/pypa/python-manylinux-demo
and https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh

Basically, you call that script that builds the wheels for multiple python versions and use auditwheel to package .so libraries, inside a specially prepared docker container (quay.io/pypa/manylinux1_XXX).

It can be achieved locally, or you can make travis call those scripts and upload the wheels.

You can also look at psycopg2 : they have a separate git repo with the tools for building each new release : https://github.com/psycopg/psycopg2-wheels

Not willing to do it myself locally on every release. Do you have an example for Travis?

Oh, psycopg has it, nevermind.

I'm also interested in having psutil python wheels on linux: any updates on using manylinux to create the wheels?

The best way to do this would be to have Travis build the wheels on every
commit and save them somewhere as "artifacts" then download them locally
and finally upload them on pypi with twine. We currently use this strategy
for windows via appveyor.com.
The same can also be done for Linux and possibly OS X.
The goal is to figure out how to do this via .travis.yml.
I currently don't have time to look into this so if somebody wants to give
it a try.... I am all for it.

On Wed, 6 Sep 2017 at 18:52, Bruno Binet notifications@github.com wrote:

I'm also interested in having psutil python wheels on linux: any updates
on using manylinux to create the wheels?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-327448042,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAplLC1ZrRcyBjqDBkuYxtIPkFZBrS_Hks5sfnl9gaJpZM4IlKUi
.

>

Giampaolo - http://grodola.blogspot.com

Re-opening. Goal: provide wheels for Linux and possibly OSX via travis CI.

@giampaolo I whish you good luck with "provide wheels for Linux and possibly OSX".

Do you already have a roadmap to reach the goal?

It's something which needs to be investigated. A quick Google search brought me here:
https://github.com/joerick/cibuildwheel
...so it appears it's definitively possible.

AFAICT whereas AppVeyor gives you space to store the wheels on their machines Travis won't, so that is the main stopper.

I am subscribed to the distutils mailing list. I asked the experts if what they think: https://mail.python.org/pipermail/distutils-sig/2017-September/031506.html

Sure, it's doable. You will need to take advantage of Travis's new "Build stages" feature. Do your testing in one build stage and wheel building/deployment in the next. You should be able to build a suitable job matrix with relatively little work. Make sure you use the manylinux1 Docker images to minimize the necessary effort.

IIUC, storing packages in a github repo is fine for low-volume things; but artifacts uploaded as /releases are differently CDN'd?

Guys - I'm happy to give you access to the Rackspace CDN for you to upload to. Further upload to pypi can be fairly well automated as well : https://github.com/MacPython/cython-wheels#quickstart

@giampaolo Why is there need for wheel storage space with Travis? Why not just build them and upload to PyPI directly?

If that turns out to be possible then I am all for it. That should happen
"on request" when we tag a new version though. I am not interested in
having intermediate builds for each commit.

On Sat, 9 Sep 2017 at 19:18, Alex Grönholm notifications@github.com wrote:

@giampaolo https://github.com/giampaolo Why is there need for wheel
storage space with Travis? Why not just build them and upload to PyPI
directly?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328270900,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAplLPE4d352Bv-j247bm71nlPvPgzBsks5sgnQagaJpZM4IlKUi
.

>

Giampaolo - http://grodola.blogspot.com

If you're fine with the wheel building happening on every build but the PyPI upload only on tags, then yes, that is totally doable.

The way that we do it over at numpy / scipy / matplotib etc, is have a separate repo for doing the wheel builds, which we trigger at release time. This uploads to Rackspace, allowing us to check that the builds are all good, and have passed, before uploading from Rackspace to PyPI. In practice, that has been a useful extra step to avoid messing up on the PyPI release. Several projects (such as numpy, scipy, matplotlib) also do daily wheel builds on the master branch, from the same repository, to make sure the wheel build is still working.

e.g. https://github.com/MacPython/numpy-wheels

What if the build on tag fails because one of test suites fail due to a
false positive? Perhaps the wheel build should happen regardless of the
test results.

On Sat, 9 Sep 2017 at 19:33, Alex Grönholm notifications@github.com wrote:

If you're fine with the wheel building happening on every build but the
PyPI upload only on tags, then yes, that is totally doable.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328271579,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAplLDYH8sNdvNCE9nyeBvFpzFCPLb18ks5sgnengaJpZM4IlKUi
.

>

Giampaolo - http://grodola.blogspot.com

I don't understand what you mean by "fail due to a false positive".
@matthew-brett why is uploading to Rackspace necessary to check that the builds are fine? Can that not be done on Travis?

The wheel building repos all build wheels, test, and, if the tests pass for a particular matrix entry, automatically upload to a Rackspace CDN. This has the advantage that, if any of your tests run fail, you can fix, and run the build again, before releasing to PyPI. For example, lets say some Travis step times out randomly (as happens reasonably often). With the Rackspace step, you can just restart the failed build to make sure you've got all the wheels built, before releasing. For other systems, you'll either have to rebuild all the wheels, or you'll end up uploading some but not all wheels to PyPI, potentially breaking someone's install when the release doesn't have a particular wheel, and starts compiling, when the user wasn't expecting that.

@matthew-brett that would not happen using Build Stages. If any of the wheels fail to build, none would get deployed and the build would fail.

Right - that's what I meant above when I was saying about 'rebuild all the wheels'. I believe you would have to do that if you used build stages. Happy to be corrected.

Yes, the wheels either have to be rebuilt or you have to store them somewhere between jobs. Travis has example code that uses S3 for storing build artifacts.

There is one problem with automating this on Travis, namely Windows wheels. As Appveyor is required to produce those, it's going to be tough to orchestrate wheel building between these two CIs. That would definitely require intermediate storage.
I have a project of my own that will need to produce binary wheels for multiple platforms. Unfortunately the code isn't quite ready – otherwise I would make that into a practical example.

https://github.com/matthew-brett/multibuild/blob/master/appveyor.yml

On Saturday, September 9, 2017, Alex Grönholm notifications@github.com
wrote:

There is one problem with automating this on Travis, namely Windows
wheels. As Appveyor is required to produce those, it's going to be tough to
orchestrate wheel building between these two CIs. That would definitely
require intermediate storage.
I have a project of my own that will need to produce binary wheels for
multiple platforms. Unfortunately the code isn't quite ready – otherwise I
would make that into a practical example.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328298667,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGy61ZyU5s8fOMBL60qbidYikMkiYaks5sgudFgaJpZM4IlKUi
.

@westurner You'll have to clarify – why did you post a link to your appveyor config?

I posted a link to the appveyor.yml from the multibuild project that I
shared earlier in the thread.

Is this not what you need? A standard set of CI wheel building scripts for
Linux and MacOS on Travis, and AppVeyor for Windows?

On Saturday, September 9, 2017, Alex Grönholm notifications@github.com
wrote:

@westurner https://github.com/westurner You'll have to clarify – why
did you post a link to your appveyor config?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328308695,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGyzAzBKfFOcuJgJpEzRAoJVtke0Bhks5sgxnwgaJpZM4IlKUi
.

We already know how to build wheels on all platforms. The question at hand was implementing continuous delivery in a way that would make sure that either all platform specific wheels are uploaded or none at all.

On Sun, 10 Sep 2017 at 03:30, Alex Grönholm notifications@github.com
wrote:

There is one problem with automating this on Travis, namely Windows
wheels. As Appveyor is required to produce those, it's going to be tough to
orchestrate wheel building between these two CIs.

I am fine with how windows wheels are currently handled and automated.

Giampaolo - http://grodola.blogspot.com

So, transaction-like atomicity?

IDK whether warehouse supports such a feature?

  • All of the uploads are public.
  • Each uploaded file is "published" as soon as the upload and db
    transaction are complete.

https://github.com/pypa/warehouse

https://en.wikipedia.org/wiki/Atomicity_(database_systems)

On Saturday, September 9, 2017, Alex Grönholm notifications@github.com
wrote:

We already know how to build wheels on all platforms. The question at hand
was implementing continuous delivery in a way that would make sure that
either all platform specific wheels are uploaded or none at all.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328310245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGy4ne392ygMBiUx7Hv4Z87vDFODeNks5sgyOxgaJpZM4IlKUi
.

Wheelhouse is as transactional as the underlying VCS.

On Saturday, September 9, 2017, Wes Turner wes.turner@gmail.com wrote:

So, transaction-like atomicity?

IDK whether warehouse supports such a feature?

  • All of the uploads are public.
  • Each uploaded file is "published" as soon as the upload and db
    transaction are complete.

https://github.com/pypa/warehouse

https://en.wikipedia.org/wiki/Atomicity_(database_systems)

On Saturday, September 9, 2017, Alex Grönholm <[email protected]

We already know how to build wheels on all platforms. The question at
hand was implementing continuous delivery in a way that would make sure
that either all platform specific wheels are uploaded or none at all.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824#issuecomment-328310245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADGy4ne392ygMBiUx7Hv4Z87vDFODeNks5sgyOxgaJpZM4IlKUi
.

What underlying VCS?

Just to make things clear, "building wheels on git tag" (aka on each release) doesn't have to be a strict requirement. Whatever the solution we end up with it just has to be practical. For instance, the current solution used for Windows works very well for me: Appveyor generates the wheels on git push and stores them, I download them locally then upload them on PYPI. In practical terms I can do this with 2 make commands:

make win-download-exes          # download from appveyor
make win-upload-exes            # upload on PYPI

I wouldn't mind to have to do the same for OSX and Linux.

Actually, I had completely forgotten I had started up a multibuild repo for psutil, a couple of years ago. I tool the liberty of finishing it up - it's now building manylinux wheels, and will build OSX wheels when travis-ci OSX builds come back up (they're down at the moment).

https://github.com/MacPython/psutil
https://travis-ci.org/MacPython/psutil

Unfortunately all the linux wheels are generating a segfault in the memory test:

https://travis-ci.org/MacPython/psutil/builds/273798729

No segfault on OSX, but I don't think the scripts are being installed:

https://travis-ci.org/MacPython/psutil/jobs/273798748#L716

Also, I think the tests are assuming they are running in the development tree, because one test is running the setup.py script:

https://travis-ci.org/MacPython/psutil/jobs/273798748#L708

Maybe skip that one if setup.py not present?

@giampaolo Ok so how about this? You set up an (free tier) S3 account and have Travis build and upload the wheels there on every tag. Travis has a deployment provider for this.

Sorry - I should say that I have authorized the psutil-wheels repo to upload to Rackspace CDN, which it will do on successful build / test. At the moment none of the matrix entries pass.

Various projects also run their wheel builds daily, and upload test wheels to Rackspace for people to test with pip install --pre. That is more or less trivial to set up with the current repo.

I invited @giampaolo as an admin to the psutil-wheels repo - do y'all want me to add anyone else? @giampaolo - if you accept the invitation, you're an admin, so you can add whoever you like.

Unfortunately all the linux wheels are generating a segfault in the memory test:
https://travis-ci.org/MacPython/psutil/builds/273798729

Ouch! I don't know why I can't see this happening on my travis builds but it looks like a high priority bug.

Anyway, what you did in https://github.com/MacPython/psutil looks promising (and thanks for the invitation). Since this discussion is already pretty long I suggest we move it on https://github.com/MacPython/psutil bug tracker.

OK, here it is: https://github.com/MacPython/psutil/issues/1.
Please let's continue there and use this ticket for discussing other solutions.

@giampaolo do you have a clue to proceed now?

Did the advices you got from people from the distutils mailing list help you?

@giampaolo Shouldn't manywheels project solve this, aka building via that should make this work on Linux/Mac too?

manylinux wheels should work on many Linux distros, but IDK about Mac?

https://github.com/pypa/manylinux/blob/master/README.rst#example links to
an example of building manylinux wheels with Travis CI

-
https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh

https://github.com/joerick/cibuildwheel :

  • Builds manylinux, macOS and Windows (32 and 64bit) wheels using Travis
    CI, Appveyor, and CircleCI
  • Bundles shared library dependencies on Linux and macOS through
    auditwheel and delocate
  • Runs the library test suite against the wheel-installed version of your
    library

On Friday, May 17, 2019, Bernát Gábor notifications@github.com wrote:

@giampaolo https://github.com/giampaolo Shouldn't manywheels project
solve this, aka building via that should make this work on Linux/Mac too?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/psutil/issues/824?email_source=notifications&email_token=AAAMNS3XJOPKJAJVCS4HTFDPVZ43BA5CNFSM4CEUUURKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVUJHNQ#issuecomment-493392822,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAMNS55CTTNNQP5MLRMYQDPVZ43BANCNFSM4CEUUURA
.

For me it would be enough to support linux. I think there are just too many other OSs (solaris, aix, ....).

We can use Azure Pipelines to build all.

How?

The setup at https://github.com/MacPython/psutil is nearly complete - no? It builds macOS wheels too.

I think you'll find there's a pretty big demand for macOS wheels.

I personally find Azure pipelines pretty ugly to set up, compared to Travis and Appveyor - and you're nearly done, with https://github.com/MacPython/psutil - see https://github.com/MacPython/psutil/issues/1 - and you'd have to solve those problems on Azure too.

In my (short) experience with it, Azure Pipelines is far more elegant in design compared to Travis and Appveyor. It is, however, quite "green" as well so they may not have covered all their bases yet (and it has some minor issues too). But if I needed to run my CI on all three platforms, I would definitely use Azure Pipelines (which I have).

I've tried Azure too - but I gave up after a while. Regardless, the barrier here is not the platform - you can use the same machinery on Travis and Azure - the barrier is getting the tests fixed over at https://github.com/MacPython/psutil/issues/1

Does azure let you store the wheels somewhere so you can retrieve them via wget or something?

Yeah, pipelines can generate artifacts that are attached to build. You can even define release stages afterwards to upload them to PyPi if the test stage succeeds (perhaps on git tag only/user button push). It's easy to get started and setup given we use tox via https://github.com/tox-dev/azure-pipelines-template (see example https://github.com/tox-dev/tox/blob/master/azure-pipelines.yml - example run https://dev.azure.com/toxdev/tox/_build/results?buildId=1766).

@matthew-brett I would definitely recommend using the tox template to set up if you're using tox. Makes the config a lot simpler/easier.

@matthew-brett

The setup at https://github.com/MacPython/psutil is nearly complete - no? It builds macOS wheels too.

If I'm not mistaken the blocker here is https://github.com/giampaolo/psutil/issues/1126#issuecomment-468668759.
Long story short: in order to use manylinux we need an old CentOS version depending on this C routine which cannot rely on CPU_ALLOC macro (only most recent Linux distros have it):
https://github.com/giampaolo/psutil/blob/fbece8e45991e8cc070bfd5e9efcd4974227abcd/psutil/_psutil_linux.c#L350
...and that causes a segfault. So in order to proceed we should fix #1126 first.

cPython, which also implements this functionality (os.sched_getaffinity), doesn't have this problem as it relies on CPU_ALLOC:
https://github.com/python/cpython/blob/b49858b4b7b4c9d85ef6946ad020f83e4fa1caa7/Modules/posixmodule.c#L6421

So fixing #1126 is controversial. I guess we may do a runtime check instead of the pre-processor / #ifdef check, but am not sure how (there's a lot of that in _psutil_windows.c but I never did it in _psutil_linux.c).

Also, there are other #ifdef checks in _psutil_linux.c which may not compile on old CentOS, meaning the resulting wheel will have some disabled functionality also on recent Linuxes. Basically this is why this issue (producing wheels) got stuck.

I may be able to try looking at this sometime later during this year. In the meantime, for posterity, any doc/tutorial on how to setup the old CentOS distro (docker?) is appreciated.

@giampaolo - did you try the instructions at https://github.com/MacPython/psutil/issues/1#issuecomment-347019442 ? Did they work? Let me know, I'm happy to extend / update them.

Long story short: in order to use manylinux we have to use an old CentOS version

The Manylinux2010 platform has just started rolling out a few weeks ago with an updated toolchain. That could potentially make the issue go away without intervention in this codebase, I guess.

Manylinux2010 uses CentOS 6. Does CentOS 6 have the macro?

@giampaolo Would you like help to automate deploy of wheels on PyPI via cibuildwheel ? If yes, with which CI service?

I've set up a GitHub workflow to build wheels for Linux/macOS:

Run ls -alh dist/
total 4.3M
drwxr-xr-x 2 runner docker 4.0K Mar 15 06:24 .
drwxr-xr-x 3 runner docker 4.0K Mar 15 06:24 ..
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 257K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 263K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 257K Mar 15 06:24 psutil-5.7.0-cp27-cp27mu-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 263K Mar 15 06:24 psutil-5.7.0-cp27-cp27mu-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 261K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 269K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 261K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 270K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 266K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 273K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp38-cp38-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 265K Mar 15 06:24 psutil-5.7.0-cp38-cp38-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 275K Mar 15 06:24 psutil-5.7.0-cp38-cp38-manylinux2010_x86_64.whl

@huntzhan It would be awesome if you could upload those wheels to https://github.com/private-pypi/psutil/releases
Cheers!

@rostrovsky
You can find the wheels in https://pywharf.github.io/pywharf-pkg-repo/psutil/

It would be awesome to have an official manylinux wheel for psutil on PyPi. The environment where my app runs abstracts building of the docker image and doesn't allow the app to participate in the docker image build step; the only customization to the docker build that my app is allowed to do is to provide a requirements.txt file for python packages, and the apps are not allowed to apt-get anything at all.

Was this page helpful?
0 / 5 - 0 ratings