Cibuildwheel: MacOS travis-ci build stalls

[josh146]

I noticed today that our MacOS travis-CI build using cibuildwheel has started stalling at the following point of the cibuildwheel setup:

+ pip install --upgrade setuptools
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl (573kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

This hasn't affected our Windows/Linux builds, and no changes to our devops pipeline has occurred (and even then, only superficial Python commits in our codebase were committed).

This issue happens no matter how many times we restart the build, and seems odd - this step is usually instantaneous on previous Mac cibuildwheel builds.

Since this is a command that is called by cibuildwheel, has there been a recent change that breaks this step?

[YannickJadoul]

I can reproduce this on my build as well: https://travis-ci.org/YannickJadoul/Parselmouth/jobs/483834720, but I have no clue what's going on.
Let's run a branch with verbosity flags for this command? Unless anyone has better suggestions?

[YannickJadoul]

Weird, Travis builds of cibuildwheel fail in the exact same way, and -vvv does not provide any more insight: https://travis-ci.org/YannickJadoul/cibuildwheel/jobs/483882146#L3949

[YannickJadoul]

The problem seems to be with Python 3.4, though, because the Python 2.7 does work. Is this the same for you, @josh146?

[YannickJadoul]

Installing pip<19 seems to work, so I'm guessing it's a problem with pip and Python 3.4: https://github.com/YannickJadoul/cibuildwheel/commit/f876231da4adeca96636411da067fc234733a6f5 and https://travis-ci.org/YannickJadoul/cibuildwheel/builds/483890015

Not sure what we can do about it though. This fix seems very ad-hoc and ugly?

This seems to be the same/a similar issue: https://github.com/pypa/pip/issues/6169

[bmatthieu3]

It stalls too with Python3.5 https://travis-ci.org/cds-astro/cds-healpix-python/jobs/483904177 . Seems to be a problem with pip 19.0.1

[josh146]

I'm not using either Python 2.7 _or_ Python 3.4, but I see the issue on Python 3.5

[YannickJadoul]

Ah, weird that it did work with 2.7 in those builds, then; but thanks for the confirmation.
If you need a quick patch, https://github.com/YannickJadoul/cibuildwheel/commit/f876231da4adeca96636411da067fc234733a6f5 will fix it on macOS (though the title of the commit is not correct anymore, after amending the commit); but I assume a fix will be released soon enough by pip?

EDIT: Although, maybe it is an interesting option to provide a release with this small addtion, since all builds using cibuildwheel are now failing. What do you think, @joerick?

[josh146]

From further reading, it seems like this bug occurs when upgrading a package that was previously installed via sudo, but is attempting to be upgraded without sudo. Previous versions of pip would silently fail. Maybe this is why it only affects the MacOS builds?

[joerick]

Ooh, nice insight @josh146.

Sorry I'm away on holiday so unable to help with this! If this is breaking builds now, I think a sensible approach would be a patch to install pip==18.1 for now, and then wait to see how the bug shakes out. Maybe we'll have to change which commands have sudo down the line, but let's wait and see. I think it was working before!

(Side note: I'm thinking about pinning all the versions of pip/setuptools/wheel to give better stability and build repeatability. We can upgrade them when we need features or to fix reported bugs. Thoughts?)

Sent from my phone

On 26 Jan 2019, at 07:12, Josh Izaac [e-mail] wrote:

From further reading, it seems like this bug occurs when upgrading a package that was previously installed via sudo, but is attempting to be upgraded without sudo. Previous versions of pip would silently fail. Maybe this is why it only affects the MacOS builds?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[matkoniecz]

I noticed the same problem (pip freezing on upgrading without sudo) on Ubuntu PC (16.04), so it seems that missing sudo is the problem.

[YannickJadoul]

@matkoniecz I'm still failing to reproduce the problem manually; I'm only seeing it on the online build :-(

Is there anything in particular you do to trigger the problem on Ubuntu? Is pip installed with/without sudo, locally/globally? Can I do it for any package?

I've just tried installing cibuildwheel<0.10 with sudo, then updating to the latest cibuildwheel, but that does not seem to trigger it on my Ubuntu 18.04 machine.

(Just wondering, because the issue on the pip repo does not seem to advance a lot. So I'm wondering if I could debug/help reporting more details there.)

[matkoniecz]

I get stall with pip install --upgrade pillow later sudo pip install --upgrade pillow worked fine.

I will try whatever it is reproducible.

[YannickJadoul]

Thanks for that! When I do this, I'm getting nice errors, though:

~$ pip3 install --upgrade pillow
Collecting pillow
  Using cached https://files.pythonhosted.org/packages/85/5e/e91792f198bbc5a0d7d3055ad552bc4062942d27eaf75c3e2783cf64eae5/Pillow-5.4.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: pillow
  Found existing installation: Pillow 5.3.0
    Uninstalling Pillow-5.3.0:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'ImageDraw.py'
Consider using the `--user` option or check the permissions.

[YannickJadoul]

I believe I've figured out the problem, indeed something with permissions: see https://github.com/pypa/pip/issues/6169

I think this means we should just use sudo, in our case, but I'll have a closer look tomorrow.

Meanwhile, I'd suggest using pip install git+https://github.com/YannickJadoul/cibuildwheel.git@pip-19-stalling-workaround (to install this branch using pip 18: https://github.com/YannickJadoul/cibuildwheel/tree/debug-issue-122)

@joerick Enjoy your holiday! I don't know if I have a strong opinion about pinning them. In general I'd say we haven't had a lot of problems with these packages (and if we do, the rest of the world also has them), and maybe we'd like to be in on new pip features without extra work/releases on cibuildwheel?

[matkoniecz]

@YannickJadoul Thanks for checking this! Let me know if further testing would be useful.

[josh146]

@YannickJadoul amazing! I had stumbled on another GitHub issue that mentioned it was something to do with permissions, but for the life of me couldn't find it again - hence why my second comment was so vague.

[YannickJadoul]

@josh146 Actually, your comment was the start of my debugging. Otherwise I'd have had absolutely no clue where to start. So thanks for that :-)
And in the end, it seems that adding sudo will indeed be the solution, so you were spot on.

[YannickJadoul]

@matkoniecz Thanks for that! I'll check this hypotheses tomorrow (this should be testable by changing permissions to the site-packages folder) and I'll let you know if I don't manage to reproduce the issue.

[YannickJadoul]

Update: a solution is on its way: https://github.com/pypa/pip/pull/6215
I think I will wait to see the exact outcome of that to see what we should do about the issue.

Adding sudo or --user to pip install invocations will work, but since it worked without those before, we might want to see if it will work like that after the pip fix?

[joerick]

Temporary fix in #123, releasing now...

[joerick]

Released as 0.10.1