Pip: Issue with "pip download --platform" semantics

[robertf224]

• Pip version: 9.0.1
• Python version: 2.7.13
• Operating system: OSX 10.11.6

I want to download python packages on my Mac with pip to bundle with a product, which will be transferred to a Linux x86_64 machine without external internet access where they will eventually be "pip install"'d.

I'm having trouble doing this with "pip download". Ideally, I would run "pip download --platform linux_x86_64" (also side note, docs for platform options are basically nonexistent, had to grep the source code) and it would download wheels for that platform when possible, and otherwise source distributions.

However, using "--platform" returns this "ERROR: --only-binary=:all: must be set", and when I add that setting, I can't download packages that don't have linux wheels (because it won't default to using source distributions). I don't understand the rationale behind this restriction.

[robertf224]

Found a temporary workaround by hijacking the internal platform string function:

import pip
pip.pep425tags.get_platform = lambda: 'linux_x86_64'
pip.main(['download', '<package>'])

[mwilliamson]

Is the restriction because it's intentional behaviour, or because the implementation would need some work to make it function correctly? If the latter, I'd be willing to to take a look.

[xavfernandez]

I don't understand the rationale behind this restriction.

wheel's metadata can easily be parsed: an environment markers like flake8; sys_platform == 'win32' can easily be interpreted by pip to understand that flake8 must only be downloaded if --platform=win32 was set (or if running on a Windows).

However, with sdist and their interpreted setup.py all bets are off: pip would have to interpret code like https://github.com/pyca/cryptography/blob/c1f916918103d4a5436dda8972fc4a21bf05fb12/setup.py#L44-L58 or https://github.com/onitu/onitu/blob/33c928a55de8f79098aa5931efb8d1ee17a6fbb6/drivers/local_storage/setup.py#L6-L9

This could be (partially) done, by monkey-patching sys before executing setup.py but this is likely to fail in some way so we decided to only implement the always correct and easy part.

[mwilliamson]

Is interpreting setup.py necessary solely for installing dependencies? In which case, could sdists be downloaded if --no-deps was also set?

[robertf224]

btw to solve my problem ended up writing a script that takes a requirements.txt file and uses the the docker image from here (https://github.com/pypa/manylinux) to pip download manylinux wheels and creates ones that don't already exist on pypi

[xavfernandez]

In which case, could sdists be downloaded if --no-deps was also set?

Yes, this could be allowed.

[jonathancross]

btw to solve my problem ended up writing a script

@robertf224 Can you provide a link the script here?

[robertf224]

I wrote it at work so I'll start the process of open-sourcing it (assuming I have to go through the official channels, not sure because it's 2 scripts and a readme) and I'll link here when that's done

[cmcginty]

+1 This seems like a useful feature when trying to collect wheels for a multi-platform package. Ideally there would be support for multiple --platform options and then all the available wheels matching the platforms would be downloaded.

[einarbmag]

I'm having this problem and I imagine many people trying to vendor an app targeting a different system must be having it too. Anyone working on a fix? I would have thought that it's an easy fix... "if binary not available: download source"? I'm trying to implement @robertf224's workaround but I can't get it to target a different python version.

[robertf224]

Hey all, sorry for the delay, here's the script I wrote (just for targeting linux in general using the new manylinux spec):
https://github.com/robertf224/pip-download-manylinux

[einarbmag]

Thanks @robertf224, that's really helpful!

[johnthagen]

Say you want to prefer universal wheels, and selectively pull down source distributions for select packages you know you'll need to build (for example because you are targeting PyPy3 and the package maintainers do not provide wheels for that platform).

Ideally you would write something like (tested on pip 19.2.3):

# Download only Pillow as source, and universal wheels for everything else.
$ pip download --only-binary=:all: --no-binary=Pillow --implementation=py Django Pillow
ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

Removing the --implementation=py works, but now non-universal wheels could be downloaded instead of universal wheels.

Using --no-binary=:all: has the drawback that setup.py's are run at build time, and sometimes do surprising things. Also, some package maintainers only upload .whl distributions to PyPI.

[pfmoore]

@johnthagen what's wrong with

pip download --no-binary=:all: Pillow
pip download --only-binary=:all: --implementation=py Django

or something similar? Why is it essential for this to work as a single pip invocation?

[johnthagen]

@pfmoore Thanks for the potential work around!

In my case, I'm actually doing this as part of a large requirements.txt file (the example above was designed to be minimal). In that case there may be arbitrarily many packages with dependencies. Without a single pip invocation, I wouldn't think that it could properly do dependency resolution in all situations. If both invocations had the same sub-dependency it would get downloaded twice (and potentially the wrong version if there two invocations had different constraints).