Pip-tools: Error with pip : pip-compile does not support URLs as packages

[gmocquet]

I'm using pip with pip-compile (installed this way: pip install pip-tools)

I got the following error when I run the pip-compile -v command:

pip-compile does not support URLs as packages, unless they are editable. Perhaps add -e option? (constraint was:
aldryn-django==1.8.7.0 from
https://control.aldyn.com/api/v1/apps/serve/aldryn-django/1.8.7.0/592213b1-e515-4447-8ef0-850713571a42/aldryn-django-1.8.7.0.tar.gz#egg=aldryn-django==1.8.7.0
(from -r requirements.in (line 2)))

I have tried with the -e option, but this causes another problem.

Below is an short extract of my requirements.in file:

\# <INSTALLED_ADDONS>  # Warning: text inside the INSTALLED_ADDONS tags is auto-generated. Manual changes will be overwritten.
https://control.aldryn.com/api/v1/apps/serve/aldryn-django/1.8.7.0/592213b1-e515-4447-8ef0-850713571a42/aldryn-django-1.8.7.0.tar.gz#egg=aldryn-django==1.8.7.0
...
\# </INSTALLED_ADDONS>

I'm using Docker container based on the python:2.7-slim image.

The requirements.in work well on one other similar docker container.

I don't know why on mine, pip-compile does not work...

Have you any idea?

I case of bad use of pip, I have open a ticket on stackoverflow : http://stackoverflow.com/questions/34267352/error-with-pip-pip-compile-does-not-support-urls-as-packages

[rfleschenberg]

What is the reasoning behind pip-tools not supporting URLs for non-editable installs? Is it possible to introduce a switch to override this behaviour?

For me, It would be great if pip-tools supported non-editable installs from Github repositories. I'd be happy to help implementing this, if you agree.

[nvie]

There's no reason preventing it per se, it's just not been implemented. The main thing that's tricky is to figure out how to distill the dependencies from it, and what name the package ends up with locally when you install it. You cannot derive these properties from the URL itself, you'll need to download the package, unpack it, inspect it. So there's definitely some work involved. If you feel brave, I invite you to try and add it to pip-tools, it'd be a cool feature :)

[rfleschenberg]

Thanks for the quick reply! I naively thought we could delegate most of this work to pip :) I will see if I can come up with a solution.

[miketwo]

I wonder if an easier short-term solution would be to allow it to pass-through certain lines to the final requirements.txt file. We have some packages that we build and host in-house, and while I would love to get to the point where we have an internal PyPi-like server, right now it's just a zip file at a particular URL. It would be nice to be able to tell pip-compile "Do your normal thing, but just pass-through these lines".

Maybe a --pass-through-urls option?

[rfleschenberg]

Sounds like a great idea to me. I never got around to come up with a real solution. This would solve my use-cases and should be reasonably easy to implement.

[miketwo]

It looks like there would have to be a corresponding --ignore-urls or similar on the pip-sync side, cause it can't parse them for dependencies (for the same reasons).

[rfleschenberg]

405 is related, I think?

[merwok]

“The main thing that's tricky is to figure out how to distill the dependencies from it”

The same issue exists with editable requirements (#466).

“and what name the package ends up with locally when you install it”

This could be solved with the same syntax as editable reqs: add #egg=requirement-name

[nim65s]

Here is a PoC for this. Do you think I can continue and make a PR with this kind of stuff ?

from io import BytesIO
from zipfile import ZipFile
import mock
import setuptools
try: # python 3
    from urllib.request import urlopen
except: # python 2
    from urllib import urlopen

url = 'https://github.com/django/django/archive/master.zip'
zipfile = ZipFile(BytesIO(urlopen(url).read()))

for f in zipfile.namelist():
    if f == 'setup.py' or f.endswith('/setup.py'):
        setupfile = f
        break

with mock.patch.object(setuptools, 'setup') as mock_setup:
    exec(zipfile.open(setupfile).read(), {})

_, kwargs = mock_setup.call_args

assert 'pytz' in kwargs['install_requires']

[nim65s]

Hum, no. We should use pip download.

[AndreasBackx]

Any update on this, could this perhaps be merged?

[ChiKenNeg]

Is there any news with this? seems like pip-compile-multi isn't covering URL as its based upon pip-compile..

[toejough]

I'm feeling this lack, as well. I was trying to follow a good compile/sync flow, but either they fail if I specify a patch branch for one of my dependencies, or they uninstall my dependency and reinstall the broken pypi version.