Poetry: --find-links equivalent

[severinsimmler]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Using:

• Python 3.7.3
• Poetry 1.0.0b1

Feature Request

Because the CPU version of PyTorch is not on PyPI, they recommend installing it this way:

pip install torch==1.2.0+cpu --find-links https://download.pytorch.org/whl/torch_stable.html

However, there is no way to do this with Poetry?

Pointing directly to the wheel:

poetry add https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl

results in

Updating dependencies
Resolving dependencies... (2.0s)


Package operations: 1 install, 0 updates, 0 removals

  - Installing torch (1.2.0+cpu https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl)

[EnvCommandError]
Command ['/home/simmler/.cache/pypoetry/virtualenvs/epss-WeFenOJ_-py3.7/bin/python', '-m', 'pip', 'install', '--no-deps', 'torch==1.2.0+cpu'] errored with the following return code 1, and output: 
Collecting torch==1.2.0+cpu
  Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0)
No matching distribution found for torch==1.2.0+cpu

And adding https://download.pytorch.org/whl/torch_stable.html as a repository to the pyproject.toml does not work, because it (the PyTorch wheel index page) is not PEP conform.

[cc-a]

Hi. I was thinking of having a go at this for Hacktoberfest.

The approach I was thinking of:

• Implement a variation of the LegacyRepository class
• Extend the expected fields for the source toml section to add a flag that specifies this type of simple html repository
• Implement the relevant logic in poetry.py to determine the correct repository class to use

Sound about right?

[severinsimmler]

Sounds like a plan :+1:

[woodlandhunter]

@cc-a a --find-links that is equivalent to pip's would also accept file paths (a directory). See pip docs here. We could really use this. It's also mentioned in these issues:

558

1289

Your comment doesn't mention this scenario, would --find-links=DIR be covered by what you are working on?

[cc-a]

I'll have a crack at it, probably in a second PR. In principle the same logic should be required just working with file paths instead of urls.

[golyalpha]

Right now the workaround is defining remote file dependencies (in pytorch's case, platform specific).
This works for me on the current release:

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl --platform win32

Unfortunately, this will limit you to only one platform, as you can't have the same dependency for multiple platforms with different file links.

[germn]

After installing torch with

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl

each time I add any package poetry does following:

- Updating torch (1.3.1 -> 1.3.1 https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl)

It doesn't break anything, but takes some time.

Anyone knows a way to avoid "updating"?

[golyalpha]

Not really, it's a workaround for a current limitation in Poetry, but until that limitation is fixed, you could probably limit the impact by using a local cache (like devpi) @germn

[allen-chin]

@germn Something like below should work fine for your case.
pyspark = {file = "pyspark-3.0.0.dev0.tar.gz"}

[golyalpha]

@allen-chin The sideeffect of that is continual reinstallations of the package. Just because the package is local, doesn't mean Poetry won't reinstall it.

[cpbotha]

I created a valid simple index that can be used with index-url and as a tool.poetry.source for all of the pytorch.org official packages. You can find the new index here: https://vxlabs.com/pypi/

I massaged the official PyTorch find-urls file to look more like a PEP 503 compatible simple index, but it's still pointing to all of the official packages at pytorch.org.

It's currently working like a charm in my poetry setup.

[aviramha]

@cpbotha Plan on adding 1.5.0 soon?
BTW Any news on this issue?

[cpbotha]

Currently swamped, have not had time to get back to this and probably won't for the coming weeks.

There's also the issue on pytorch's side to make available the right sort of index for their packages, see https://github.com/pytorch/pytorch/issues/25639 , but no-one has had time yet to dive in.

[timorkal]

@cpbotha great job, but I have an issue when I am using your source with poetry when trying to lock.
pyproject.toml:
torch = "1.3.1+cpu"
torchvision = "0.4.2+cpu"

[SolverProblemError]
Because torchvision (0.4.2+cpu) depends on torch (1.3.1)
and package depends on torch (1.3.1+cpu), torchvision is forbidden.
So, because package on torchvision (0.4.2+cpu), version solving failed.

[lironmo]

same issue here.
@cpbotha i tried to add your private pip repository with no success:
poetry config repositories.torch-mirror https://vxlabs.com/pypi

[daniel-j-h]

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

[severinsimmler]

Most of us only have this problem because of PyTorch, right? To be honest, since I opened this issue I no longer see the problem here with Poetry, but with PyTorch. Why the hell isn't a big project like PyTorch published on a PEP-conform index? I opened an issue (https://github.com/pytorch/pytorch/issues/26340) some time ago -- so far without any concrete reaction.

I actually like the strict policy of Poetry. PEP exists for a reason.

[kousu]

@severinsimmler I'm having the same issue (https://github.com/neuropoly/spinalcordtoolbox/issues/1526#issuecomment-674330466). Thanks a lot for trying to get the ball rolling with pytorch!

I agree poetry should stick close to the core pypa specs, and they want to kill --find-links and dependency_links in favour of https://www.python.org/dev/peps/pep-0508/#environment-markers (https://github.com/pypa/pip/issues/6162, https://github.com/pypa/pip/issues/4187) so you shouldn't try to reintroduce that feature here. But PEP 508's URL mode doesn't cover wheels with compiled platform-specific versions :/ so we're stuck.

But if pytorch got themselves together and published to pypi this would be a non-issue.

[Korijn]

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

See https://github.com/python-poetry/poetry/issues/2543 for the separate issue

For what it's worth, I don't think poetry needs this --find-links feature. Rather, the pytorch team needs to adopt proper packaging practices..

[golyalpha]

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

[Korijn]

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload _all_ versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

Fair enough, I have not digged deep enough to uncover this root cause. Apologies for my tone.

I do think that it's odd that hosting your own index is apparently too hard for such a project. Aren't there tons of such tools available to do that?

[golyalpha]

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

[Korijn]

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

Yes, well, I guess I was also not clear enough. One can also host a static HTML site that meets the PEP 503 format. That also allows it to be included as an index in pyproject.toml. See https://github.com/pytorch/pytorch/issues/25639 and https://github.com/pytorch/pytorch/issues/39779

[golyalpha]

Well, if the guys at PyTorch manage to figure it out, that's good. Poetry still probably should support the feature from pip.

[RafalSkolasinski]

Hey guys, I read this thread (and the https://github.com/pytorch/pytorch/issues/26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

[Korijn]

Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

If it's _just_ torch (and not also vision), you can if you set up a PEP 503 compatible index with the package on it. There's a public one hacked together by @cpbotha you can use: https://vxlabs.com/pypi/

[sinoroc]

I will add this as a potential workaround:

• https://github.com/uranusjr/simpleindex/issues/2

It does not seem to be a valid workaround, yet. It might be that there is an issue with _URL redirections_ in _poetry_, so that the workaround fails for now.

[espdev]

Any news on the PR?

For example, I need to use numpy+mkl on Windows 10 2004 Update because of this issue. I can download whl from gohlke web page only manually. And it would be nice to use --find-links and the local wheelhouse directory for searching wheel-packages instead of remote wheelhouse:

Something like this:

[tool.poetry.dependencies]
python = "^3.7"
numpy = { version = "^1.19.4+mkl", markers = "sys_platform == 'win32'" }
pandas = "^1.1.4"

[[tool.poetry.source]]
name = "numpy"
url = "wheels/"  # relative path to local directory with wheel files
find_links = true