Poetry: Install dependencies from a local directory containing archives

[pappasam]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

Like "pip", poetry should allow applications to install dependencies from a local directory containing archives. This would enable developers to compile Python wheels in the first stage of a Docker multi-stage build then depend on only the compiled archives in another stage. If you're interested, please see my blog post outlining the reasons for this workflow.

The following configuration option could work but it's just a starting suggestion:

 [[tool.poetry.source]]
name = "local-files"
path = "instance" # I'm the relevant option :)

[[tool.poetry.source]]
name = 'default'
url = 'https://pypi.org/'

[tool.poetry.dependencies]
python = "3.7"
requests = "^2.20.0"
uwsgi = "^2.0.17"

In the above example, projects could defer to the main package index if no correctly-formatted wheels / archives are found in an instance folder (in this case, relative to pyproject.toml).

See below for a sample Dockerfile (using pip for now, hopefully poetry in the future):

###########################################
# Throwaway image with C compiler installed
FROM python:3.6-alpine as bigimage

# install the C compiler
RUN apk add --no-cache linux-headers g++

# instead of installing, create a wheel
RUN pip wheel --wheel-dir=/root/wheels uwsgi==2.0.17.1

###########################################
# Image WITHOUT C compiler but WITH uWSGI
FROM python:3.6-alpine as smallimage

COPY --from=bigimage /root/wheels /root/wheels

# Ignore the Python package index
# and look for archives in
# /root/wheels directory
RUN pip install \
      --no-index \
      --find-links=/root/wheels \
      "uwsgi>=2.0.17.1"

Thanks again for all the work on this tool; Poetry looks like an amazing project and I look forward to being able to use it as a daily driver.

[zachliu]

I have exactly the same issue :astonished:

[lballabio]

This would be great. It would solve a different use case for us (finding internal dependencies in a mono-repo project).

[hangtwenty]

A thought: “poetry install” does more than installing from wheels. I am thinking that some flag or even different subcommand could be called for, when you want to tell Poetry, “just install using these wheels, and don’t do any resolution or checking unless something is missing”

[hangtwenty]

Also

Poetry uses pip under the hood, which means it seems to respect pip env vars. PIP_FIND_LINKS and PIP_WHEEL_DIR may work with poetry install already. I need to confirm...

[richin13]

I tried running PIP_WHEEL_DIR=$PWD/wheels PIP_FIND_LINKS=file://$PWD/wheels/ poetry -vvv install

And got:

Using virtualenv: /home/rmadriz/.pyenv/versions/3.7.3/envs/poetry-wheels-gyydo
Updating dependencies
Resolving dependencies...
   1: fact: poetry-wheels is 0.1.0
   1: derived: poetry-wheels
   1: fact: poetry-wheels depends on psycopg2-binary (2.7.4)
   1: selecting poetry-wheels (0.1.0)
   1: derived: psycopg2-binary (2.7.4)
PyPI: 1 packages found for psycopg2-binary 2.7.4
PyPI: Getting info for psycopg2-binary (2.7.4) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: psycopg2-binary-2.7.4.tar.gz
PyPI: Downloading wheel: psycopg2_binary-2.7.4-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
   1: selecting psycopg2-binary (2.7.4)
   1: Version solving took 8.518 seconds.
   1: Tried 1 solutions.


Package operations: 1 install, 0 updates, 0 removals

Writing lock file

  - Installing psycopg2-binary (2.7.4)

With the following pyproject.toml/folder tree

[tool.poetry]
name = "poetry-wheels"
version = "0.1.0"
description = ""
authors = ["Ricardo Madriz <my-email>"]

[tool.poetry.dependencies]
python = "^3.7"
psycopg2-binary = "2.7.4"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

.
├── wheels
│   ├── psycopg2_binary-2.7.4-cp37-cp37m-linux_x86_64.whl
│   ├── psycopg2_binary-2.8.1-cp37-cp37m-manylinux1_x86_64.whl
│   └── uWSGI-2.0.17.1-cp37-cp37m-linux_x86_64.whl
└── pyproject.toml

am I missing something @hangtwenty ? :(

[hangtwenty]

Gotcha. Thanks for checking that, I guess that doesn't do it after all. It may make sense for Poetry to just support arbitrary args/config to pip if it does not already respect pip config in XDG_CONFIG_HOME etc (I haven't checked into that either).

You could do poetry export to a requirements, then do pip wheel and pip install based on that... Pip installs and Poetry installs can play nicely together. So that happens as a "base" for me, and the wheelhouse details are only occasionally refreshed (it's the base layer of a multilayer build, in my case). With poetry install happening after that

_This_ is the approach I've been using recently, and I'm caching the 'wheelhouse' (and virtualenv) between stages on a multistage docker build. It's working pretty well - I'd like to put together a generic version in a library or at least a gist

[richin13]

Very interesting. I'll try with poetry export and pip wheel/install and see how that goes.
Sounds like a very good idea tbh

[hangtwenty]

I'll add a note here from #76 — this works —

[tool.poetry.dependencies]
my-package = { file = "path/to/wheel.whl" }

can be useful as far as ad-hoc or known wheels (in a commercial setting, maybe it's internal shared code etc)

but doesn't address the overall "wheelhouse" pattern

[hangtwenty]

Also, I just wrote some thoughts about interface changes or workflows, that could work well with both wheelhouse-related, and cases like handling subdependency conflicts ... over here, https://github.com/sdispater/poetry/issues/697#issuecomment-481844699

Might be too peculiar of ideas for them to wind up included in Poetry itself, so I'm still considering writing an accompanying utility or wrapper

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

[boris99q]

This is still a relevant issue - It would be nice to have that feature. On big projects, when there are a lot of local packages installing via path and relative paths is disaster.

Example:
├── package a
│ ├── src
│ ├──── pyproject.toml
├── package b
│ ├── src
│ ├──── pyproject.toml

let's say that package b has dependency on package a, the following pyproject need to be given:

.
.
. 
[tool.poetry.dependencies] 
package_a = { path="../../package_a"} 
.
.
.

This is just getting awful when discussing on much bigger projects.

more niter behavior would be:

[tool.poetry.dependencies] 
package_a = "*"

and a link to the local package would be given as same as given by the pip --find-links ""