Poetry: Support for local path dependencies during development only

[hozn]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.

Here's the background / what I'm trying to accomplish:

• I have a large modular python project. It is generally a main application with dependencies on other components that we maintain -- shared libraries, essentially.
• Adding a new feature in the main application often involves changing one or more of these shared libraries.
• During development, I'd like to be able to experiment with different code in those shared libraries and integration test it with my main application before actually writing tests for / committing / tagging / pushing / etc. the changes to those shared libraries.

Here's how I would do this with setuptools / pip:

• Ensure that my main application (my-app e.g.) has a requirements.txt file that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, if my-app will use an updated my-lib1 version 4.1 and my-lib2 version 1.12.3, I might have the following inmy-app/requirements.txt:
  my-lib1 ^4.1 my-lib2 ^1.12.3
• Setup a virtualenv for my top-level application and activate it.
• Change into each shared library directory and run pip install -r requirements.txt && python setup.py develop (Sometimes pip install -e . would also work, but the behavior was a little more consistent actually using python setup.py develop. Argh, the mess that is python packaging !?!)
• Finally I'd run pip install -r requirements.txt in my main application; because I'd already done python setup.py develop I'd have installed (symlinked) in the right versions to satisfy for my-lib1 and my-lib2.
• After finishing development, I'd package up the shared libraries and deploy them to our registry; then when my-app was pushed to build server, it'd find the versions it needed.

This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.

I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATION

If I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHS

But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.

So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!

[ankostis]

The solution to this existing problem requires some kind tof "redirection": the "path dependencies" contain the exact path, while another "type of dependency" would allow each user to configure ("redirect to") its exact path.

Actually, not even a new type of dependency is needed, as long as the user can override and re-direct any regular dependency to its own path.

[hozn]

@ankostis , I agree that allowing the user to override -- e.g. with commandline arguments -- during the install step would meet my requirements here. Something like:

poetry install --path-dep ../path/to/my-lib1 --path-dep ../path/to/my-lib2

It would probably still be required that the dependency at that local path matched the version specified in my pyproject.toml file.

[avanzal]

I'm currently facing the same challenges as described.

Basically, I have a number of library packages and a number of app packages using those libraries, and would like to:

1. Use poetry to install dependencies during local development (where app and library packages are on disk)
2. Use poetry to build dists for the library packages

The problem seems to be a disconnect between path dependencies, poetry install, and poetry build:

1. While developing, it's pretty handy to use path dependencies (instant updates without build+publish, edits across multiple libraries at once, etc); but
2. If I specify any path dependencies in pyproject, I can't do the required poetry build (doesn't allow path dependencies)
3. If I specify only non-path dependencies in pyproject, poetry install / poetry update will fetch from the configured repository instead of local dirs, which is counter to point 1

I've tried (and failed) on some workarounds:

• I can't use poetry add --path, because the dependency already specified in pyproject. Also it's pretty painful to do this for each library.
• I tried adding the libraries as non-path dependencies to tools.poetry.dependencies and as path dependencies to tools.poetry.dev-dependencies, hoping poetry build and poetry install would pick from these separately, but this doesn't work (poetry build still fails)

Found some partial / potential workarounds, though none of these cover all requirements:

• Use poetry run pip install -e ... for local dependencies, and write some scripting automate this somewhat as it would be pretty painful to do by hand for all the connected libraries.
• Use only non-path dependencies in the library pyprojects, and add all local libraries (including sub-dependencies) as path dependencies to the app's pyproject. Running poetry install on the app uses local disk via the specified path dependencies. However, in my use case each library package has its own test suite, so they also need to do a poetry install, and since they're now non-path dependencies this doesn't work with local development.
• Each developer could run their own local PyPi server and publish their dev packages to it. Really not ideal though, especially if you're changing the libraries a lot.

And also:

• It would be really useful if path dependencies also supported version = "..." (eg. xyz-log = {path="../xyz-log", version="^1.0.0"}), would help with branched code etc.

I _think_ this problem would be solved if there was a configurable package resolver - eg. you could configure it to first look up the package name in a particular directory and if it exists then install it as path dependency, else go look for a git repo <base-url>/<package-name>.git, else go look in PyPi, etc - whatever the user configures. This is on similar lines to the command line parameter idea of poetry install --path-dep ... from the comments

[bersace]

I'm facing this issue while maintaining two poetry-managed libraries. I did the following:

~/src/dep$ poetry install
~/src/dep$ cd ../main/
~/src/main$ poetry install
~/src/main$ echo $HOME/src/dep >> $VIRTUAL_ENV/lib/python3.6/site-packages/easy-install.pth
~/src/main$ python -c 'import dep; print(dep.__file__)'
/home/.../src/deps/dep.py
~/src/main$

That's not the best solution, but still fire and forget.

[pbryan]

I too am in a similar situation. I'm substituting a dependency (e.g. x = "^1.3") with a path dependency (x = { path = "../x") in pyproject.toml, which works. Ideally, there would be a method of overriding any dependency (or dev-dependency) in a "poetry-installed" project without modifying pyproject.toml (maybe in some kind of "override" file).

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hozn]

Um, yes this is still relevant! Unless something has changed this is still just as broken/incomplete as it was when the issue was created.

[hozn]

@sdispater , could we get feedback on this issue?

[freddrake]

I'm using path dependencies similar to some of the others who've followed up is this issue already, but am interesting in having those dependencies built and installed into the virtual env just like anything else, not as "editable" installs using .egg-link metadata. My goal is to be able to use the virtual env as input to a process that builds a platform package (.deb in my case).

Being able to control the type of installation separately from the type of source would be quite handy.

[niander]

Any update on this?

[idmitrievsky]

Can anyone weigh in on what are the obstacles for this functionality, so we could begin working on eliminating them?

[abn]

@hozn Your use case sounds very similar to one I have porposed at #2270. It would be good to get more eyes on that one and possibly feedback as a way to solve this.

Additionally, there is also #755 which might be another solution once implemented. However, that will only work if you are okay working with VCS copies. This would also imply that you can simply use something like git-submodules, which might not be ideal.

@idmitrievsky at the moment, I do not think there is a specific solution proposed here. If #2270 would be a solution that would be awesome, it is something I do want to work on soon once we have some feedback.

If poetry were to allow for dev dependencies to override specified dependencies, this would sort of break the locking mechanism itself. A possible compromise might be to allow for a local configuration that allows for the pre-pending of certain paths in the project's virtualenv without using it to determine dependencies or update lockfile etc. If that is something that is desirable, I do think that will sit well inside a plugin once the plugin system is completed.

[idmitrievsky]

@abn thank you for your response! I will keep an eye on the plugin system then.

[applied-mathematician]

Add me to the list of those facing this issue. My team is working on a large django project that incorporates multiple python apps (eventually installed on multiple nodes) and shared local libraries. While I already sold them on moving to poetry, getting a dev environment set up currently seems to more of a hassle than it's worth. I'd prefer to wait than use the work-arounds others have mentioned for simulating an editable install.

@abn I like your #2270 proposal and would like to help with any feedback/planning to get things rolling ASAP. My gitlab ticket of moving our project to poetry is going to hang until one of the proposed solutions gets implemented.

[m1nde]

Would this work? I'd create a seperate pyproject.toml in a new folder which lists all of my main package's in-company dependencies as editable path dependencies. I then run poetry install followed by poetry shell. In this shell, with the venv activated, I then do my development in my main package. The dependencies in my main package's pyproject.toml can stay regular version dependencies, which allows poetry build. Since my shell already has a venv activated, poetry run commands will import the editable dependencies.

[misterjoa]

Hello everyone,
we are also facing this as we have a monorepo and want to be able to build dev packages locally and use defined versions of packages in prod.
Seems related to https://github.com/python-poetry/poetry/issues/2046

[janhurst]

I'm also facing a similar challenge working with a "main" application that uses a shared library that I have added as a Git submodule. Apologies in advanced I'm pretty new to poetry, coming from conda and manual venv.

I want to be able to work on the shared library, and interact with it's Git repo, but a CI/CD produced version of the shared library from my private Azure Artifacts feed when I build my application (in a Docker container).

I had expected behaviour similar to what I think the OP suggested where the local path was used when I poetry install when working in my dev environment, but a repo/feed version was used when I poetry install --no-dev in my Docker build.

Using something like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "./my-lib1-in-a-git-submodule/", develop = true }

I think this partially mitigates the issue the OP mentions about co-workers organizing their projects differently.... I have the luxury of "forcing" a project layout by using a Git submodule, but regardless I think this would still be useful?

[kesiraa]

How about having separate toml files for various environments. ?
i.e pyproject.dev.toml, pyproject.prod.toml
This will also allow us to test newer python versions (i.e 3.9 in dev to test, while 3.8 is in the prod)

[micahscopes]

If you haven't tried it, yarn has a nice feature for this called yarn link. It could be nice to have something like this in poetry!

[briggySmalls]

@janhurst I have _exactly_ the same issue that you do. Our production docker containers use poetry to build a wheel before installing it in a fresh image. Currently the shared library in the dev-dependencies causes this build step to fail.

[mrijken]

I have made https://pypi.org/project/poetry-dev/ in order to support this use case.

[francoposa]

I would really like this to support absolute paths, so we can reliably set it up the same way inside and outside of a dockerfile:

(learn-spark) [~/repos/learn-spark]% poetry add /usr/local/spark/python 

[ValueError]
'/usr/local/spark/python' does not start with '/Users/franco/repos/learn-spark'

So for now I'm using relative path, which won't be the same as the docker container when it gets built...

(learn-spark) [~/repos/learn-spark]% poetry add ../../../../usr/local/spark/python 

[abn]

@francojposa you could try using the url dependency. file:///usr/local/spark/python. However, I am guessing you are aware of the potential breakages this could introduce as this is not really conisdered portable.

[francoposa]

@abn That doesn't work. This isn't a total dealbreaker, but a pain. Do you think supporting absolute paths would be a relatively simple addition/would the project be open to a PR?

It seems that even though it gets declared with a relative path, Poetry ultimately understands it as an absolute path.
When I switched from the local path requirement to just a standard pyspark requirement and ran poetry lock it printed out

- Updating pyspark (3.0.0 /usr/local/spark/python -> 3.0.1)

Use case is to run the same Poetry requirements during both local dev and in a Docker container that already has Spark baked in.

For vanilla pip/requirements.txt, we just pip install -e /usr/local/spark/python/.

[danmur]

It's not ideal but I used an optional dependency for my build process:

[tool.poetry.dependencies]
my-package = { version = "^0.1.0", optional = true }

[tool.poetry.dev-dependencies]
my-package = {path = "../my-package"}

[tools.poetry.extras]
full = ["my-package"]

poetry install in dev, and pip install myotherpackage[full] to deploy

[francoposa]

I don't want to derail this thread, I think my issue is more accurately described by https://github.com/python-poetry/poetry/issues/1692

[freddrake]

I don't want to derail this thread, I think my issue is more accurately described by #1692

This isn't really about using absolute paths, but whether path dependencies can be used in particular circumstances.

[kkawabat]

I'm also facing a similar challenge working with a "main" application that uses a shared library that I have added as a Git submodule. Apologies in advanced I'm pretty new to poetry, coming from conda and manual venv.

I want to be able to work on the shared library, and interact with it's Git repo, but a CI/CD produced version of the shared library from my private Azure Artifacts feed when I build my application (in a Docker container).

I had expected behaviour similar to what I think the OP suggested where the local path was used when I poetry install when working in my dev environment, but a repo/feed version was used when I poetry install --no-dev in my Docker build.

Using something like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "./my-lib1-in-a-git-submodule/", develop = true }

I think this partially mitigates the issue the OP mentions about co-workers organizing their projects differently.... I have the luxury of "forcing" a project layout by using a Git submodule, but regardless I think this would still be useful?

I am trying to do something similar to this

[tool.poetry.dependencies]
mylibdep = {git = "[email protected]:mygit/mylibdep.git", tag = "1.49.0"}

[tool.poetry.dev-dependencies]
mylibdep = {path = "../mylibdep", develop = true}

but I am getting this error when trying to poetry lock

Because mylib depends on both mylibdep (1.49.0 git tag 1.49.0) and mylibdep (1.49.0 ../mylibdep), version solving failed.

I think the problem is that poetry install installs both mylibdep and causes a conflict. Is there a way to prevent the production dependency from installing conditioned on whether -no-dev is on or not? Looking at the doc it seems like I can use environment-markers but don't know poetry enough.

Something like below to prevent conflict?:

[tool.poetry.dependencies]
mylibdep = {git = "[email protected]:mygit/mylibdep.git", tag = "1.49.0",  markers = "POETRY_FLAG_NO_DEP"}

[tool.poetry.dev-dependencies]
mylibdep = {path = "../mylibdep", develop = true}

[dorranh]

@kkawabat, IIRC the issue is that poetry dev and standard dependencies form a kind of super set, so the lock command fails since you have two conflicting versions of the same package defined (one as the git dependency and as a local dependency). Are you just installing the package in the docker image build or are you trying to build a wheel? If only the former, have you tried using only the path dependency and ensuring that your CI script checkouts both the repo _and_ the submodule? I think that should get you what you're after - my colleague had to do something similar to work with git submodules.

[sinoroc]

The following technique seems to work.

[tool.poetry.dependencies]
python = "^3.8"
foosball-lib = "*"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
foosball-lib = {path = "../foosball-lib", develop = true}

See this repo for an example: https://github.com/sinoroc/foosball

What issues do you encounter with this technique?

[MakuZo]

@sinoroc This will always install the foosball-lib from the path directory, even with the --no-dev flag.

[sinoroc]

This will always install the foosball-lib from the path directory, even with the --no-dev flag.

I just tested it and confirmed it. That is true, and that is strange. This is not necessarily what I would have expected. I also tried after having removed the poetry.lock lockfile beforehand, and the result was the same.

$ poetry --version
Poetry version 1.1.4
$ poetry install --no-dev
Updating dependencies
Resolving dependencies... (1.5s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing foosball-lib (0.1.0 /home/sinoroc/workspace/foosball/foosball-lib)