Poetry: Path based dev-dependencies break 'install --no-dev' when the directory does not exist

[spither]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.10
• Poetry version: 0.12.10
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/spither/47f144b1f355990d562b5ba92bf15fe3

Issue

I cannot run poetry install --no-dev away from my development environment because I have dev-only path based dependencies.

> poetry install --no-dev

[ValueError]             
Directory ../dev-only does not exist

The error is correct, the directory doesn't exist, but it's only listed as a dev-dependency and I'm trying to install --no-dev. I was not expecting any dev-dependency checks to take place.

In my case I'm trying to install the non-dev dependencies for a Django site deployment. The path based dev-dependencies only exist in development, not on the deployment system.

[sdispater]

Poetry locks everything, including dev dependencies, before installing to ensure that there is no conflict between dev and runtime dependencies, so every dependency must be resolvable.

[spither]

This happens with an existing lock file in place.

Is resolving dev dependencies (especially with an existing lock file) really needed when --no-dev is specified? Is there anything I can do to avoid it?

I don't have a need or desire to copy these dependencies to my deployment system. They wouldn't/shouldn't be used for anything there and might add confusion by being present.

[rphes]

I've also run into this issue, my use case being that I want to use a local (and editable) version of a package during development and a commit from a git repository for deployment situations.

[jjuris]

Hello,

I also found in this situation when I used poetry in my project. As a workaround, I simply commented out lines under development dependencies section in the project, but it is not nice solution -> un/commenting lines in configuration file is pretty annoying.

@sdispater, I would like to help you solve this problem.

The simplest solution would be "Ignore dev dependencies when --no-dev option is used."

If it is ok for you let me know and I will try to find it and fix it. If you have better idea, let me know and we can discuss it.

Have a nice

Day

Jozef Juris

[jjuris]

Hello,

I think I found a solution for this case. It is pretty simple. The idea to ignore dev-dependencies seems to be fine. Existing tests are passing and the poetry is acting in desired manner.

Unfortunately I am not sure if this solution is elegant enough, but I did not find other way to work with --no-dev flag without hacking a lot of code.
What do you think @sdispater ?

Best regards

Jozef

1119 commit

poetry.py

        import sys
        if "dev-dependencies" in local_config and not "--no-dev" in sys.argv:
            for name, constraint in local_config["dev-dependencies"].items():
                if isinstance(constraint, list):
                    for _constraint in constraint:
                        package.add_dependency(name, _constraint, category="dev")

                    continue

                package.add_dependency(name, constraint, category="dev")

[hozn]

I'm also interested in a solution for this. My scenario may be slightly different from OP, but like @rphes , I'd like to be able to use local path dependencies during development (so that I can make associated library changes concurrently) but have the final image builds pull in only real tagged dependencies. I assume that path dependencies is the answer here, though admittedly I'm not sure how I'd have the same dep exist as both a path dependency (for development) and a real verisoned dep (for packaging/production) -- @rphes , how do you solve that aspect?

This feels like a key missing feature in poetry for handling large-scale project development (where a project consists of many related libraries).

[LouisTrezzini]

hello, thanks for the great work on Poetry

we are also facing this issue, is there any plan to solve it in an upcoming release?

[reem]

Also facing this issue, makes it incredibly inconvenient to use path based dependencies in development and external dependencies in production - "solutions" include maintaining two duplicate pyproject.toml files or commenting/uncommenting constantly, neither of which is really acceptable.

[tall-josh]

I know it's a MASSIVE hack but this was killing my docker build and I didn't want two pyproject.toml so I just put the [tool.poetry.dev-dependencies] heading at the bottom of my pyproject.toml and used RUN sed -i -n '/tool.poetry.dev-dependencies/q;p' pyproject.toml before poetry install.

I know. I feel dirty, but I regret nothing!

[pwoolvett]

Yet another workaround:

1. create a "virtual" setup.py with a redirect depending on the dependency existence

   $ cat /path/to/virtual_dependency/setup.py
   from pathlib import Path
   import sys
   
   path = Path("/path/to/real/dependency/setup.py")
   if path.exists():
     sys.path.insert(0, str(path.parent))
     from setup import setup
   else:
     from distutils.core import setup
   
      setup(
          name="real_dependency",
          version="0.0.1",
          description="""Mock real_dependency""",
          packages=[""],
          package_data={"": []},
      )
   

2. poetry add --optional /path/to/virtual_dependency

tested with Poetry version 1.1.4

[sinoroc]

@pwoolvett
Maybe something like the following could work as well:

#!/usr/bin/env python3

import pathlib

import setuptools

def main():
    install_requires = []
    dep_name = 'Schroedinger'
    dep_path = pathlib.Path(f'../{dep_name}')
    if dep_path.is_dir():
        install_requires.append(f'{dep_name} @ {dep_path}')
    #
    setuptools.setup(
        install_requires=install_requires,
    )

if __name__ == '__main__':
    main()