Poetry: Poetry removes importlib_metadata and other requirements when run as `--no-dev`

[vad]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: https://hub.docker.com/_/python:3.6-alpine
• Poetry version: 1.0.3
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/vad/874bcd90092a98b21ae73a2982246b3c

Issue

I'm using poetry inside a docker with poetry config virtualenvs.create false. Poetry requirements are listed in my poetry.lock as dev (they probably are dependencies of some other package). When I run poetry install --no-dev the first time, the command runs fine but poetry removes its own requirements!

...
  - Removing attrs (19.3.0)
  - Removing cffi (1.14.0)
  - Removing cryptography (2.8)
  - Removing idna (2.8)
  - Removing importlib-metadata (1.5.0)
  - Removing jsonschema (3.2.0)
  - Removing pycparser (2.19)
  - Removing pyparsing (2.4.6)
  - Removing pyrsistent (0.15.7)
  - Removing zipp (3.0.0)

The second time poetry is run, I get:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/poetry/utils/_compat.py", line 15, in <module>
    from importlib import metadata
ImportError: cannot import name 'metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 5, in <module>
    from poetry.console import main
  File "/usr/local/lib/python3.6/site-packages/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/lib/python3.6/site-packages/poetry/console/application.py", line 5, in <module>
    from .commands.about import AboutCommand
  File "/usr/local/lib/python3.6/site-packages/poetry/console/commands/__init__.py", line 2, in <module>
    from .add import AddCommand
  File "/usr/local/lib/python3.6/site-packages/poetry/console/commands/add.py", line 5, in <module>
    from .init import InitCommand
  File "/usr/local/lib/python3.6/site-packages/poetry/console/commands/init.py", line 16, in <module>
    from poetry.utils._compat import OrderedDict
  File "/usr/local/lib/python3.6/site-packages/poetry/utils/_compat.py", line 18, in <module>
    import importlib_metadata as metadata
ModuleNotFoundError: No module named 'importlib_metadata'

[burkovae]

Indeed this is very annoying when I run CI/CD (im my case in gitlab) when separating development and production environment. Is there any applicable workaround ?

[burkovae]

Indeed this is very annoying when I run CI/CD (im my case in gitlab) when separating development and production environment. Is there any applicable workaround ?

To answer my own question: the solution in my case is to use poetry export --without-hashes -f requirements.txt > requirements.txt (bacause "Development dependencies are excluded from the export by default")

[matthijskooijman]

I am seeing this same issue here, with a manually created virtualenv, where I installed poetry (1.0.6) into, and then ran poetry install --no-dev which removes, among others, importlib-metadata. Starting poetry afterwards ends up importing importlib from the system python instead of the virtualenv, resulting in the above exception. Also here importlib-metadata is listed as a dependency of another dev-only package, so it only breaks with --no-dev.

In a way, poetry does exactly what I ask of it - make the installed packages equal to what's listed. There's one exception, though: poetry does not uninstall itself (I assume this is some hardcoded exception in the code?). Maybe this exception should be extended to also include its own dependencies?

Note that this is different from requiring poetry and its dependencies, since when poetry is run from outside the virtualenv, there is no need to install it also into the v

[matthijskooijman]

One extra observation: It seems that poetry only removes packages that are mentioned in its pyproject.toml and/or poetry.lock (maybe just one, haven't checked).

I tried installing an extra package (mysqlclient) using pip directly, then ran poetry install --no-dev and it would again remove importlib-metadata and some others (all of which are listed as (indirect) dev-dependencies), but it would not remove mysqlclient, presumably since it was not listed at all.

A workaround that seems to work, is to add poetry itself as an optional dependency:

poetry add --optional poetry

This adds a ton of extra packages to poetry.lock (all dependencies of poetry), but does not install them on poetry install. Also, it prevents removing them on poetry install --no-dev.

[glennmatthews]

I can confirm that this issue is still seen with Poetry 1.0.9.

Another apparent workaround: if I delete the poetry.lock file, this issue isn't seen.

[klarose]

I am seeing a further side-effect of this. I have a package which depends on some source-based python packages. When I install with poetry install --no-dev, some of those packages (a random selection) are not linked into the system path. Normally they would be.

The first thing the install does is remove a bunch of transitive dependencies of my dev packages, which were actually installed by the system (e.g. virutalenv, filelock, etc). I have a suspicion that with the loss of these packages, something in the installation process goes bad.

Either way, my fix is to either include these packages explicitly in my non-dev dependencies, which seems incorrect, since they are part of the system, and I have no idea what new ones will be introduced when poetry upgrades, new dependencies are added, etc. Or, I can poetry install && poetry install --no-dev. The latter is reasonable, but inefficient.

It would be nice if poetry could either recognize that it is uninstalling packages it never touched, or at least provide an option to not remove packages for CI-based workflows like this where you're starting from a clean slate every time.

[abn]

@klarose are you using poetry to manage a system environment? When poetry creates an environment, only pip, wheel and setuptools will typically be present. Even in CI environments I'd recommend the use a virtual environment.

As for this issue, one option coule be to make it so that for --no-dev we do not remove anything, but only update versions that are changed. And current behavior could then be replicated by --no-dev --remove-untracked. The issue here is that this looses some of the benefits of the install command keeping everything correctly installed. Alternatively we can consider introducing --no-remove

Additionally, performing removals from system environments is a bad idea. If you add a dependency to your project that was already in the environment, there is no good way poetry could tell it was previously present. The current behavior is, believe it or not, a good feature. But yes, we should probably add a warning when a system environment is used (if we can detect it reliably).

[klarose]

FWIW, it turns out that my workaround doesn't work consistently, so I wouldn't go with it.

@abn I'm trying out a venv now to see if it helps. The reason we decided not to do that before is that it just seems like an extra bit of overhead when we're running in a container whose purpose is already quite similar to a venv.

[klarose]

Okay. venvs didn't work as well. Clearly something else is going on.

[ishikawa]

I also encountered this issue (Python 3.6). My workaround (like others): add some dependencies as optional and add those to extras section.

$ poetry add --optional importlib-metadata
$ poetry add --optional virtualenv

pyproject.toml

[tool.poetry.extras]
myextra = [..., "importlib-metadata", "virtualenv"]

then

$ poetry install --no-dev --extras myextra

[finswimmer]

Hello,

the problem here is, that poetry and the package are installed in the same environment. One should never mix the installation method within one environment. Either use poetry or pip. But not both at the same time. Because they don't know each other and break things when a package gets installed or removed by one or another.

When using virtualenvs.create false make sure poetry is installed with the installer script or as an alternative by pipx.

fin swimmer