Poetry: All dependencies are always updated

[wylee]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option). [N/A]

• OS version and name: macOS Catalina (10.15.1)

• Poetry version: 1.0b4, 1.0b5, 1.0b6
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/wylee/53e74a3413df25e1b594803e193d40d4

Issue

Since 1.0b4, when running poetry update, all of my dependencies are always updated. I've tried clearing the cache and deleting poetry.lock.

1582 is similar but seems to affect only project that use a private package index.

[finswimmer]

Hello @wylee ,

thanks for reporting.

Could you please try to describe a step-by-step instruction to reproduce the issue? I wasn't able to do this until now.

fin swimmer

[wylee]

I started with Poetry 0.12.17 then upgraded to 1.0b3. This issue wasn't present on either of those versions. Then I upgraded to 1.0b4 and this started happening. I figured it was a temporary issue, so I waited for a couple more beta releases (b5 and b6) to come out. I'm now on 1.0b6, and I'm still seeing this behavior.

This is the output for the pyproject.toml I linked to above every time I run poetry install:

Installing dependencies from lock file


Package operations: 0 installs, 4 updates, 0 removals

  - Updating markupsafe (1.1.1 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.1.1)
  - Updating jinja2 (2.10.3 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 2.10.3)
  - Updating pyyaml (5.1.2 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 5.1.2)
  - Updating runcommands (1.0a44 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.0a44)

And every time I run poetry update:

Updating dependencies
Resolving dependencies... (0.2s)


Package operations: 0 installs, 4 updates, 0 removals

  - Updating markupsafe (1.1.1 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.1.1)
  - Updating jinja2 (2.10.3 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 2.10.3)
  - Updating pyyaml (5.1.2 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 5.1.2)
  - Updating runcommands (1.0a44 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.0a44)

[wylee]

I just tried deleting my project's virtualenv and then re-running poetry install and then poetry update. With a fresh virtualenv (created by poetry), this issue isn't present, so it appears to somehow be caused by updating poetry and using it with an existing virtualenv.

[wylee]

So, I was able to reproduce this in the following way:

• Uninstall poetry
• Install poetry 0.12.7
• cd project
• Remove project virtualenv
• poetry install && poetry update (no issue here)
• Upgrade poetry to 1.0b3
• poetry install && poetry update (no issue here)
• Upgrade poetry to 1.0b4
• poetry install && poetry update (now dependencies are upgraded every time)
• Remove project virtualenv
• poetry install && poetry update (issue is gone)

One thing I noticed here is that starting with 1.0b4 virtualenvs are created in ~/Library/Caches/pypoetry/virtualenvs even though my poetry config looks like this:

[settings]
[settings.virtualenvs]
in-project = true
path = ".venv"

Edit: It appears the config format changed slightly in 1.0b3, but that doesn't seem to be relevant to this issue.

[wylee]

I stumbled across another way this can happen with 1.0b4, 1.0b5, and 1.0b6:

• Start fresh by removing the project virtualenv and poetry.lock and making sure the virtualenv is deactivated
• Run poetry install
• Run poetry update (dependencies are not upgraded)
• Activate the project's virtualenv
• Run poetry update (now dependencies are upgraded every time)
• Deactivate the project's virtualenv (or manually unset the VIRTUAL_ENV environment variable)
• Run poetry update (dependencies are not upgraded)

[sdispater]

Thanks the detailed steps to reproduce. Unfortunately I am still unable to reproduce.

What's the output of poetry debug info and how did you install Poetry?

[wylee]

I installed poetry using pipx.

Output of poetry debug info:

> poetry debug info

Poetry
Version: 1.0.0b6
Python:  3.8.0

Virtualenv
Python:         3.6.9
Implementation: CPython
Path:           /home/me/Projects/bycycle/bycycle.webui
Valid:          False

System
Platform: darwin
OS:       posix
Python:   /usr/local/var/pyenv/versions/3.6.9

Based on that, I think I figured out what's happening. When VIRTUAL_ENV doesn't match what poetry is expecting, updates always happen because it considers the virtualenv invalid. In my development environment, I don't use . .venv/bin/activate. Instead, I have a custom script for setting $PATH and $VIRTUAL_ENV when I cd into a project directory, and it was setting $VIRTUAL_ENV incorrectly.

So, that's my mistake (and sorry for the noise), but I do think it might be good if poetry showed at least a warning, or even aborted, in the case of an invalid virtualenv.

[dmontagu]

I am also getting this in versions 1.0.0b4-1.0.0b7; in my case, at least, it appears to be related to pip installing dependencies in a location poetry isn't expecting (e.g., if they are pip installed with the --user flag).

When this is the case, it seems like poetry is running some pip-related command in the background (maybe?), but it doesn't result in any changes, which is why it always installs them all.

I'll see if I can make an easily reproduced docker image.

[pappasam]

I experience this issue when using an in-project virtualenv (.venv/). Similarly to the others, I did not experience the issue on earlier beta versions. The problem is present on Poetry version 1.0.0b8.

• I installed poetry using the recommended curl approach.
• System: Ubuntu 18.04

[sdispater]

@pappasam What's the output of poetry debug info?

[pappasam]

@sdispater hmm, strange, but I've re-run in the same project and no longer have the problem. Not sure how to replicate, I apologize:

Poetry
Version: 1.0.0b8
Python:  3.8.0

Virtualenv
Python:         3.8.0
Implementation: CPython
Path:           /home/sroeca/src/Personal/serdataclasses/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /home/sroeca/.asdf/installs/python/3.8.0

[jonathanunderwood]

I too am seeing this problem having upgraded from version 0.12.17 to 1.0.0b8.

$ poetry debug info

Poetry
Version: 1.0.0b8
Python:  3.7.5

Virtualenv
Python:         3.7.5
Implementation: CPython
Path:           /Users/XXXXX/.virtualenvs/hatch-api
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7

Doing a poetry update --lock seems to fix the issue, interestingly.

[Red-Eyed]

@sdispater

I faced this issue again on 1.0.3 windows.
However, this issue is not present on version 1.0.0

Could you please reopen this issue?

Thanks!

[huseyinyagli]

I have the same issue on 1.0.3 on mac

[Red-Eyed]

and BTW. it's fine on linux (I use archlinux personally)

[slaking]

This should be a pretty basic feature of a dependency manager: NOT wastefully (both in time and in network usage) always updating every package. I'm on Mac and have the same problem. I tried reverting to 1.0.0 too and while it's better, it still seems to always update a couple of packages. Please add a test after fixing this issue. Until then, I'm switching back to pipenv.

[Red-Eyed]

This issue is blocking as poetry updating everything to the same version.
Poetry uninstalls all deps then installs all deps.

The most interesting and annoying effect of this is that it uninstalls dependencies that poetry depends on!

python -m poetry install
Installing dependencies from lock file


Package operations: 95 installs, 9 updates, 0 removals

  - Installing decorator (4.4.1)
  - Installing ipython-genutils (0.2.0)
  - Updating six (1.14.0 C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages -> 1.14.0)
  - Updating zipp (2.2.0 C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages -> 2.2.0)
  - Updating attrs (19.3.0 C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages -> 19.3.0)
  - Updating importlib-metadata (1.1.3 C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages -> 1.1.3)
  - Installing parso (0.6.1)
  - Updating pyrsistent (0.14.11 C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages -> 0.14.11)

[EnvCommandError]
Command ['C:\\tools\\miniconda3\\envs\\gen3_poetry\\Scripts\\pip.exe', 'uninstall', 'pyrsistent', '-y'] errored with the following return code 2, and output:
Found existing installation: pyrsistent 0.14.11
Uninstalling pyrsistent-0.14.11:
  Successfully uninstalled pyrsistent-0.14.11
ERROR: Exception:
Traceback (most recent call last):
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\cli\base_command.py", line 186, in _main
    status = self.run(options, args)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\commands\uninstall.py", line 82, in run
    uninstall_pathset.commit()
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\req\req_uninstall.py", line 450, in commit
    self._moved_paths.commit()
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\req\req_uninstall.py", line 290, in commit
    save_dir.cleanup()
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\utils\temp_dir.py", line 175, in cleanup
    rmtree(self._path)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_vendor\six.py", line 703, in reraise
    raise value
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\site-packages\pip\_internal\utils\misc.py", line 136, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\shutil.py", line 400, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\tools\miniconda3\envs\gen3_poetry\lib\shutil.py", line 398, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Vadym\\AppData\\Local\\Temp\\pip-uninstall-wxminyz9\\pvectorc.cp37-win_amd64.pyd'

and this leads to the above exception:
it tries to remove pvectorc.cp37-win_amd64.pyd, but this library is used by poetry, and Windows throws error that says that you cannot remove file that is being used.

[dashmug]

I just tried adding autopep8 into my project right now and got this...

$ poetry add -D autopep8
Using version ^1.5 for autopep8

Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file


Package operations: 1 install, 66 updates, 0 removals

  - Updating <package>

Poetry is installed via Homebrew on Mac.

poetry debug info

Poetry
Version: 1.0.3
Python:  3.7.6

Virtualenv
Python:         3.7.6
Implementation: CPython
Path:           /Users/noel/Library/Caches/pypoetry/virtualenvs/core-backend-X6-Pn97V-py3.7
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7

[tribals]

Yeah... This is the reason why I switched from pipenv to poetry. Now it behaves exactly the same...

@sdispater Why do you think that I want to always update dependencies? Actually, I use dependency manager in order to control updates. And by control I mean that I will update only when I decided to, not when dependency manager wants...

[Red-Eyed]

@sdispater Please, reopen this issue.

[Seanny123]

I have the same issue on Fedora 31 and using conda, as I describe in #2281

[grski]

Same but inside docker container. Running it without creating virtualenv.
I was using 1.0.5, downgrading to 1.0.0 fixed the issue.

[willmcmain]

My team noticed that some of us were having this issue and others weren't, and we found that people who installed poetry with homebrew were the ones who had the issue. That might be relevant to tracking down what's causing this.

[Red-Eyed]

OMG, this is CLOSED issue, please, reopen it.

[wylee]

@willmcmain I've been installing poetry using pipx, and that's been working fine (never tried installing it via Homebrew).

I wonder if the Homebrew package configures things a certain way that could cause this. Here's the current formula (don't have time to inspect it right now):

https://github.com/Homebrew/homebrew-core/blob/master/Formula/poetry.rb

[grski]

@wylee it's not a homebrew thing - I ran into the same issue on linux (two different distros - alpine & manjaro)
Besides it seems it's already in progress - no need to reopen this:
https://github.com/python-poetry/poetry/issues/2225
https://github.com/python-poetry/poetry/issues/2281

[wylee]

@grski I don't think it's a homebrew thing per se, just that the homebrew formula, especially how it sets up sys.path, might give some clue as to what the issue is.