Poetry: [development] Unable to install poetry's dependencies

[agilgur5]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS High Sierra 10.13
• Poetry version: 0.12.17
• Link of a Gist with the contents of your pyproject.toml file: It's this repo, self.

Issue

Following CONTRIBUTING.md, I cloned and cd'd into the repo and then when I run poetry install I get:

Installing dependencies from lock file

[NonExistentKey]
'Key "hashes" does not exist.'

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

In verbose mode:

Using virtualenv: /Users/<user>/Library/Caches/pypoetry/virtualenvs/poetry-py3.7
Installing dependencies from lock file

[NonExistentKey]
'Key "hashes" does not exist.'

Exception trace:
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/<user>/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/<user>/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/<user>/.poetry/lib/poetry/console/commands/install.py in handle() at line 55
   return_code = installer.run()
 /Users/<user>/.poetry/lib/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /Users/<user>/.poetry/lib/poetry/installation/installer.py in _do_install() at line 169
   locked_repository = self._locker.locked_repository(True)
 /Users/<user>/.poetry/lib/poetry/packages/locker.py in locked_repository() at line 86
   package.hashes = lock_data["metadata"]["hashes"][info["name"]]
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/tomlkit/items.py in __getitem__() at line 981
   return self._value[key]
 /Users/<user>/.poetry/lib/poetry/_vendor/py3.7/tomlkit/container.py in __getitem__() at line 524
   raise NonExistentKey(key)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

Also, this occurs with both Python 3.7 and Python 2.7.

I went like 10 commits back and it still occurred, as did with 20 commits back, and 25 commits back. Eventually I narrowed it down to https://github.com/sdispater/poetry/commit/e54f3907f1e110813b690febcfb89ebd75224bea being the root cause. The commit before that (or more before it) work fine and poetry install works on them.

[b0g3r]

I think you need to use poetry preview version (for example 1.0.0b3) for installing master branch

[agilgur5]

@b0g3r yep, using 1.0.0b3 worked to install. That leads to a few follow-ups though:

1) Why did https://github.com/sdispater/poetry/commit/e54f3907f1e110813b690febcfb89ebd75224bea break compatibility with Poetry stable (i.e. 0.12.17)? Not sure if it was intentional / meant to break compat or not.
2) In its pyproject.toml, Poetry doesn't actually specify a build-system, which would normally list which version of poetry it depends on to work. I'm assuming there's a reason for that, but that would be the root cause of this type of issue, no?
3) In lieu of 2, assuming it can't be specified, this should probably be documented in CONTRIBUTING.md. I can add a PR for that, but not sure what the min version required to use master is (and if that differs for develop).

[pikacic]

I think it's fair to state that poetry should be built with itself (I mean the checkout out version).

My Impression is that poetry 1.x can read poetry 0.12.x lock file, but not the other way around, so e54f390 just updates the format, I believe, under the assumption that the master version is always the version to be used.

I don't now yet how to do it (I just removed poetry.lock to be able to run the test), but CONTRIBUTING.md should explain how to bootstrap... even if just by saying

remove poetry.lock before calling poetry install

Of course, it would be better to document how to locally install the master version and use it for poetry itself.

[pikacic]

OTOH, backing what @b0g3r suggested, we could just update CONTRIBUTING.md to state that for developing you must install the latest preview version (and keep it up to date).

AFAICT, @sdispater deployed b2 with the support for the new lock format before committing e54f390, so the preview version implicitly was meant to work to build the master branch.

[vanyakosmos]

Temporary hack for installing dependencies:

pip install cleo tomlkit jsonschema pyparsing keyring pkginfo cachecontrol cachy html5lib lockfile
python -m poetry install

[janw]

The occurrence of lock-file incompatilibities is exactly what I envisioned to be handled more gracefully when proposing #1410. Maybe that is something worth picking up?

[jiamo]

poetry install --no-dev should not remove the package need by poetry itself.

[finswimmer]

Hello,

with https://github.com/sdispater/poetry/pull/1684 poetry should now be able to build itself. I think this resolves the original issue and we can close this issue. Just leave a comment if you disagree.

@jiamo : How is your comment related to this issue?

fin swimmer

[agilgur5]

@finswimmer does that solve #1523 as well? That was labeled as a duplicate of this issue, but it seems like _users_ of poetry were affected as well (not just _contributors_ of poetry), especially after seeing all the related issues pop up here. That might be due to the lockfile incompatibility?

I haven't seen this in my own usage of poetry outside of here, so I'm not really sure exactly how those folks happen upon this error. Possibly #1523 should be reopened as a separate, but related issue, or this one should be renamed given all the links here. Hopefully someone else can chime in who has experienced it outside of contributing.

[jiamo]

@finswimmer In production env you will only need --no-dev. But when you want to use poetry again. It failed on missing packages that poetry needs. (I am sure install --no-dev will remove the package poetry needs) I google the jsonschema pyparsing to come here.

[vancouverwill]

@finswimmer does that solve #1523 as well? That was labeled as a duplicate of this issue, but it seems like _users_ of poetry were affected as well (not just _contributors_ of poetry), especially after seeing all the related issues pop up here. That might be due to the lockfile incompatibility?

I haven't seen this in my own usage of poetry outside of here, so I'm not really sure exactly how those folks happen upon this error. Possibly #1523 should be reopened as a separate, but related issue, or this one should be renamed given all the links here. Hopefully someone else can chime in who has experienced it outside of contributing.

I am not developing poetry and am just using poetry as a user but see this error regularly for the last 48 hours but haven't been able to figure it out. My solution has been to keep checking out the working poetry.lock from git and use that but that seems to only work sometimes and the sometimes seems to be if I delete the virtual environment first which is really painfully slow.

OS version and name: Ubuntu 18.04
Poetry version: 1.0.0
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/vancouverwill/3d386cede415de3ed5f281cd640360bc

[mbelang]

If you delete the lock and regenerate should fix the problem.

[vancouverwill]

thanks for the quick reply. That works the first time I run poetry install but the next time I run it I get

Requirement already satisfied: zipp>=0.5 in /home/will/.cache/pypoetry/virtualenvs/communitymodels-L1wR3w1Y-py3.7/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema<4.0,>=3.0->communitymodels==0.20.7) (0.6.0)
Requirement already satisfied: more-itertools in /home/will/.cache/pypoetry/virtualenvs/communitymodels-L1wR3w1Y-py3.7/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata; python_version < "3.8"->jsonschema<4.0,>=3.0->communitymodels==0.20.7) (8.0.2)
Installing collected packages: pandas, communitymodels
  Found existing installation: pandas 0.25.3
    Uninstalling pandas-0.25.3:
      Successfully uninstalled pandas-0.25.3
  Found existing installation: communitymodels 0.47.1
Cannot remove entries from nonexistent file /home/will/.cache/pypoetry/virtualenvs/communitymodels-L1wR3w1Y-py3.7/lib/python3.7/site-packages/easy-install.pth
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Interesting thing there is where it is trying to install
~/.cache/pypoetry/virtualenvs/communitymodels-L1wR3w1Y-py3.7/
Up until a few days ago it would always install at
~/.cache/pypoetry/virtualenvs/communitymodels-py3.7/
But since I've noticed this weird behaviour it seems to coincide with poetry installing at a this directory with a hash inbetween the app name and the python version, whereas before it was always just the app-name followed by the python version.

p.s. Even if that worked that is less than ideal solution for production software as when generating a whole new set of dependencies from scratch, isn't that non-deterministic the exact graph of dependencies it installs? At least in concurrent languages there would be no guarantee of the order of package installation so the exact permutation of dependencies could always be different. I may be wrong but that is my assumption, happy to be find out I am wrong though and learn more about how this works :smile:

[mbelang]

@vancouverwill Would you mind pasting me the dependencies section of your pyproject.toml I will try it on my side. The only difference that I see with my setup is that I set the venv to be created in app directory instead of poetry cache (for local setup). For CI/CD I do use the a cache folder and I never saw that issue and I know we use panda in some projects. As for my Docker project we do not use venv as it is not really useful in the context of a container.

[vancouverwill]

Okay thanks here is the pyproject.toml file https://gist.github.com/vancouverwill/3d386cede415de3ed5f281cd640360bc

How do you mean you do not use venv in the context of a container? Presumably with poetry you always have a venv but the choice is whether to have it in project in the .venv folder or in your home folder at ~/.cache/pypoetry/virtualenvs/...., otherwise if there was no venv at all then where would the dependencies get installed to? Thanks :smile:

[mbelang]

@vancouverwill I cannot reproduce though I had to remove the packages section as I do not have you source.

I did poetry install, then I did poetry install again.

here are my poetry configs poetry config --list

cache-dir = "/home/xxxxxxx/.cache/pypoetry"
repositories.dialogue.url = "https://pypi.fury.io/xxxxxxxxx"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/xxxxxxxx/.cache/pypoetry/virtualenvs

For containers I do this to install all deps as system deps: poetry config virtualenvs.create false then I do poetry install --no-dev

[vancouverwill]

@mbelang I found the issue it was that my docker container had Poetry 0.12.3 and in my host machine I have Poetry 1.0.0
We don't have time to change all the places which use our docker base image right now so easiest solution was to downgrade Poetry in my host machine and run poetry install. Seems after that no more issues :+1: but thought worth putting here incase helps someone.

[mbelang]

@vancouverwill Happy that you found the problem. It's a classic :) On our side we decided to update everywhere.

[vancouverwill]

@vancouverwill Happy that you found the problem. It's a classic :) On our side we decided to update everywhere.

Haha yeah 🤦‍♂️
Curious are they expected to be incompatible (version 0.x and 1.x)? Cheers

[mbelang]

Unfortunately yes... In fact 1.0.0 is compatible with 0.12.x lock format but not the other way around.

https://github.com/python-poetry/poetry/issues/1410

[vancouverwill]

Okay thanks

[vancouverwill]

Thanks !

[anpr]

Just for reference: I just upgrade to the latest poetry pip install --upgrade poetry and it worked again.

[datgrog]

FYI: using poetry installed by pip did not work too, even with the cmd above. I just pip uninstall it and reinstall with the recommanded way curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python and it worked again!