Poetry: Relative installs break on subsequent installs due to `pip-wheel-metadata`

[micimize]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS + 10.14.5

• Poetry version: 0.12.6 (sudo pip3 install poetry==0.12.16)

parent pyproject.toml

[tool.poetry]
name = "parent"
packages = [{ include = "parent" }]
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.7"
dependency = {path = "../dependency"}

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

poetry install generates pip-wheel-metadata files in my relative-path dependencies, which break later installs with FileExistsError. My current workaround is simply to rm -rf ../dependency/pip-wheel-metadata/ before installs.

FileExistsError: '/path/to/dependency/pip-wheel-metadata/dependency-0.1.0.dist-info' installation failure

Using virtualenv: /path/to/parent/.venv
Installing dependencies from lock file


Package operations: 5 installs, 0 updates, 0 removals, 60 skipped

  - Skipping ... installed
  - Installing relative-dep (0.1.0 ../dependency)

[EnvCommandError]
Command ['/path/to/parent/.venv/bin/python', '-m',     
'pip', 'install', '--no-deps', '-U', '-e', '/path/to/dependency
'] errored with the following output:
Obtaining file:///path/to/dependency
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    Complete output from command /path/to/parent/.venv/bin/python /path/to/parent/.venv/lib/python3.7/
site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/1_/d
rxyjr152tvfmv8jp80hyzrw0000gp/T/tmp2tvqzsf2:
    Traceback (most recent call last):
      File "/path/to/parent/.venv/lib/python3.7/sit
e-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/path/to/parent/.venv/lib/python3.7/sit
e-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/path/to/parent/.venv/lib/python3.7/sit
e-packages/pip/_vendor/pep517/_in_process.py", line 69, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/1_/drxyjr152tvfmv8jp80hyzrw0000gp/T/pip-build-env-h22cq9gb/overla
y/lib/python3.7/site-packages/poetry/masonry/api.py", line 39, in prepare_metadata_for_build_wheel
        dist_info.mkdir()
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/
pathlib.py", line 1251, in mkdir
        self._accessor.mkdir(self, mode)
    FileExistsError: [Errno 17] File exists: '/path/to/dependency/pip-wheel-metadata/dependency-0.1.0.dist-info'

    ----------------------------------------
Command "/path/to/parent/.venv/bin/python /path/to/parent/.venv/lib/python3.7/site-packages/pip/_vendo
r/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/1_/drxyjr152tvfmv8jp80hyzrw0
000gp/T/tmp2tvqzsf2" failed with error code 1 in/path/to/dependency
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Exception trace:

 /usr/local/lib/python3.7/site-packages/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /usr/local/lib/python3.7/site-packages/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /usr/local/lib/python3.7/site-packages/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /usr/local/lib/python3.7/site-packages/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /usr/local/lib/python3.7/site-packages/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /usr/local/lib/python3.7/site-packages/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /usr/local/lib/python3.7/site-packages/poetry/console/commands/install.py in handle() at line 55
   return_code = installer.run()
 /usr/local/lib/python3.7/site-packages/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /usr/local/lib/python3.7/site-packages/poetry/installation/installer.py in _do_install() at line 290
   self._execute(op)
 /usr/local/lib/python3.7/site-packages/poetry/installation/installer.py in _execute() at line 306
   getattr(self, "_execute_{}".format(method))(operation)
 /usr/local/lib/python3.7/site-packages/poetry/installation/installer.py in _execute_install() at line 331
   self._installer.install(operation.package)
 /usr/local/lib/python3.7/site-packages/poetry/installation/pip_installer.py in install() at line 30
   self.install_directory(package)
 /usr/local/lib/python3.7/site-packages/poetry/installation/pip_installer.py in install_directory() at line 205
   return self.run(*args)
 /usr/local/lib/python3.7/site-packages/poetry/installation/pip_installer.py in run() at line 112
   return self._env.run("python", "-m", "pip", *args, **kwargs)
 /usr/local/lib/python3.7/site-packages/poetry/utils/env.py in run() at line 536
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /usr/local/lib/python3.7/site-packages/poetry/utils/env.py in run() at line 385
   raise EnvCommandError(e)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

This was not the case for this same structure and pypproject.tomls in an earlier, now-lost environment.

I have not been able to replicate the previous working environment, but it had poetry==1.0.0a2 and python 2 + it's pip, whereas I now don't have any "activated" pip<19 (though behavior doesn't change by installing different pip versions, as poetry manages it's own pip 19.0.3)

[vovapolu]

@micimize Any progress on this issue? Any workaround? It's really annoying to remove pip-wheel-metadata before every install.

[micimize]

@vovapolu I'm on 1.0.0a3 now, but commenting out the build system bock in submodules, clearing artifacts (pip-wheel-metadata, etc), poetry lock, and then poetry install from the top level project seems to be working for me.

# [build-system]
# requires = ["poetry>=0.12"]
# build-backend = "poetry.masonry.api"

[vovapolu]

Well, indeed removing pip-wheel-metadata every time solves the problem. Or you suggesting to do poetry lock firstly?

[micimize]

@vovapolu I'm saying that after I commented out the build-system and did those steps once, I didn't have to keep removing pip-wheel-metadata every time

[Hartorn]

Hello

I'm having the same problem, with Poetry 0.12.17 and python 3.7.4
I think this #1245 should fix it.

[mgasner]

We would very much like to adopt Poetry, but this issue is blocking us.

We are an open-source monorepo, with several subdirectories that are independently published to PyPI. Users may use one, or all of our packages, and our packages have interdependencies.

Because, e.g., package_c requires package_b and package_b requires package_a, it's currently impossible to install both package_b and package_c for local development -- package_b's existing relative install of package_a will trigger this issue when a user tries to install package_c.

The workaround is not viable for an open source project.

[Natureshadow]

Does the merged PR above fix this, or not?

[micimize]

@Natureshadow It does not appear to be fixed on 1.0.0b1, but I'm not sure why. When I try to poetry install, I get

File "/private/var/folders/1_/drxyjr152tvfmv8jp80hyzrw0000gp/T/pip-build-env-dfxxevnh/overlay/lib/python3.7/site-packages/poetry/masonry/api.py", line 39, in prepare_metadata_for_build_wheel
        dist_info.mkdir()

but the same lines in ~/.poetry/lib/poetry/masonry/api.py are now

    dist_info = Path(metadata_directory, builder.dist_info)
    dist_info.mkdir(parents=True, exist_ok=True)

I've tried deleting ~/.poetry and virtual environments and reinstalling, but nothing has helped

[Natureshadow]

File
"/private/var/folders/1_/drxyjr152tvfmv8jp80hyzrw0000gp/T/pip-build-env-dfxxevnh/overlay/lib/python3.7/site-packages/poetry/masonry/api.py",
line 39, in prepare_metadata_for_build_wheel
dist_info.mkdir().

Looks like another location with the same bug.

[Hartorn]

@micimize @Natureshadow
Going blind on this, but did you put requires = ["poetry>=1.0.0b1"] instead of 0.12 ?
I feel like dependency is downloaded when doing the build, so version needs to be the same as local version

[micimize]

@Hartorn I'll be damned - this specific issue was solved by replacing requires in all the dependencies as well.
I ran into a different issue that has the same workaround, but I think it is a more niche issue.
Opened #1315 for that, closing this