Poetry: Running poetry in parallel fails on file access

[pbecotte]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04 on WSL
• Poetry version: 1.0.0a4
• Link of a Gist with the contents of your pyproject.toml file: n/a
  

  Issue

While setting up tox based testing with Poetry, I kept getting odd errors about a missing pyproject.toml file. A minimal reproduction is

python -m virtualenv .test/py36 --python python3.6
python -m virtualenv .test/py37 --python python3.7
VIRTUAL_ENV=.test/py36 poetry install -vvv --no-dev || echo "failed 36" &
VIRTUAL_ENV=.test/py37 poetry install -vvv --no-dev || echo "failed 37" &
wait

This will get one of the processes erroring out on the last step with

[FileNotFoundError]
[Errno 2] No such file or directory: '/home/pbecotte/PycharmProjects/rye/pyproject.toml'

Traceback (most recent call last):
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 112, in handle
    status_code = self._do_handle(args, io)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 160, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/pbecotte/.poetry/lib/poetry/console/commands/install.py", line 96, in handle
    builder.build()
  File "/home/pbecotte/.poetry/lib/poetry/masonry/builders/editable.py", line 17, in build
    return self._setup_build()
  File "/home/pbecotte/.poetry/lib/poetry/masonry/builders/editable.py", line 38, in _setup_build
    str(self._poetry.file), str(self._poetry.file.with_suffix('.tmp'))
  File "/usr/lib/python3.7/shutil.py", line 577, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.7/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

From following the stack trace, I can see that Poetry is temporarily renaming the pyproject.toml file to trick pip into doing an editable install, which is causing the second process to fail.

I'd be happy to add a PR to fix this, but am kind of lost on the right approach. Should there be a lock/wait before the file rename (can you use a file lock on a file that gets renamed?)? Is there a way to directly invoke pip to do the right thing without moving any files? Should we use the previously suggested approach of invoking setuptools directlly? Should I just suck it up and put retry logic into my Makefile?

[kierun]

Obviously, this happens on the latest beta too.

Version: 1.0.0b2
Python:  3.7.4

Virtualenv
Python:         3.7.4
Implementation: CPython
Path:           /home/usr/.cache/pypoetry/virtualenvs/usignals-efiEjv8b-py3.7
Valid:          True

System
Platform: linux
OS:       posix
Python:   /home/usr/.pyenv/versions/3.7.4

[johnfraney]

I came up against this today using the most recent version of Poetry, v1.0.0b9.

My workaround was to avoid using poetry install to install dependencies, and instead to use poetry export to generate a requirements file and pip install to install dependencies from that requirements file. This allows Tox to work in parallel because no files are renamed when installing dependencies.

Here's my tox.ini for reference:

[tox]
skipsdist = True
envlist = py36, py37, py38, pypy3

[testenv]
whitelist_externals = poetry
skip_install = true
commands_pre =
    poetry export -f requirements.txt -o .tox/requirements.txt
commands =
    pip install -r .tox/requirements.txt
    pytest

I hope this helps!

[chrisrink10]

With a tox.ini like this:

[tox]
envlist = py36,py37,py38,...
isolated_build = true

[testenv]
whitelist_externals = poetry
commands =
    poetry install -v
    poetry run ...

I was able to force my tests to run sequentially while allowing other CI tasks to proceed in parallel doing this.

[testenv:py37]
depends = py36

[testenv:py38]
depends = py37

Certainly slower than @johnfraney 's option above. But I don't want to use skipinstall and skipdist.

[kierun]

@johnfraney I tried your tox.ini with a tiny modifications:

poetry export --verbose --dev -f requirements.txt -o .tox/requirements.txt

Sadly, it fails for me with:

ERROR: Double requirement given: astroid==2.3.3 (from -r .tox/requirements.txt (line 15)) (already in astroid==1.6.6 (from -r .tox/requirements.txt (line 10)),
name='astroid')
ERROR: InvocationError for command /home/usr/repos/umbrella/nifty_logging_colours/.tox/py37/bin/pip install -r .tox/requirements.txt (exited with code 1)

Which to be fair, it rather odd since a poetry install --dev works just fine…

[franknarf8]

It seems you can use the --no-root flag of the poetry install command to avoid this concurrency issue. tox will install the root package anyway if you do not specify the install skip_install setting.

Something of the sort :

[testenv]
whitelist_externals = poetry
commands =
    poetry install -n -v --no-root
    poetry run pytest [...]

[kierun]

@franknarf8 Thank you kind internet stranger!

[paulkernfeld]

I ran into this while trying to do a parallel install of multiple apps that use shared libraries like my_lib = {path = "libs/my_lib"}.

The --no-root workaround suggested by @franknarf8 didn't work for me, maybe because the FileNotFoundError actually refers to the pyproject.toml file of the library.

The poetry export ... | pip install /dev/stdin solution also didn't work for me, because pip can't install a poetry package as a dependency (see #761).

[pbecotte]

I actually wound up writing my own task runner (rye on pypi) so that I could.more easily handle the concurrency and task graph.

no-root should work for most use cases though, as the "rename the file and run pip install ." Behavior doesn't happen.