Poetry: PEP-517 and pip install --editable with pip 19

[albireox]

• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] I have searched the documentation and believe that my question is not covered.

Question

Hi,

Apologies if this has been answered before but I've looked at a number of issues and discussions about this and I'm still confused about the details.

As of pip 19.2.1 it is not possible anymore to do pip install -e . on a poetry-managed project with a pyproject.toml regardless of whether the build-system is defined. As I understand it, this was done so on purpose.

Is there a working workaround around this issue? In my organisation we work with multiple projects that are not in PyPI and that are not even intended to be installed to site-packages. We do python setup.py build and then add the resulting build directory to PYTHONPATH. It may be argued this is not good practice but ...

I've been really liking poetry but this is kind of a dealbreaker for me. I can convince people in my project to do pip install -e . instead of python setup.py build (and yet I'll get complaints) but anything beyond that, such as forcing people to use poetry for building the project, is not really workable.

Alternatively, is there a way to generate a setup.py from the pyproject.toml? I assume there is since the distribution source file contains one. If so, maybe it could be added to poetry install as an option that generates the setup.py, the same way the poetry.lock file is generated. It's not a good solution but it would solve the problem for now.

[brycedrennan]

@albireox Interesting. do you know what specific change in pip caused this?

https://pip.pypa.io/en/stable/news/

[albireox]

I think the two relevant conversations are pypa/pip#6314 and pypa/pip#6334.

[jtrakk]

If I'm reading those two issues correctly, I think the summary is:

• Pip currently does not allow installing PEP 517 projects (i.e. projects with a pyproject.toml) in editable mode. Pip should define and implement a standard for allowing pyproject.toml packages to be installed editable, but nobody's yet volunteered to do that.
• In the meantime, Poetry could implement a workaround: when an editable install is attempted, temporarily remove the pyproject.toml and put it back after installing.

[albireox]

I think the problem is not that there is a pyproject.toml file in the directory, but that there is no setup.py and that poetry relies on specifying the build-backend, which is not yet supported in pip.

I don't understand the internal of poetry enough but I think the workaround would require creating a temporary setup.py. The machinery for that must already exist since the file is created when deploying.

[jtrakk]

@albireox Poetry already does create a setup.py. I think it's the pyproject.toml that's the problem, because PEP 517, which says what to do if there's a pyproject.toml, does not support editable installs.

[albireox]

I think that's correct. But ultimately pip fails because there isn't a setup.py file. If there were, I think it would run that and ignore the pyproject.toml file. So a workaround could be to introduce a poetry command to generate the setup.py file (or even better, update it every time pyproject.toml changes). That way we could treat the generated setup.py the same way we do with the lockfile and commit it. People without poetry would still be able to use pip to install the package.

[jtrakk]

Ok I just tried this. Poetry already creates and deletes a setup.py during poetry install. If I comment out the part where it deletes the setup.py afterwards, I get a setup.py in the directory. Then pip install -e works. So we could have a command build the setup.py and not remove it.

[albireox]

That sounds like a good temporary solution. This could be a flag in the poetry install command. Maybe something like poetry install --keep-setup.

[dmontagu]

I have also found myself trying to "catch" the generated setup.py before it was deleted while struggling to debug issues with my custom build steps (I adapted an example for building cython extensions, but had to tweak it in order to build cmake+pybind11 extensions as part of the install). Having an option to keep the setup.py would have been useful!

[albireox]

Trying to solve a different issue (see #1516) I came across a solution that seems to also fix this problem. See https://github.com/sdispater/poetry/issues/1516#issuecomment-547649400 and this repo.

Basically one can create a custom setup.py that still works (mostly) fine with poetry. I'm sure this will cause some other things to break down the line but ...

[jtrakk]

From following the discussions in PyPA, I believe the situation is basically:

I don't think there is much if any controversy here, it's just that no one has created a proof of concept with setuptools yet

If somebody wants to take a crack at the proof of concept, a specification is laid out here. I don't think it would be too difficult to get the ball rolling.

[finswimmer]

Hello,

this is not an issue by poetry, as others already pointed out. So I will close this.

A workaround similar to @albireox 's suggestion is to run poetry build once, extract the generated setup.py from the sdist build and place it beside the pyproject.toml.

fin swimmer