Poetry: Support generation of poetry manged setup.py file

[abn]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

As things stand today, poetry packages cannot be installed using pip or any other package managers from a VCS. This is useful when working with unreleased/unpublished versions of dependencies, where dependencies are managed by 'pip' or another tool. This was briefly touched on in #34 for a similar use case.

With this in mind, it could be great if poetry provided a solution, in a limited capacity, similar to what is implemented by poetry-setup. In order to achieve this, poetry could provide a command, eg: poetry setuptools:setupfile, that generates a usable setup.py file using the mechanism used today by SdistBuilder.

In theory, this could be migrated to be a plugin once #693 is implemented.

As an example. this, if implemented, could be used by projects as a pre-commit hook to generate setup.py file on commit. This should allow projects using pip etc. to install unpublished projects developed using poetry (if the maintainer wants it).

In addition to the above use case, this also allows for tools like PyCharm etc., to auto detect requirements and metadata.

Can follow this up with a proposed implementation PR.

Relates-to: #34

[webknjaz]

@abn other way to do it is to actually publish each commit to Test PyPI :)

[abn]

@webknjaz that would definitely work for a subset of use cases. The generation of setup.py and treating it similar to a lock file allows for a broader range of compatibility.

[cjw296]

Won't pip 19.0 remove the need for this? (as it understands pyproject.toml) - you'll need to add some manual stuff to it though, until https://github.com/sdispater/poetry/issues/744 is resolved.

[abn]

@cjw296 I will see if I can validate that, if pip understands poetry projects, that will most likely cover a large chunk of the use cases. However, without this change, the other tools need to either support PEP-518 or poetry projects explicitly. With that in mind, I do think this is still useful even if pip support is completed.

[radek-sprta]

Snaps require setuptools and setup.py, at least for now, so having way to generate setup.py from poetry would help with that.

[sdll]

Any updates on that?

[stevegore]

dephell will do this for you:

dephell deps convert --from pyproject.toml --from-format poetry --to setup.py --to-format setuppy

Or you can add this to pyproject.toml and just run dephell deps convert:

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "setuppy", path = "setup.py"}

[abn]

I have updated the PR to be a new export format instead of new command.

[KelSolaar]

I would be super glad to have that feature out-of-the-box. I looked quickly at dephell but the fact it added almost 30 dependencies to my project made me uncomfortable.

[danielhoherd]

One great reason to have this feature is that you cannot do pip install -e . unless there is a setup.py file.

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/username/python-tool
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

For now the dephell command up above did what I was looking for, but requried attrs>=19.2.0. Also dephell has 49 dependencies.

[orsinium]

Also dephell has 49 dependencies.

DepHell is intended to be an outside tool like you have no poetry in your dependencies. The recommended way to install dephell is curl -L dephell.org/install | python3 that will put dephell into its own jail.

Also, most of the dependencies are part of dephell's own ecosystem (https://github.com/dephell/). And even after this, there are much fewer dependencies than even in pip (https://github.com/pypa/pip/tree/master/src/pip/_vendor), the only difference is they aren't vendorized for good reasons. Installation of all dependencies from the ground zero takes 17 seconds on TravisCI. So, it's more about decoupling than having a fat dependency tree.

[stevegore]

I feel I should point out that the pyproject.toml addition I've mentioned above doesn't actually required you to add dephell as a dependency of your project. Instead, I've got poetry and dephell both installed as part of the Docker image I use to build our Python projects.

[albireox]

For what it's worth, here is a small script that uses poetry itself to generate the setup.py. It seems to work fine so far.

[pawamoy]

@albireox nice, thank you for the script.

Here are two other versions to find Poetry's lib.

• In any virtualenv (pyenv, pipx, etc.):

  from pathlib import Path
  from shutil import which
  
  which_poetry = which("poetry")
  with open(which_poetry) as fd:
    shebang = fd.readline()
  poetry_python_lib = next(
    Path(shebang[2:]).parent.parent.glob("lib/python*")
  ) / "site-packages"
  

• More specifically, when installed with pipx, with support for Windows as well:

  import os
  from pathlib import Path
  
  if os.name == "nt":
    poetry_python_lib = Path("~/.local/pipx/venvs/poetry/Lib/site-packages").expanduser()
  else:
    poetry_python_lib = next(
        Path("~/.local/pipx/venvs/poetry/lib").expanduser().glob("python*")
    ) / "site-packages"
  

[nim65s]

@pawamoy: thanks, but you have an extra ) after .parent.parent

[pawamoy]

Woops, thanks @nim65s, corrected the snippet :slightly_smiling_face:

I also added support for Windows for the pipx version.

[RafalSkolasinski]

For what it's worth, here is a small script that uses poetry itself to generate the setup.py. It seems to work fine so far.

Is there a reason why we don't add it to poetry itself? Lack of connection to standard package layout with setup.py is what may stop some people from adopting poetry in their projects.

[danielhoherd]

Another reason to generate setup.py is this error from tox:

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/user/code/foo
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

EDIT: as noted in the following comment, this was a problem with my configurations

[abn]

@RafalSkolasinski setup.py is specific to using distutils/setuptools as the build backend. Poetry is a PEP 517 compatible build backend and does not require the setup.py file.

@danielhoherd unsure what is happening here, tox should work file without the setup.py file. You can see poetry's tox.ini as an example. Also see here for more information.

[RafalSkolasinski]

Would it be a good idea to add to documentation why setup.py is not required when using poetry?

Are there any limitations to noy having it? Like, am I able to "pip install -e ." or install directly from Github?

Edit: it seems that missing setup.py does not allow to install package in editable mode

machine42 ➜  poetry-demo pip install -e .
Defaulting to user installation because normal site-packages is not writeable
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/rskolasinski/private/playground/poetry-demo
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

which seems to be quite a limitation for some use cases.

[abn]

Would it be a good idea to add to documentation why setup.py is not required when using poetry?

Personally, I think PEP 517 implications have been already documeted elsewhere. A quick google search lead me to this article, which summarised the implication as of November, 2018. There aslo is a more recently updated article by Brett Cannon here, which I recommend.

It might be good to add some basic information and resources in the introduction section. Happy to accept pull requests.

Are there any limitations to noy having it? Like, am I able to "pip install -e ." or install directly from Github?

PEP 517 has been supported by pip since version 19. The requirement for a setup.py file for an editable install is more a pip concern than poetry. As mentioned in the article above by Brett, you can use a setup.py shim file in your project. This is not something that is required for projects managed by poetry in general. There are no real-world limitations by not having it, the intention is to reduce the number of files required to manage a python project.

If your project requires to be installed in editable mode, I would recommend adding a shim. But this should not be the default behaviour for poetry managed projects.

[merwok]

tox and pip install implement the PEPs to support many build backends.

pip install -e is not compatible yet, but there is ongoing discussion to standardize that and make it work too. In the meantime, each tool can provide a bespoke way to do something like develop/editable mode (for example, flit has flit install -s which gets the same result).

[RafalSkolasinski]

@abn @merwok Thanks guys for the reply and the links, seems I have some readings to do ; - ).
Personally I was never a fun of setup.py, found it a bit over complicating things.

I see that pip install . does work properly. I hope it will also work properly when the source is on GitHub using pip install git+...?

I also notice that if activate manually a virtualenvironment (not managed by poetry) then poetry install will give be equivalent of pip install -e . - so that this also makes me happy.

The only thing I seem to be missing is to have equivalent of pip install --user -e .
I am working on some scripts that ideally would be installed into ~/.local/bin/... and would like to install them there in editable mode for development.

[abn]

I see that pip install . does work properly. I hope it will also work properly when the source is on GitHub using pip install git+...?

It should, yes. As this is equivalent of cloning a local copy and performing pip install /path/to/clone.

The only thing I seem to be missing is to have equivalent of pip install --user -e .
I am working on some scripts that ideally would be installed into ~/.local/bin/... and would like to install them there in editable mode for development.

If your virtual environment is active, any such scripts will be available in your path, as /path/to/venv/bin should be in your PATH.

[RafalSkolasinski]

If your virtual environment is active, any such scripts will be available in your path, as /path/to/venv/bin should be in your PATH.

I meant installation without virtualenvironment that would be equivalent with pip installation with --user flag, as it going into ~/.local/bin and~/.local/lib.

[RafalSkolasinski]

I see there exists https://github.com/python-poetry/poetry/issues/1214 which address my concern.
It seems this is the only downside of not having setup.py to me it seems.

[Drachenfels]

There is backward compatibility issue to be taken into account. I am working in an environment that assumes that setup.py is a source of truth. It's either has to be updated and committed or removed altogether. If poetry had this option out of the gate it would make my life easier to convince other people to start using it (simply by making it optional and allowing old-school devs to keep their old ways).

[jdeyton]

@stevegore

dephell will do this for you:

dephell deps convert --from pyproject.toml --from-format poetry --to setup.py --to-format setuppy

Or you can add this to pyproject.toml and just run dephell deps convert:

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "setuppy", path = "setup.py"}

The format of the dephell command is --from-path instead of --from and --to-path instead of --to.

i.e.

dephell deps convert --from-path pyproject.toml --from-format poetry --to-path setup.py --to-format setuppy

[pawamoy]

For those having issues with the script from @albireox with newer versions of Poetry, check it out again: https://github.com/sdss/flicamera/blob/master/create_setup.py, it was updated 🙂 Thanks @albireox ❤️ !!

[kolypto]

In fact, poetry generates a setup.py when you build the project, and places it into one of the dist/*.tar.gz files :)
Let's extract it:

$ poetry build
$ tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1

now your poetry project is pip-editable ;)

[abn]

now your poetry project is pip-editable ;)

You can also just use a setup-shim for most cases.

cat "import setuptools; setuptools.setup()" > setup.py

Note that having a setup file can cause some side-effects.

[JasonAtallah]

In fact, poetry generates a setup.py when you build the project, and places it into one of the dist/*.tar.gz files :)
Let's extract it:

$ poetry build
$ tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1

now your poetry project is pip-editable ;)

I was getting errors with --wildcards and --no-anchored.

Running just tar -xvf dist/*.tar.gz '*/setup.py' seems to work for me.
I then run cp $(find . -name setup.py) setup.py to copy the setup.py over to the root of the my repo.

[Aathish04]

Has there been any consensus on whether such a script/tool for keeping the generated setup.py file should be added to poetry itself?

I imagine it would be extremely useful, at least until pypa finally decide what to do when you need an editable install with only a pyproject.toml .