Poetry: [Discussion] Poetry to provide a decent hook-based plugin system

[webknjaz]

It's a follow-up for https://github.com/sdispater/poetry/pull/672#issuecomment-443363390.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

So setuptools have this nice hook points system where they allow replacing certain utils by some third-party implementations.
It can catch up with what's available by iterating through a list of entrypoints with names corresponding to hooks.
This is a fairly popular technique among setuptools-based dists. For example, pytest and tox support having third-party dists being automatically activated and used when installed given they "advertise" their entry points as required by the plugin system. These two tools use https://github.com/pytest-dev/pluggy to achieve this.

Back to Poetry, I think lots of people agree that providing ability to extend such a great tool is a widely wanted feature.
This would allow adding custom dist/package format generators, custom metadata generators etc. They could be used to retrieve long description from some esoteric formats or places converting that into supported markups. This would also make possible having a custom version constructor like popular plugins to setuptools facilitating version sourcing from Git/Hg (#140/#672). Or, another example, exporting a pinned env and direct dependencies into pip-friendly formatted requirements. And who knows, maybe someone will even try out having another resolver replacement?

Any thoughts on how this might look like?

[jgirardet]

Hi,
At the time I was searching a way to specify python version at install, I thought a bit about it.

What does we want ?

• add some custom command:
  the easiest I think. It just needs to look for new commands in the setup tools entry point and add it in console/application.py

• add some flags/option to existing command:
  already more tricky. should it be added for all commands ? I see something like it : a do_plugin_task() a start of handle which would apply some change before the builtin code. it isn't something hard except that poetry runs via cleo which define the option in the class docstring. so if you want to add an option to an existing command, it has to be changed at instantiation time. It means parsing the docstring change it and instantiate the "new" command. You need to have a clear logic in it since many plugins could change the same command.

What is different in poetry for doing this ?
Poetry is not installed via pip so using the standard way of entry point maybe not be ok (I'm not sure about it). It should come from a poetry command. someting like : poetry plugin install some_plugin and the plugin could be in pypi.

Should we try to implement this ?
I would be ok, maybe others too but my feeling is that @sdispater since the beginning of the project, likes to implement himself the new core features of poetry. It's me feeling, I'm not sure about sebastien's mind about it.

Anyway, the idea of a plugin system is itself a great thing !!!

[sdispater]

Just for information: I am currently rewriting the core of Cleo to make it more flexible and pluggable.

Regarding the actual plugin system, I am not sure what it will look like just yet. But the general idea is to be able to add commands and command options to Poetry like @jgirardet said. I also think being able to hook into the Poetry object creation would be good since it would allow things like #672.

Now, i will likely implement the plugin system myself and make a PR to get feedback.

This is something I had in my mind from the start but there were more pressing matters than this but since Poetry is on the path of stabilization I think it's a good time to start experimenting with it.

I can't guarantee that it will make it to the 1.0 release but I will try.

[webknjaz]

Cool, thanks for the info.

[webknjaz]

Hey @sdispater, is https://poetry.eustace.io/docs/plugins/ something that implements what you wanted? Is it already possible to hook into that?

[webknjaz]

@sdispater any updates?

[funkyfuture]

while @webknjaz inital post points to uses that would extend poetry's functionality as a packaging tool and kinda venv manager, the idea to allow custom commands points to a different path. from the discussions that i came across, they all aimed at either extending poetry to a task runner or to interact with version information stored in the config file. for these cases, poetry would only act as a framework to implement cli commands around a pyproject.toml. at that point poetry would be two different things. in a better, possible world poetry and these other command implementations would share a common framework.

[vBLFTePebWNi6c]

Hi, @sdispater ! Our team is looking for pipenv's alternatives and poetry seems great, if it'd have plugin system, which is mandatory for our development process. I've seen that related PR is in progress -- can you give us eta for this feature?

[mrh1997]

Another use case for hooks would be calling preprocessors like cython or template precompilers.
Ideally these libraries could provide a plugin, that is called automaticially before the "build" command. The plugins could retrieve the details which files to process and how from a custom section of the pyproject.toml.

One important thing is that this requires the plugin interface living within the projects python virtual environment, while the plugin interface discussed above lives in the python interpreter where poetry is installed.

The big advantage of a plugin interface living in the project interpreter would be, that it not only solves the preprocessing-requirement, but also the other requirements of this issue on a per project basis. This would mean that adding a plugin is as simple as adding a development dependency to pyproject.toml. It will ensure that all developers working on the project are using the same plugin and plugin-version.

[jedie]

Is there a pre publish hook or any work a round or hack to get this?

EDIT: i created https://pypi.org/project/poetry-publish/ and used poetry via subprocess...

[gsemet]

That would be really useful if plugins could be installed inside the virtualenv of the package, not system/user wide!

[PetterS]

Would be nice if the plugin system allowed customizing the execution of package installation. This would allow a significant speed-up by enabling parallel installation: see https://github.com/python-poetry/poetry/pull/2374/files

[sdispater]

@PetterS Improving the installation time (and a refactoring of the installation logic) is planned and already part of the roadmap for the 1.1 release (see https://github.com/python-poetry/poetry/issues/1856).

[PetterS]

Cool, did not realize this was covered in "Download distributions separately before installing them.".

[fredrikaverpil]

Is it possible to somehow hook into poetry today, which e.g. makes it possible for me to inject the version from setuptools_scm into pyproject.toml just before building a wheel using poetry build?
Preferably, also some way of undoing the change to pyproject.toml after the wheel was built.

I'm looking for a "quick fix" until there is a proper setuptools_scm support with poetry.

This is what I do currently, but it's not at all very nice...:

# scm_version_hack.py

from setuptools_scm import get_version


def write_toml(filepath="pyproject.toml", reset=True):
    contents = []
    new_contents = []
    with open(filepath, "r") as infile:
        contents = infile.readlines()
    for line in contents:
        if line.startswith("version ="):
            if reset:
                new_contents.append(f'version = "0.0.0"  # DO NOT CHANGE VERSION HERE\n')
            else:
                new_contents.append(f'version = "{get_version()}"\n')
        else:
            new_contents.append(line)
    with open(filepath, "w") as outfile:
        outfile.writelines(new_contents)


def main():
    print(get_version())


if __name__ == "__main__":
    main()

python -c "import scm_version_hack; scm_version_hack.write_toml(reset=False)"
poetry build
python -c "import scm_version_hack; scm_version_hack.write_toml(reset=True)"

[mtkennerly]

@fredrikaverpil, you may be interested in this pseudo-plugin I made: https://github.com/mtkennerly/poetry-dynamic-versioning

[fredrikaverpil]

Thanks @mtkennerly that looks nice! 👍
However, we're a number of people who then need to make sure we have the same local setup (as well as the CI), and I guess this will break if poetry is updated?

So I think then my current workaround is probably better in the interim.

[gsemet]

I wonder why this feature would not be provided by Poetry out-of-the-box, of course on a voluntary basis.