Poetry: Project metadata

[hynek]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

Since poetry stores its metadata like version, homepage etc in pyproject.toml, is there a way for project to introspect themselves _after_ being installed?

There is the "standard" of putting meta data into dunders into the main __init__.py such that they can be introspected at runtime.

My setup.py-based approach is based on parsing those __init__.py s but I could live with inverting that relationship, so is there a way to fill __init__.py data based on the data in pyproject.toml?

FWIW, flit seems to at least extract the __version__ from there.

[pmav99]

For the record to do that, you need to ship the *.toml file with your wheels

[hynek]

That would require to depend on both setuptools (for pkg_resources) and toml (to parse it) – no?

[pmav99]

Toml is not really required for simple key-values: https://github.com/sdispater/poetry/issues/144#issuecomment-449718997

[hynek]

IIRC there were some issues when parsing toml using an ini parser.

Regardless, I really don't think that loading and parsing a file in your __init__.py is a good idea. There should be a better way.

I'd personally prefer if poetry could actually get its metadata from the __init__.py like flit does it for __version__.

[drunkwcodes]

It's recommended to use pyproject.toml as the source of the version number.
There are a lot of srcs to place version number, including in git tags, which are not in any file.

I think this should not be handled by poetry, which prefers semantic versioning.
The lack of official PEP440 implementation is another cost to complete this feature.

A wiki to gather solutions of this kind of puzzles is probably what poetry needs.

[chrahunt]

This use case should be covered by importlib_metadata. In your __init__.py:

from importlib_metadata import version

__version__ = version(__package__)

This will use the package data that is persisted in METADATA or PKG-INFO by pip on installation from wheel or sdist.

[drunkwcodes]

@chrahunt It is off-topic.

[chrahunt]

The specific question here is

is there a way to fill __init__.py data based on the data in pyproject.toml?

The approach I mention above answers this question, since the METADATA/PKG-INFO which are present in the installed package are generated by poetry from the pyproject.toml.

I should also mention that importlib_metadata is not a random package, but a backport of importlib.metadata which will be available in Python 3.8. What I posted will, I believe, be the conventional way for Python packages to provide this metadata moving forward (regardless of what build tool they use).

[drunkwcodes]

@chrahunt It is still different from parsing version number from pyproject.toml.

And the path introduces another coupling to pip by a new, distant route which should be prevented.

[chrahunt]

If we take the question as-stated: "based on the data in pyproject.toml", then I think it is OK to derive it from the metadata which is based on the data in pyproject.toml. Can we think of any use cases that are not covered by using importlib_metadata?

And the path introduces another coupling to pip by a new, distant route which should be prevented.

There is nothing pip-specific here. The mentioned behavior is true for any Python packages as specified in PEP 345 (PKG-INFO for sdist) and PEP 427 (METADATA for wheels). Poetry already implements these and in fact it must in order for the generated packages to be considered valid.

[chrahunt]

Just to be clear, I'm not suggesting any change in poetry for this, I think that an approach like I mentioned above (or a better one if we can come up with it) needs to be documented somewhere since it will be a common problem poetry users will have.

[drunkwcodes]

@chrahunt But pyproject.toml will be also in the package. And the metadata will be in either place. The issue still needs to be open after you posted a even not a workaround.

[chrahunt]

But pyproject.toml is also in the package.

pyproject.toml is not in the package - to check please run poetry build on a plain poetry-based project and extract the wheel. We would need to include it explicitly as mentioned by @pmav99, but there is not really a need if the required data is in <project>-<version>.dist-info/METADATA, as generated by poetry (and accessible in a generic way using importlib_metadata).

[nickpresta]

I'd like to add my +1 to the original request and my experience with a confusing bug due to this:

I have a package with a pyproject.toml defined as such (most things omitted for brevity):

version = "1.5.0"

include = ["pyproject.toml", "mypackage/**/*.py"]

In mypackage/version.py, I have the following:

import pkgutil
import re

version_regexp = re.compile(r'''^version = "([^"]*)"''', re.M)

data = pkgutil.get_data(__package__, "../pyproject.toml")
match = version_regexp.search(data.decode("utf-8"))
if match:
    __version__ = match.group(1)
else:  # pragma: no cover
    raise RuntimeError("Unable to find version string")

This seems to work and doesn't depend on a toml package for parsing (at the risk of the regexp failing).

When installing this package, my site-packages looks like:

$ ls venv/lib/python3.7/site-packages/mypackage*
venv/lib/python3.7/site-packages/chunnelx:
version.py

venv/lib/python3.7/site-packages/mypackage-1.5.0.dist-info:
INSTALLER  METADATA   RECORD     WHEEL

In addition to:

$ cat venv/lib/python3.7/site-packages/pyproject.toml
[tool.poetry]
name = "mypackage"
version = "1.5.0"

# rest omitted

This works "fine" until I installed another package that is doing the same thing (e.g. mypackage2). Now, thepyproject.tomlfile insite-packagesgets overriden during installation, potentially in a non-deterministic way depending on package installation order, withmypackage2'spyproject.tomland the code inmypackage` is using the wrong version.

It seems like an ideal solution would be some way to use data_files or additional meta-data functionality.

Both this ticket and https://github.com/sdispater/poetry/issues/890 seem to cover the related functionality to make something like this easier/possible.

There is an open PR: https://github.com/sdispater/poetry/pull/901 -- is there something I can do to help get this merged in (reviews, testing, etc)?

[drunkwcodes]

The override can be avoided by not getting the version number in runtime.
But the metadata should be checked if in sync with __init__.py manually in development.

I don't know if it is a fasion to get version number like this thread describes but I hate it (without effective verification.) You are exploiting poetry version.

[chrahunt]

In #144 (also linked above) it was mentioned that bumping version in both pyproject.toml and __init__.py is planned. Maybe it just needs an implementation?

Something like poetry version --check would also be good, to verify in CI.

[drunkwcodes]

@chrahunt I mean the checksum in metadata in the case of getting version in runtime that you mentioned.

By the way, how is it going about your pyenv maintenance?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[merwok]

This is still relevant :robot: Poetry could get a way to extract metadata from code (like flit and setuptools), call an arbitrary function, or (if it takes that stance) document how to use importlib.metadata to get version.

[finswimmer]

The solution given by @chrahunt using importlib_metadata IS the solution to the issue. It is not a workaround .