Poetry: support namespace packing

[mozartilize]

I follow the docs here: https://packaging.python.org/guides/packaging-namespace-packages/
but it does not work, maybe poetry not support yet?

[uSpike]

How would the maintainers feel about an additional optional field in pyproject.toml called module. For example:

[tool.poetry]
name = "abc-foo"
module = "abc.foo"
...

corresponds with a layout like:

pyproject.toml
abc/
    foo/
        __init__.py

[RaptDept]

See https://github.com/sdispater/poetry/issues/356#issuecomment-410193960

[dfee]

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

[moigagoo]

@dfee what if the package name is different and doesn't correlate with the dir structure?

[dfee]

My directory structure is as follows:

repo/
    abc.core/
        pyproject.toml
        abc/
            core/
                __init__.py
    abc.asdf/
        pyproject.toml
        abc/
            asdf/
                __init__.py
    abc.qwerty/
        pyproject.toml
        abc/
            asdf/
                __init__.py

That should explain this line "abc.core" = {path = "../abc.core"} for everyone else.

@moigagoo is your question about something that looks like the following?

repo/  # warning: not my structure
    abc.core/ # same as above
        pyproject.toml
        abc
            core/
                __init__.py
    abc.asdf/ # still just a package root (irrelevant name as I understand it)
        pyproject.toml
        abc.asdf/ # I don't think this complies with PEP420?
            __init__.py
    abc.qwerty/
        pyproject.toml
        my_qwerty/ # I don't think this complies with PEP420?
            __init__.py

Yeah, I just don't know that anything besides my current structure is compliant with PEP420

[sdispater]

Poerty support this use case https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages by explicitly declaring the various namespaced packages with the packages property.

[maggyero]

The name of a _distribution_ is declared in the tool.poetry.name property of the pyproject.toml file and appears:

• in the name of the dist/{distribution_name}-{version}-py3-none-any.whl Wheel distribution file and in the name of the {distribution_name}-{version}.dist-info/ directory of the Wheel distribution file created when the distribution is built;
• in the name of the site-packages/{distribution_name}.egg-link symbolic link created when the distribution is installed with edit mode;
• in the name of the site-packages/{distribution_name}-{version}.dist-info/ directory created when the distribution is installed without edit mode;
• in the output of pip list (under the misleading "Package" column which should be rather named "Distribution") when the distribution is installed.

The name of a _root module_ of a distribution is declared in the file system and appears:

• in the name of the {root_module_name}.py file or {root_module_name}/ directory of the Wheel distribution file created when the distribution is built;
• in the name of the site-packages/{root_module_name} source code directory created when the distribution is installed without edit mode.

By default, Poetry looks up the root module to include in a distribution by the distribution name declared in the tool.poetry.name property of the pyproject.toml file, first in the directory of the pyproject.toml file, then in the src/ subdirectory if it exists.

If your root module cannot be found by this default mechanism (the root module is located elsewhere, the root module has a different name than that of the distribution or there are _multiple_ root modules), you have to declare it in the tool.poetry.packages property of the pyproject.toml file, as described in the documentation: add a dictionary item to the list for each root module, with an include property declaring its name as @sdispater said earlier (e.g., foo.py for a non-package module, bar for a package, baz/qux for a package with only its qux subpackage) and, if necessary, a from property declaring the relative path of its parent directory (e.g., src or lib).

For instance, let us say that you need to build three distributions named application_1, application_2 and library with the following repository layouts:

├── .git
├── src
│   └── organization
│       └── application
│           └── application_1
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

├── .git
├── src
│   └── organization
│       └── application
│           └── application_2
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

├── .git
├── src
│   └── organization
│       └── library
│           ├── library_1
│           │   └── __init__.py
│           ├── library_2
│           │   └── __init__.py
│           └── library_3
│               └── __init__.py
└── pyproject.toml

where application_1/src/organization/application/application_1/__main__.py is:

from organization.library.library_1 import f
from organization.library.library_2 import g

print(f())
print(g())

and application_2/src/organization/application/application_2/__main__.py is:

from organization.library.library_3 import h

print(h())

and library/src/organization/library/library_1/__init__.py is:

def f():
    return "foo"

and library/src/organization/library/library_2/__init__.py is:

def g():
    return "bar"

and library/src/organization/library/library_3/__init__.py is:

def h():
    return "baz"

In order to build the distributions, application_1/pyproject.toml should be:

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and application_2/pyproject.toml should be:

[tool.poetry]
name = "application_2"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and library/pyproject.toml should be:

[tool.poetry]
name = "library"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[wallrite]

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

@dfee Do you have any problems building your project? I have the same structure and faced problem with poetry build.

I think I've found related issue #2046

[pawamoy]

@maggyero, just to be clear, in

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

...it is not mandatory to use application_1 as name, right? We could use any name we want, and pip will in any case install the application_1 package in the organization.application namespace'd one.

[maggyero]

@pawamoy No it is not mandatory to use "application_1" as name, you can call your _distribution_ as you want. The binding between the distribution name and the top-level Python module name "application_1" will be recorded in the file top_level.txt of the distribution.

For instance you can see that the setuptools distribution has three top-level Python modules:

$ cat /usr/local/lib/python3.8/site-packages/setuptools-49.2.0-py3.8.egg-ino/top_level.txt
easy_install
pkg_resources
setuptools

Indeed:

$ ls -l /usr/local/lib/python3.8/site-packages/
total 16
drwxr-xr-x   4 maggyero  admin   128 18 jul 13:40 __pycache__
-rw-r--r--   1 maggyero  admin   126 13 jul 14:37 easy_install.py
lrwxr-xr-x   1 maggyero  admin    81 18 jul 13:41 homebrew-protobuf.pth -> ../../../Cellar/protobuf/3.12.3/lib/python3.8/site-packages/homebrew-protobuf.pth
drwxr-xr-x   7 maggyero  admin   224 18 jul 13:40 pip
drwxr-xr-x   8 maggyero  admin   256 18 jul 13:40 pip-20.1.1-py3.8.egg-info
drwxr-xr-x   6 maggyero  admin   192 18 jul 13:40 pkg_resources
drwxr-xr-x  45 maggyero  admin  1440 18 jul 13:40 setuptools
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 setuptools-49.2.0-py3.8.egg-info
-rw-r--r--   1 maggyero  admin  2081 18 jul 13:40 sitecustomize.py
drwxr-xr-x  14 maggyero  admin   448 18 jul 13:40 wheel
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 wheel-0.34.2-py3.8.egg-info

[Luttik]

Edit I made a dumb comment here that was addressed in https://github.com/python-poetry/poetry/issues/356. tl;dr
If you want to use a nested namespace like organization.domain.tool you can use nested folders (organization/domain/tool) and use this in your package.toml

packages = [
    { include = "organization", from = "src" }  # only use from="src" if that is how you've structured your code.
]

poetry will automatically know how to interpret the folders nested under organization.