Setuptools: Execute python code before parsing setup.cfg

[rickstaa]

I'm currently trying to make my python packages PEP517/518 compatible. For this, I'm replacing the setup.py with a setup.cfg file and a project.toml file. This was quite straightforward to do for most of my packages, but for one package, I was not able to do this translation.

Package description

The package in question contains a (parent) python package that includes several packages that can also be installed as stand-alone python packages. For this, I created the following project structure:

    ├── main_package
    │   ├── python_module1
    │   ├── python_module3
    │   └── stand_alone_subpackage
    │       ├── stand_alone_subpackage
    │       │   └── subpackage_module
    │       │       └── script.py
    │       ├── pyproject.toml
    │       ├── requirements.txt
    │       ├── setup.py
    │       └── setup.cfg
    └── setup.py

I use the following setup.py file to install the subpackage in the parent package while getting rid of the redundant stand_alone_subpackage namespace.

from setuptools import setup, find_namespace_packages
import re

requirements = [
    "gym",
    "matplotlib",
    "numpy",
    "torch",
    "joblib",
    "tensorboard",
    "mpi4py",
    "psutil",
    "tqdm",
]

# Retrieve package list
PACKAGES = find_namespace_packages(include=["machine_learning_control*"])

# Add extra virtual shortened package for each of namespace_pkgs
namespace_pkgs = ["simzoo"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

# Run python setup
setup(
    name="main_package",
    version="0.0.0",
    description=(
        "A parent python package."
    ),
    install_requires=requirements,
    packages=PACKAGES,
    package_dir=PACKAGE_DIR,
)

Why this works can be found inside a StackOverflow question I opened some months ago. The main thing it does is adds the virtual shortened modules to the PACKAGES and PACKAGES_DIR variables. This way thePythonClassinside thestandalone_subpackage` can be imported using the following import command:

from main_package.stand_alone_supbpackage import PythonClass
``

instead of using the actual (longer) module namespace:

```python
from main_package.stand_alone_subpackage.stand_alone_supbpackage import PythonClass

Translated setup.py file

I translated the setup.py to the following setup.cfg file:

[metadata]
name = main_package
version = 0.0.0

[options]
packages = find:
include_package_data = True
install_requires =
    gym
    matplotlib
python_requires = >=3.5

I, however, did not yet find a way to inject the extra virtual (shortened) modules before the packages = find: command is parsed. One possible way would be to read the PACKAGES and PACKAGE_DIR variables from a file. This would be similar to using the file: and attr: arguments for the package version parameter. While doing so, this file has to be created before the setup.cfg file is parsed. I, however, did not find a way to do this in setuptools 50.3.2. My question is, therefore, does anybody know a way of achieving the more complicated setup procedure I am trying to achieve?

Update

I just found out that you can combine both the setup.cfg and setup.py code. This allows me to define most of the setup information inside the setup.cfg file while keeping the packages and package_dir arguments inside the setup.py file. This allows me to inject the virtual shortened modules inside the setup.py file. I currently use the following setup.py file:

from setuptools import setup, find_namespace_packages
import re

with open("README.md") as f:
    readme = f.read()

# Add extra virtual shortened package for each of namespace_pkgs
PACKAGES = find_namespace_packages(include=["machine_learning_control*"])
namespace_pkgs = ["simzoo"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

# Run python setup
setup(
    packages=PACKAGES, package_dir=PACKAGE_DIR,
)

If anybody knows if it is possible to put everything inside the setup.cfg file please let me know. Thanks a lot in advance!

[webknjaz]

@rickstaa you cannot put Python code in setup.cfg the only way to achieve what you want is to keep a part of the code you need in setup.py (and put the rest into setup.cfg).

Also, do you know about the find_namespace: directive? Also, doesn't [options.packages.find] section and its options (where, include, exclude) provide you with enough filtering?

[nschloe]

As webknjaz said, a setup.cfg file doesn't allow for much logic at all. If you're getting more complex than normal (and that's what your subpackage setup is), you can always keep a stripped-down setup.py around. It's not unusual either, and necessary for example when you build packages with C++ sources in them. Check out pygalmesh (a package of mine): All metadata in setup.cfg, and the rest of the logic in setup.py. Perhaps this will help.

[rickstaa]

@webknjaz Thanks a lot for your explanation! The solution works when I keep both a setup.py and setup.cfg. I think for now that gives me the result I was aiming to. Namely, giving users the ability to import the main_package.stand_alone_subpackage.stand_alone_subpackage.subpackage_module under a shortened main_package.stand_alone_subpackage.subpackage_module namespace. In this, the redundant folder is needed to be able to give users the ability to install the stand_alone_subpackage as a stand-alone pipy module.

Just out of curiosity after your comment, I also tried achieving the same behaviour using the find_namespace: directive. I, however, was not able to find the right setup.cfg syntax for achieving the desired result. currently use the following setup.py config file:

[options]
include_package_data = True
packages=find_namespace:

[options.packages.find]
where=
    .
    ./main_package/stand_alone_subpackage/stand_alone_subpackage
exclude=
    main_package.stand_alone_subpackage
include=
    main_package.stand_alone_subpackage.stand_alone_subpackage

But this doesn't seem to work because of the following reasons. An example repository of the things I tried can be found here. First as far as I understand it the where option doesn't support a list of paths. Further, although the exclude and include keywords work, they do not provide a way to change the import namespace. I, therefore, don't think I can achieve the desired setup without using the additional logic that is added to the setup.py file. Another way to achieve what I want is to replace all the submodules by name_space packages and update the folder structure to the one that is documented by the setuptools documentation:

├── stand_alone_subpackage_1
│   ├── main_package
│   │   └── submodule_1
│   │       └── __init__.py
│   └── setup.py
└── stand_alone_subpackage_1
    ├── main_package
    │   └── submodule_2
    │       └── __init__.py
    └── setup.py

This would, however make the GitHub folder structure more complicated. I, therefore, came to the following conclusions:

• The best way to achieve what I'm trying to achieve is to have a parent folder that only contains name_space packages and using the folder structure that is recomended by the setuptools documentation.
• If people want another folder structure they are able to do this using a setup.cfg file that contains the metadata and a setup.py file that contains some extra packaging logic.

@webknjaz and @nschloe If you could let me know if you think my conclusions are correct, I would be very grateful! I really appreciate any help you can provide.

@nschloe Also thanks a lot for the example repository and telling me that such a setup is not unusual. I need C++ sources in a further stage of my project, so I think the repository is very helpful.

[webknjaz]

Sounds about right

[rickstaa]

@webknjaz Amazing thanks a lot! I will close the issue for now!