Poetry: Can not exclude package data file only for the wheel

[sinoroc]

There seems to be an issue when trying to exclude package data files of the wheel only.

Example project here: https://github.com/sinoroc/poetry-gh-2015

Project file structure:

$ tree Thing
Thing
├── CHANGELOG.rst
├── LICENSE.txt
├── pyproject.toml
├── README.rst
├── test
│   └── test_unit.py
└── thing
    ├── data
    │   ├── file.all
    │   ├── file.bin
    │   ├── file.not
    │   └── file.src
    └── __init__.py

All files are in the git repository except *.bin data files (supposed to be a build artifact).

We want the *.src data files and CHANGELOG.rst in the _sdist_ only. We want *.bin data files in the _wheel_ only. We want *.all data files in both the _sdist_ and the _wheel_. We do not want any of the .not data files in the distributions.

We also want the test package, but only in the _sdist_.

.gitignore

/thing/data/*.bin

pyproject.toml:

packages = [
    { include = 'thing' },
    { include = 'test', format = 'sdist' }
]
include = [
    { path = 'CHANGELOG.rst', format = 'sdist' },
    { path = 'thing/data/*.bin', format = 'wheel' },
    { path = 'thing/data/*.src', format = 'sdist' },
]
exclude = [
    { path = 'CHANGELOG.rst', format = 'wheel' },
    { path = 'thing/data/*.src', format = 'wheel' },
    'thing/data/*.not',
]

Content of the _sdist_:

$ python3 -m tarfile -l dist/Thing-0.1.0.tar.gz 
Thing-0.1.0/CHANGELOG.rst 
Thing-0.1.0/LICENSE.txt 
Thing-0.1.0/README.rst 
Thing-0.1.0/pyproject.toml 
Thing-0.1.0/test/test_unit.py 
Thing-0.1.0/thing/__init__.py 
Thing-0.1.0/thing/data/file.all 
Thing-0.1.0/thing/data/file.src 
Thing-0.1.0/setup.py 
Thing-0.1.0/PKG-INFO 

Content of the _wheel_:

$ python3 -m zipfile -l dist/Thing-0.1.0-py3-none-any.whl 
File Name                                             Modified             Size
thing/__init__.py                              1980-01-01 00:00:00            0
thing/data/file.all                            1980-01-01 00:00:00            0
thing/data/file.bin                            1980-01-01 00:00:00            0
thing/data/file.src                            1980-01-01 00:00:00            0
thing-0.1.0.dist-info/LICENSE.txt              1980-01-01 00:00:00            8
thing-0.1.0.dist-info/WHEEL                    2016-01-01 00:00:00           83
thing-0.1.0.dist-info/METADATA                 2016-01-01 00:00:00         1515
thing-0.1.0.dist-info/RECORD                   2016-01-01 00:00:00          577

Somehow the thing/data/file.src appears in the _wheel_, which is unexpected and not what we want.

Example project: https://github.com/sinoroc/poetry-gh-2015

[finswimmer]

exclude = [
    { path = 'CHANGELOG.rst', format = 'wheel' },
    { path = 'thing/data/*.src', format = 'wheel' },
]

At the moment this syntax isn't support. exclude must be a list of path(globs). We should add the ability to work just like include.

fin swimmer

[zoj613]

@sinoroc Have you found a workaround for this yet? I am need of a similar strategy but it seems like no files are excluded conditional on the type of packaging.

[zoj613]

A workaround I came up with is to just only include the necessary files only in package, then use include to add files based on if they should be in sdist or wheel distributions.

[sinoroc]

@zoj613 What would you change in my example to make it work with your workaround?

[zoj613]

@sinoroc Here is how I would write the pyproject.toml

name = 'thing'
...

packages = [
    { include = 'thing/*.py' },
    { include = 'thing/**/*.all' }
]
include = [
    { path = 'CHANGELOG.rst', format = 'sdist' },
    { path = 'thing/data/*.src', format = 'sdist' },
    { path = 'test', format = 'sdist' },

    { path = 'thing/data/*.bin', format = 'wheel' },
]
exclude = ['thing/**/*.not']

This approached worked for me. Basically, include everything thats required by both distributions in packages, then use include to selectively add files based on format. The use exclude to ignore files in both distributions.

[zoj613]

@sinoroc here is what I did for my project. It works just as expected https://github.com/zoj613/htnorm/blob/main/pyproject.toml

[sinoroc]

At this point... What is the difference between packages and include?

[zoj613]

At this point... What is the difference between packages and include?

my guess is that packages determines the base files that get included regardless of your distribution format. Think of it as the opposite of exclude. include is then used to selectively include files that are not python modules and also allows you to select them based on distribution format. I think if you just include non-python files in packages list then you will get a "not a python module error" or something like that.

[sinoroc]

packages determines the base files that get included regardless of your distribution format

But, _packages_ also allows to select target distribution formats. It's also OK if there are 2 ways to achieve the same thing. It's a bit confusing though... Because as far as I understood, packages and include achieve the same things, except that packages also accepts from.

[zoj613]

packages determines the base files that get included regardless of your distribution format

But, _packages_ also allows to select target distribution formats. It's also OK if there are 2 ways to achieve the same thing. It's a bit confusing though... Because as far as I understood, packages and include achieve the same things, except that packages also accepts from.

Yeah I guess the maintainers will be better at explaining this than me. I only just worked from my own understanding based on trial and error. Maybe the functionality of these needs better documentation.

[mmerickel]

The wheel should really only include python package files, and anything outside of there should be excluded by default. This is how setuptools works with MANIFEST.in and include_package_data=True. I shouldn't need to explicitly exclude files from the wheel that are not in the package.

I ran into this while trying to convert Pyramid's scaffolds into poetry and there is not a way right now to get the sdist/wheel to match with how they are built with setuptools because the wheel includes lots of erroneous files like .coveragerc, development.ini, pytest.ini, etc at the root of the repo (stuff that we do want in the sdist).