Pip: Pip ignores extras_require when description in setup.py contains \n

Created on 13 Sep 2019 · 5Comments · Source: pypa/pip

Environment

pip version: 19.2.3
Python version: 3.6.8
Poetry version 1.0.0b1

Description
This issue was discovered while publishing docs from undp/MsGraphExporter on ReadTheDocs. When pyproject.toml option description has a multi-line string enclosed in """ (which is perfectly legit for a string in TOML), it causes poetry to build the package with the description parameter in the setup.py file containing string like 'this is\na short description' which apparently causes pip install pkg.whl[docs] to fail picking up extras_require option.

Expected behavior
Having \n in long_description of setup_kwargs in setup.py does not cause such issue. So, I would expect having it in description should not affect pip either.

How to re-create the issue

Have the description in the pyproject.toml as a multi-line value enclosed in """.

# ...snip...
[tool.poetry]
name = "ms-graph-exporter"
version = "0.1.0-rc.1"
description = """A distributed Celery application to export time-domain data
periodically from Microsoft Graph API into a buffer key-value store."""
# ...snip...

Check pyproject.toml syntax with poery.
```
$ poetry check
All set!
```

Build the package with poetry

$ poetry build
Building ms-graph-exporter (0.1.0-rc.1)
- Building sdist
- Built ms-graph-exporter-0.1.0rc1.tar.gz

- Building wheel
- Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

Check setup.py from the package tar.gz to ensure it has all extras_require defined and also has \n in the description.

# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']

package_data = \
{'': ['*']}

install_requires = \
['adal>=1.2,<2.0',
# ...snip...
'typing>=3.7,<4.0']

extras_require = \
{'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
'code-lint': ['flake8>=3.7,<4.0',
# ...snip...
              'pygments>=2.4,<3.0'],
'docs': ['recommonmark>=0.5.0,<0.6.0',
          'sphinx>=2.0,<3.0',
          'sphinx_rtd_theme>=0.4.3,<0.5.0',
          'sphinx-autodoc-typehints>=1.7,<2.0'],
'test': ['pytest>=5.0.1,<6.0.0',
# ...snip...
          'pytest-variables[yaml]>=1.7,<2.0']}

setup_kwargs = {
    'name': 'ms-graph-exporter',
    'version': '0.1.0rc1',
    'description': 'A distributed Celery application to export time-domain data\nperiodically from Microsoft Graph API into a buffer key-value store.',
    # ...snip...
    'extras_require': extras_require,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)

Try to install docs extra requirements with the package.

$ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
WARNING: ms-graph-exporter 0.1.0rc1 does not provide the extra 'docs'
Installing collected packages: ms-graph-exporter
Successfully installed ms-graph-exporter-0.1.0rc1

How to resolve the issue

Ensure pyproject.toml has a single-line description.

# ...snip...
[tool.poetry]
name = "ms-graph-exporter"
version = "0.1.0-rc.1"
description = "A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store."
# ...snip...

Check pyproject.toml syntax with poery.
```
$ poetry check
All set!
```

Build the package with poetry

$ poetry build
Building ms-graph-exporter (0.1.0-rc.1)
- Building sdist
- Built ms-graph-exporter-0.1.0rc1.tar.gz

- Building wheel
- Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

Check setup.py from the package tar.gz to ensure it has all extras_require defined and description without \n.

# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']

package_data = \
{'': ['*']}

install_requires = \
['adal>=1.2,<2.0',
# ...snip...
'typing>=3.7,<4.0']

extras_require = \
{'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
'code-lint': ['flake8>=3.7,<4.0',
# ...snip...
              'pygments>=2.4,<3.0'],
'docs': ['recommonmark>=0.5.0,<0.6.0',
          'sphinx>=2.0,<3.0',
          'sphinx_rtd_theme>=0.4.3,<0.5.0',
          'sphinx-autodoc-typehints>=1.7,<2.0'],
'test': ['pytest>=5.0.1,<6.0.0',
# ...snip...
          'pytest-variables[yaml]>=1.7,<2.0']}

setup_kwargs = {
    'name': 'ms-graph-exporter',
    'version': '0.1.0rc1',
    'description': 'A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.',
    # ...snip...
    'extras_require': extras_require,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)

Install docs extra requirements with the package without any issue.

$ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
[... snip ...]
Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 Pygments-2.4.2
adal-1.2.2 alabaster-0.7.12 amqp-2.5.1 asn1crypto-0.24.0 attrs-19.1.0 babel-2.7.0
billiard-3.6.1.0 celery-4.3.0 celery-redbeat-0.13.0 certifi-2019.9.11 cffi-1.12.3
chardet-3.0.4 commonmark-0.9.0 cryptography-2.7 docutils-0.15.2 future-0.17.1 gevent-1.4.0
greenlet-0.4.15 idna-2.8 imagesize-1.1.0 importlib-metadata-0.22 kombu-4.6.4
more-itertools-7.2.0 ms-graph-exporter-0.1.0rc1 packaging-19.1 pycparser-2.19
pyparsing-2.4.2 python-dateutil-2.8.0 pytz-2019.2 pyyaml-3.13 recommonmark-0.5.0
redis-3.3.8 requests-2.22.0 six-1.12.0 snowballstemmer-1.9.1 sphinx-2.2.0
sphinx-autodoc-typehints-1.8.0 sphinx-rtd-theme-0.4.3 sphinxcontrib-applehelp-1.0.1
sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1
sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 tenacity-5.1.1
typing-3.7.4.1 urllib3-1.25.3 vine-1.3.0 zipp-0.6.0

_Originally posted by @OK-UNDP in https://github.com/readthedocs/readthedocs.org/issues/6151#issuecomment-531048562_

wrong project auto-locked support

Source

OK-UNDP

All 5 comments

The description gets written to the "Summary" field in ms_graph_exporter-0.1.0rc1.dist-info/METADATA. The multi-line version looks like:

Metadata-Version: 2.1
Name: ms-graph-exporter
Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
periodically from Microsoft Graph API into a buffer key-value store.
Home-page: https://github.com/undp/MsGraphExporter
...

I don't think that the metadata is valid. Consider:

The "Summary" field is described as a "one-line summary of what the distribution does" ([ref])(https://packaging.python.org/specifications/core-metadata/#summary)
The only multi-line field within the metadata specification has specific instructions for how to encode it (see here)
Not only is a warning traced about the "missing" extra, but none of the dependencies are installed. I verified this locally after making sure that I hadn't already installed them. So there's a bigger issue parsing that file than just the extras getting missed.

After replacing the multi-line summary with an encoded version like

Metadata-Version: 2.1
Name: ms-graph-exporter
Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
       |periodically from Microsoft Graph API into a buffer key-value store.
Home-page: https://github.com/undp/MsGraphExporter
...

and re-zipping, the resulting wheel seems to be installed correctly (with dependencies and extras).

chrahunt on 13 Sep 2019

👍1

When building with the poetry-generated setup.py using setuptools (by extracting the sdist, deleting the pyproject.toml, and running pip wheel .), the generated wheel has a METADATA file with:

Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
Requires-Dist: adal (<2.0,>=1.2)

Truncating like this may be better than encoding like I did above, since any other tools may be expecting literally a single line. If you file an issue with Poetry you may want to mention it.

chrahunt on 13 Sep 2019

👍1

@chrahunt, thanks fort digging deeper into this! So, it seems I'd better file an issue with Poetry to truncate multi-line descriptions, or flagging it by running poetry check.

OK-UNDP on 13 Sep 2019

You did most of the work - it was very clear what to check. :) Thank you for that.