Warehouse: long_description_content_type=text/markdown doesn't always work

[simonw]

This page does not show correctly rendered markdown: https://pypi.org/project/datasette/

The setup.py file looks like this:

https://github.com/simonw/datasette/blob/fb988ace7c7e2bee5ac142a0eab22431d0675a77/setup.py#L13-L18

setup(
    name='datasette',
    description='An instant JSON API for your SQLite databases',
    long_description=get_long_description(),
    long_description_content_type='text/markdown',
    author='Simon Willison',

According to @ambv on Twitter this is caused by a bug: https://twitter.com/llanga/status/984866099886895104 "There's a bug now where you need to upload just the tar.gz first with Twine and only then upload the rest of the files."

[ambv]

@di will know which bug I'm talking about.

[di]

Yep, this is due to building with an old version of wheel which incorrectly sets the Metadata-Version field. Should be fixed by a pip install wheel==0.31.0 and then building/uploading a new distribution.

[simonw]

That worked - I ran pip install wheel==0.31.0 before doing python setup.py bdist_wheel and the result was this page with the markdown correctly formatted: https://pypi.org/project/datasette/

[rgieseke]

I might have a similar problem: https://pypi.org/project/shortcountrynames/
The markdown rendering worked for me in other projects and I think I have wheel=0.31.0

Requirement already up-to-date: wheel in ./venv/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (0.31.0)

Repo: https://github.com/rgieseke/shortcountrynames

[di]

@rgieseke Lookng at the latest release for that project, the Description-Content-Type field is not making it to PyPI.

Can you confirm that you have setuptools>=38.6.0 and twine>=1.11.0 as well?

[rspeer]

I have the same problem: https://pypi.org/project/ConceptNet/5.6.2/ . Repo: https://github.com/commonsense/conceptnet5

I have setuptools 39.0.1, twine 1.11.0, and wheel 0.31.0, and I ran python setup.py sdist upload.

I even deleted ConceptNet 5.6.1 and incremented the version to 5.6.2 when I updated all the above dependencies. However, I'm not just going to keep incrementing the version number for the sake of PyPI.

[rgieseke]

@di Updating setuptools did the trick! Thanks for the quick reply and so much for the PyPI update!

@rspeer You can use the https://test.pypi.org/ instance to test as well ... (and delete tags later or re-try)

[rspeer]

Apparently python setup.py sdist upload is the problem. I will open a new issue about the fact that it's extremely non-obvious that you have to use "twine". (I know there's documentation that says so... but it's not the documentation that's linked to when you're looking for how to set your project description.)

[di]

@rspeer Can you link us to the documentation that showed you to use python setup.py sdist upload?

[rspeer]

https://docs.python.org/3.6/distutils/packageindex.html#the-upload-command

Also a decade of habit. If you want to change people's habits, you could at least _tell_ them they need to change.

[rspeer]

To clarify something: I'm not claiming that PyPI is linking to the Python documentation. PyPI links to packaging.python.org, of course. However, one way that people learn about PyPI in the first place is the Python documentation, where PyPI is described as that place that setup.py upload uploads to.

If you want to update people's understanding about how to upload Python packages, talking about it on your own blog and mentioning Twine twice on a very long documentation page isn't enough. You need to make getting the new tools part of the new PyPI user experience.

[osilkin98]

I actually have this problem right now, and using both wheel==0.31.0 and wheel==0.32.1 (latest at the time of posting) both break the markdown formatting

[di]

Hi @osilkin98, can you link to the release with a broken description? Can you tell us how you uploaded the release to PyPI?

[osilkin98]

@di I was typing out a response documenting how to replicate the issue and in doing so I realized that the version of wheel that was being used during the setup was installed to python3.6's packages, but the one I installed and upgraded was installed to python3.7.

I fixed the issue simply by verbosely running python -m pip install --upgrade wheel in my virtual environment. My apologies for any confusion.

[scruffystuffs]

This problem is still occurring when running python setup.py sdist bdist_wheel and twine upload from a windows machine.

Tested ubuntu 18.04 and windows 10. Ubuntu uploads everything fine, but running the exact same commands from a windows machine causes the above issue to appear with the exact same files.

setuptools, wheel, and twine are all at or beyond the levels specified in the above comments, and are at the same level for both machines.

[di]

@scruffystuffs Can you point me to a distribution on PyPI that you've uploaded that has this problem?

[scruffystuffs]

Uploaded two example cases to test.pypi.org.

Uploaded from Windows
Uploaded from Ubuntu

The files for these distributions should be identical, other than the version number, yet they render entirely differently on the project page.

If any more info is needed, let me know and I will be happy to provide it.

[di]

@scruffystuffs Looking at the metadata of the source distribution of your package, it has Windows-style line endings in the Description, so it looks like you've having the same issue as https://github.com/pypa/setuptools/issues/1440

[wesinator]

I'm still experiencing this issue trying to upload a package on Linux (Ubuntu):

Python 3.6.8
pip 19.3.1
twine 2.0.0
setuptools 41.6.0
wheel 0.33.6

setup.py

      long_description=readme,
      long_description_content_type="text/markdown",

running:

python3 setup.py sdist bdist_wheel
python3 -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*

error

HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

@di please advise

[di]

@wesinator Can you attach the distributions you're trying to upload?

[wesinator]

@wesinator Can you attach the distributions you're trying to upload?

attached:
domainbigdata-0.1.tar.gz
domainbigdata-0.1-py3-none-any.whl.zip

[jamadden]

The problem is likely in your license= argument. It seems to include the entire text of the GPL v3, which contains double newlines. That breaks setuptools by rendering the metadata invalid and unparseable.

It's possible to indicate that the license is GPL v3 in a machine-readable way using a classifier such as License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) (which I see this project includes). The license= argument can then be omitted (or a simple summary), as it is meant to be

Text indicating the license covering the distribution where the license is not a selection from the “License” Trove classifiers"

[wesinator]

The problem is likely in your license= argument. It seems to include the entire text of the GPL v3, which contains double newlines. That breaks setuptools by rendering the metadata invalid and unparseable.

It's possible to indicate that the license is GPL v3 in a machine-readable way using a classifier such as License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) (which I see this project includes). The license= argument can then be omitted (or a simple summary), as it is meant to be

Text indicating the license covering the distribution where the license is not a selection from the “License” Trove classifiers"

@jamadden I already have that exact classifier - https://github.com/wesinator/domainbigdata-python/blob/c4c3b12a37ba6e86931f0c4575406cf42cdcd090/setup.py#L23

also, not sure why the error would say description when the issue is actually the license.
If that's true then the error needs to be way more granular so I know what the f is actually happening.

[di]

@wesinator It's a bug: https://github.com/pypa/setuptools/issues/1390

[wesinator]

The problem is likely in your license= argument. It seems to include the entire text of the GPL v3, which contains double newlines. That breaks setuptools by rendering the metadata invalid and unparseable.

btw, the license was pulled directly from https://www.gnu.org/licenses/gpl-3.0.txt

Ok, removing license seems to have fixed it.
@jamadden thanks for the tip