Warehouse: HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText

[evandrocoan]

Related: https://github.com/pypa/warehouse/issues/5855 The description failed to render in the default format of reStructuredText

$ rm -rf ./dist && python3 setup.py sdist && twine upload dist/*
running sdist
running check
F:\Python\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
warning: check: missing meta-data: if 'author' supplied, 'author_email' must be supplied too

warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

writing manifest file 'MANIFEST'
creating fastfilepackage-1.0.6
creating fastfilepackage-1.0.6\source
creating fastfilepackage-1.0.6\source\cppimplementation
creating fastfilepackage-1.0.6\source\fastfilepackage
making hard links in fastfilepackage-1.0.6...
hard linking LICENSE.txt -> fastfilepackage-1.0.6
hard linking README -> fastfilepackage-1.0.6
hard linking README.md -> fastfilepackage-1.0.6
hard linking setup.cfg -> fastfilepackage-1.0.6
hard linking setup.py -> fastfilepackage-1.0.6
hard linking source/cppimplementation\fastfile.cpp -> fastfilepackage-1.0.6\source/cppimplementation
hard linking source/cppimplementation\fastfilewrapper.cpp -> fastfilepackage-1.0.6\source/cppimplementation
hard linking source\fastfilepackage\__init__.py -> fastfilepackage-1.0.6\source\fastfilepackage
hard linking source\fastfilepackage\version.py -> fastfilepackage-1.0.6\source\fastfilepackage
creating dist
Creating tar archive
removing 'fastfilepackage-1.0.6' (and everything under it)
Enter your username: addons_zz
Uploading distributions to https://upload.pypi.org/legacy/
Uploading fastfilepackage-1.0.6.tar.gz
100%|##########| 19.4k/19.4k [00:01<00:00, 16.1kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

Contents of fastfilepackage-1.0.6.tar.gz:

1. 
2. 

i.e., README.md is present on the package file and its contents are just simple markdown.

$ pip list
Package                  Version    
------------------------ ---------- 
twine                    1.13.0
setuptools               41.0.1
wheel                    0.33.4
...

I only manage to upload the package by commenting out these 2 lines:

        # long_description = readme_contents,
        # long_description_content_type='text/markdown',

https://github.com/evandrocoan/fastfilepackage/blob/master/setup.py

Then, I was able to upload things successfully:

$ rm -rf ./dist && python3 setup.py sdist && twine upload dist/*
running sdist
running check
warning: check: missing meta-data: if 'author' supplied, 'author_email' must be supplied too

warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

warning: sdist: standard file not found: should have one of README, README.txt, README.rst

writing manifest file 'MANIFEST'
creating fastfilepackage-1.0.6
creating fastfilepackage-1.0.6\source
creating fastfilepackage-1.0.6\source\cppimplementation
creating fastfilepackage-1.0.6\source\fastfilepackage
making hard links in fastfilepackage-1.0.6...
hard linking LICENSE.txt -> fastfilepackage-1.0.6
hard linking README.md -> fastfilepackage-1.0.6
hard linking setup.cfg -> fastfilepackage-1.0.6
hard linking setup.py -> fastfilepackage-1.0.6
hard linking source/cppimplementation\fastfile.cpp -> fastfilepackage-1.0.6\source/cppimplementation
hard linking source/cppimplementation\fastfilewrapper.cpp -> fastfilepackage-1.0.6\source/cppimplementation
hard linking source\fastfilepackage\__init__.py -> fastfilepackage-1.0.6\source\fastfilepackage
hard linking source\fastfilepackage\version.py -> fastfilepackage-1.0.6\source\fastfilepackage
creating dist
Creating tar archive
removing 'fastfilepackage-1.0.6' (and everything under it)
Enter your username: addons_zz
Uploading distributions to https://upload.pypi.org/legacy/
Uploading fastfilepackage-1.0.6.tar.gz
100%|##########| 17.3k/17.3k [00:03<00:00, 5.78kB/s]

If I replace distutils.core.setup by setuptools.setup everything works fine without having to remove the long_description and long_description_content_type lines. But I use distutils.core.setup instead of setuptools.setup in my project because I do not know how to build Python C API with setuptools.setup. Everywhere I look on internet just tells me how to build Python C Extensions with distutils.core.setup. No one ever uses setuptools.setup.

[evandrocoan]

I just manage to upload it by replacing:

from distutils.core import setup, Extension

With:

from setuptools import setup, Extension

1. https://docs.python.org/3/library/distutils.html
2. https://docs.python.org/3/extending/newtypes_tutorial.html
3. https://stackoverflow.com/questions/12856931/how-to-put-my-python-c-module-inside-package
4. https://stackoverflow.com/questions/42585210/extending-setuptools-extension-to-use-cmake-in-setup-py

[auvipy]

facing this error

twine upload --repository pypi dist/*
Enter your username: auvipy
ERROR:dbus.proxies:Introspect error on :1.402:/modules/kwalletd5: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying
/usr/local/lib/python3.7/dist-packages/twine/utils.py:238: UserWarning: Failed to open keyring: org.freedesktop.DBus.Error.ServiceUnknown: The name :1.402 was not provided by any .service files.
warnings.warn(str(exc))
Enter your password:
Uploading distributions to https://upload.pypi.org/legacy/
Uploading django_celery_beat-1.5.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 50.5k/50.5k [00:02<00:00, 20.5kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

[spyoungtech]

I was also having this problem suddenly, even though my setup.py and README.rst have not changed since the last successful deploy.

I am using the travis-ci PyPI deploy feature.

In my case, the extended error was as follows (as produced by twine check)

The project's long_description has invalid markup which will not be rendered on PyPI. The following syntax errors were detected:
line 220: Warning: Cannot analyze code. No Pygments lexer found for "guess".
line 220: Warning: Cannot analyze code. No Pygments lexer found for "guess".

Which was odd because this previously worked... and is accepted by sphinx/rtd. Simply removing the specification let the upload work.

Related: spyoungtech/behave-webdriver#77

[di]

Folks arriving at this issue: The cause is that in #5835 we merged a change which blocks uploads that have an invalid long_description that we won't be able to render. This is intentional and not a bug.

Before, the behavior was that if the reStructuredText or metadata was invalid, we would just fall back to a plaintext. Now, to prevent unrendered descriptions on PyPI, we just block the upload instead.

Depending on what format you're using for your long_description, there's two solutions here:

If you're using a reStructuredText long_description:

Your long_description is invalid and will not render. You should upgrade twine:

$ pip install -U twine

And then check your distributions for invalid markup:

$ twine check dist/*

And fix the errors that it reports.

If you're using a Markdown long_description:

The metadata for your distribution is invalid and is not specifying Markdown.

This means that either you haven't set long_description_content_type='text/markdown', in your setup.py file, or that the tools that you're using are out of date and don't support this metadata field. Upgrade them to latest:

$ pip install -U twine wheel setuptools

Remove previous distributions:

$ rm -rf dist

Rebuild your distributions:

$ python setup.py sdist
$ python setup.py bdist_wheel

Check that the distributions are valid:

$ twine check dist/*

[pradyunsg]

@di would it be possible to add this information to the FAQ and link to that from the error message?

[4383]

Greetings,

We facing the same issue on podman and openstack too, I've submit some related fixes:

• https://review.opendev.org/#/c/668459/
• https://github.com/containers/python-podman/pull/37

I continue to facing the same issue when I using twine check dist/* on the podman side even after adding the fix...

I've added the long_description_content_type but the error still to occur:

$ twine check dist/*
Checking distribution dist/podman-0.0.3.dev2-py2.py3-none-any.whl: warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.
Failed
The project's long_description has invalid markup which will not be rendered on PyPI. The following syntax errors were detected:
line 35: Warning: Inline literal start-string without end-string.
Checking distribution dist/podman-0.0.3.dev2.tar.gz: warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.
Passed
The command "twine check dist/*" exited with 1.

It's look like this line is still ignored... Maybe I missing something...

Please can you take a look and double check my proposed changes.

[di]

@4383 I'm not too familiar with pbr but the problem is that it's not writing the long_description_content_type metadata to the distributions you're building, hence:

warning: long_description_content_type missing

Can you share the failing distributions with us? Are you updating everything to the latest possible version?

[4383]

@di first thanks to help us :)

The main issue is that I can't reproduce the issue locally, the issue seems to occur only with travis, when I try to execute the following commands locally everything work fine...

$ pip install -U twine wheel setuptools
$ rm -rf dist
$ python setup.py sdist
$ python setup.py bdist_wheel
$ twine check dist/*
Checking distribution dist/podman-0.0.3.dev1-py2.py3-none-any.whl: Passed
Checking distribution dist/podman-0.0.3.dev1.tar.gz: Passed

The code is the same that my PR... and I can't get the generated artefacts who fail during CI execution...

I've try to update requirements (twine, wheel, etc...) to the latest version like locally and the error still to occur on the travis side.

I know pbr (I'm a core maintainer of it) and I don't think it's an issue here, I've submitted patches like this one on other projects with the same issue and it seems now ok. Locally if I remove the long_description_content_type from the setup.cfg then I can reproduce the issue, so that mean that pbr build the package correctly. If I re-add long_description_content_type to setup.cfg I can see the following entry Description-Content-Type: text/markdown in podman.egg-info/PKG-INFO.

I hope these infos can help you

[di]

@4383 Twine doesn't look at the podman.egg-info/PKG-INFO file, it would be looking at the podman-0.0.3.dev1/PKG-INFO (for a sdist) and podman-0.0.3.dev1.dist-info/METADATA (for a wheel). Do these files have the Description-Content-Type: text/markdown lines? What Metadata-Version do they have?

[daneads]

FWIW, I had the same error even after the suggested fix and after removing my long_description altogether.

I was using the full text of the license as the license parameter, which was then being used as the long_description even if I didn't explicitly specify the long_description.

e.g.:

with open('LICENSE', 'r') as f:
    license = f.read()
...
license=license,

Switched the license to a plain string (LICENSE='GNU GPLv2') and did not receive the error.

[4383]

@di if I keep my long_description_content_type in my setup.cfg then the generated files (podman-0.0.3.dev1.dist-info/METADATA, podman-0.0.3.dev1/PKG-INFO) in dist/ have the Description-Content-Type: text/markdown in their contents.

If I remove long_description_content_type from my setup.cfg then the generated files don't have the Description-Content-Type: text/markdown in their contents and then twin check dist/* fail...

[di]

@4383 What Metadata-Version do they have? Can you share the failing distributions with us?

[4383]

@di Metadata-Version: 2.1.

Sorry I can't share the failing distributions with you, I can't reproduce it locally and the code is identical locally and on the PR.

The only that I can do is to test if I can raise/avoid the error locally by removing/adding long_description_content_type from setup.cfg and it works but it's not really the issue we facing on CI. It only tell to us that normally my fix work well (locally).

From that I know travis doesn't allow us to download generated things, I will try to extract built things and cat the files contents to see config and infos.

https://travis-ci.org/containers/python-podman/builds/553366949?utm_source=github_status&utm_medium=notification

[di]

@4383 See my comment here: https://github.com/containers/python-podman/pull/37/files#r299657105

[Jonathan-LeRoux]

I'm having issues with twine and reStructuredText files. twine check dist/* on a README.rst file in my project (LWS) would keep choking at a URL because that URL had uppercase letters in it, and it appears that twine converts all text to lowercase during the check, breaking the URL:

**Note:** On Windows, the Microsoft Visual C++ Compiler for your version of Python needs to be installed. See `this page<https://wiki.python.org/moin/WindowsCompilers>`_ for more details.

gave the following error:
line 94: Error: Unknown target name: "this page<https://wiki.python.org/moin/windowscompilers>".

I tried using a different hyperlink style:

**Note:** On Windows, the Microsoft Visual C++ Compiler for your version of Python needs to be installed. See the Python wiki WindowsCompilers_ page for more details.
.. _WindowsCompilers: https://wiki.python.org/moin/WindowsCompilers

but then the target name would be lowercased, making it an unknown target:
line 94: Error: Unknown target name: "windowscompilers".

My "workaround" was to move to markdown, but I think it's an issue worth investigating for everyone else.

Here is the version info for twine:
twine version 1.13.0 (pkginfo: 1.5.0.1, requests: 2.21.0, setuptools: 40.8.0, requests-toolbelt: 0.9.1, tqdm: 4.31.1)

[jamadden]

@Jonathan-LeRoux I can duplicate the issue with plain docutils. If I make the changes mentioned here, it works.

In your second example, the problem appears to be a missing blank line between the text and the reference name (.. _WindowsCompilers:). Without that blank line, the reference name is parsed as part of the preceeding paragraph, not as a hyperlink target.

In your first example, the problem appears to be a missing space in the middle of page<https. Without that space, the parser doesn't know that this is supposed to be an embedded URL ("The bracketed URI must be preceded by whitespace and be the last text before the end string.").

[Jonathan-LeRoux]

@jamadden Thanks a lot for looking into this so quickly. I guess I was careless and unlucky both times :)
The fact that the error log was converted to lowercase threw me off on a tangent.

[di]

Going to close this as there's nothing to do here. If you have followed the steps in https://github.com/pypa/warehouse/issues/5890#issuecomment-494868157 and are still experiencing this, please open a new issue.