twine check does not notice long_description problems with markdown and old bdist_wheel

[mgedmin]

Your Environment

Thank you for taking the time to report an issue.

To more efficiently resolve this issue, we'd like to know some basic information about your system and setup.

1) Your operating system: Ubuntu 18.10

2) Version of python you are running:

python --version
Python 2.7.15+

3) How did you install twine? Did you use your operating system's package manager or pip or something else?

virtualenv ~/.venv && ~/.venv/bin/pip install -U twine

4) Version of twine you have installed (include complete output of):

twine --version
twine version 1.12.1 (pkginfo: 1.4.2, requests: 2.20.0, setuptools: 40.5.0,
requests-toolbelt: 0.8.0, tqdm: 4.28.1)

5) Which package repository are you targeting?

I don't understand the question.

The Issue

The https://pypi.org/project/magic-wormhole project was trying to use markdown on the PyPI page, without success: https://github.com/warner/magic-wormhole/issues/316.

I was hoping twine check would help identify the issue, but it didn't.

I would like twine check to be useful for these kinds of problems.

Steps to Reproduce

If the issue is predictable and consistently reproducible, please list the steps here.

• pip download --no-deps magic-wormhole==0.11.1
• twine check magic_wormhole-0.11.1-py2.py3-none-any.whl

My expectation: twine will say "you're trying to use Description-Content-Type: text/markdown, but this wheel was generated with bdist_wheel (0.29.0) which does not support it (you need wheel >= 0.31.0)", or it will at least list the specific problems that prevent this project's long_description from being rendered correctly.

Actual results:

Checking distribution magic_wormhole-0.11.1-py2.py3-none-any.whl: Passed

[di]

I'm not sure this is something that Twine should be able to do.

Twine only concerns itself with distribution artifacts, not source. From it's perspective, your wheel _is_ correct (since it doesn't have the Description-Content-Type field in the metadata).

In order for it to understand that the wheel _should_ have that field, it would have to execute setup.py (i.e. do the build step, which is what setuptools is for).

[mgedmin]

Twine only concerns itself with distribution artifacts, not source. From it's perspective, your wheel is correct (since it doesn't have the Description-Content-Type field in the metadata).

But it does!

Description-Content-Type: text/markdown

is right there in magic_wormhole-0.11.1.dist-info/METADATA in the whl file.

[di]

Right, but the Metadata-Version is 2.0, which is not a version that supports the Description-Content-Type field, so it's ignored.

By the way, here's the output I see with twine check on that distribution, which does produce a warning:

$ pip download --no-cache --no-deps magic-wormhole==0.11.1
Collecting magic-wormhole==0.11.1
  Downloading https://files.pythonhosted.org/packages/46/b7/7d017ea2b8656d44d7480d08406c0fc61dc6f04369fed6415e7a51a87aa2/magic_wormhole-0.11.1-py2.py3-none-any.whl (131kB)
    100% |████████████████████████████████| 133kB 1.8MB/s
  Saved ./magic_wormhole-0.11.1-py2.py3-none-any.whl
Successfully downloaded magic-wormhole

$ twine check magic_wormhole-0.11.1-py2.py3-none-any.whl
Checking distribution magic_wormhole-0.11.1-py2.py3-none-any.whl: warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.
Passed

$ twine --version
twine version 1.12.2 (pkginfo: 1.4.2, requests: 2.19.1, setuptools: 40.5.0,
requests-toolbelt: 0.8.0, tqdm: 4.25.0)

[mgedmin]

Right, but the Metadata-Version is 2.0, which is not a version that supports the Description-Content-Type field, so it's ignored.

Will a poor random Python packager know these subtleties? (No.)

Wouldn't it be nice if twine warned you about fields that are present but will be ignored because of the metadata version?

By the way, here's the output I see with twine check on that distribution, which does produce a warning:

Checking distribution magic_wormhole-0.11.1-py2.py3-none-any.whl: warning: long_description_content_type missing. defaulting to text/x-rst.

That's helpful! Now people can google that error message and eventually there'll be a StackOverflow answer telling them to upgrade their wheel package.

(I assume 1.12.2 is the number of the unreleased twine version from git, since PyPI has only 1.12.1?)

I think this issue can be closed -- while there still are possible improvements, "defaulting to text/x-rst" will be a big hint to users that something went wrong about their attempt to use markdown.

[di]

(I assume 1.12.2 is the number of the unreleased twine version from git, since PyPI has only 1.12.1?)

Ah, yes, apologies. I didn't realize this hasn't been released yet. 😕

[mgedmin]

Ah, yes, apologies. I didn't realize this hasn't been released yet. 

:)

This is why I like to keep the version number at 'x.y.z.dev0' in source control, except for the one commit that I tag as the x.y.z release.

[sigmavirus24]

Yeah, someone who shouldn't be committing to the repo rogue committed a bunch of junk to master including the version bump. This isn't the project's normal way of keeping versions on master

[webknjaz]

This is why I like to keep the version number at 'x.y.z.dev0' in source control, except for the one commit that I tag as the x.y.z release.

This is why I use setuptools-scm plugin and avoid hardcoding version in source at all :)

[ocefpaf]

I have a similar issue and I believe I have the all latest versions of the packages.

> twine --version
twine version 1.13.0 (pkginfo: 1.5.0.1, requests: 2.21.0, setuptools: 40.8.0,
requests-toolbelt: 0.9.1, tqdm: 4.31.1)

and readme-renderer 4.0.

I'm trying to check my project's [1-2] README with

python setup.py bdist_wheel && twine check dist/*

and I get

warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.

However, my setup.py has the line

long_description_content_type="text/markdown",

my WHEEL file is

Wheel-Version: 1.0
Generator: bdist_wheel (0.33.1)
Root-Is-Purelib: true
Tag: py3-none-any

and the METADATA file has the line Description-Content-Type: text/markdown as expected.

Maybe I'm missing something obvious but I have no idea how to proceed to investigate this. As you can see in [2] the README renders badly and I cannot debug why b/c locally it is checked as text/x-rst.

[1] https://github.com/ioos/erddapy
[2] https://pypi.org/project/erddapy/

[di]

@ocefpaf Is it possible you have old distributions without this metadata in the dist directory? Can you try removing it and re-running the commands?

[ocefpaf]

@ocefpaf Is it possible you have old distributions without this metadata in the dist directory? Can you try removing it and re-running the commands?

Nope. I start that command with rm dist/*, I left it out above for simplicity.

[jamadden]

It looks like your setup.py has license=f'{read("LICENSE.txt")}',. LICENSE.txt is a multiple-line file, but the license keyword argument is supposed to just be the name of the license. So the generated metadata is probably incorrect. (Essentially https://github.com/pypa/twine/issues/454)

[ocefpaf]

It looks like your setup.py has license=f'{read("LICENSE.txt")}',.

That is it. Thanks! I would never have guessed based on the text/x-rst warning.

LICENSE.txt is a multiple-line file, but the license keyword argument is supposed to just be the name of the license. So the generated metadata is probably incorrect. (Essentially #454)

I guess that I should add a

[metadata]
license_file = LICENSE.txt

to the setup.cfg to get the license text in then?

PS: I always "corrupted" the use of the LICENSE field in the past to actually have a license text, specially in the BSD case b/c I really need the type of BSD (1, 2 or 3) to be there, instead of the PyPI classifier License :: OSI Approved :: BSD License, which omits the license version.

[di]

Given that this issue seems to be collecting unrelated problems, and that @mgedmin said:

I think this issue can be closed -- while there still are possible improvements, "defaulting to text/x-rst" will be a big hint to users that something went wrong about their attempt to use markdown.

I'm going to close this. If you think you have a similar problem, please open a new issue.