Warehouse: Add markdown support

[nlhkabu]

I've been working styling the information panel (issue #801).

One thing that I've noticed is that a number of package long descriptions are not coming through with the correct formatting:

The first is clearly markdown, the second, I am not sure, but I have noticed a number of other packages using the same syntax.

Obviously, it would be great to render these out to HTML so the new styles will work.

[nchammas]

The first is Markdown (as you pointed out) and the second is reStructuredText.

The current PyPI does not render long descriptions in Markdown. For example, here is the project from your first screenshot on the current PyPI.

There is a related discussion over on the PyPI repo that you should probably look at. It seems there is some stuff that needs to be figured out in order for PyPI to be able to support multiple formats for the long description. Currently, only RST is supported.

Here is the project from your second screenshot on the current PyPI. The long description renders correctly since it's in RST.

[nlhkabu]

Thanks @nchammas - looks like I walked into a minefield on that one :P

So from that thread I take it that Markdown support is unlikely in the short term.

On the RST side - do we have plans to support this (ping @dstufft)? I assume we will want to do this to provide parity with the current site?

[dstufft]

Hmm, we should be using the exact same code to render descriptions as PyPI is. Sounds like a bug of some sort.

[dstufft]

Looking at this again, it appears RST rendering is working the same as it does in the legacy site. I think the confusion comes from the fact that our test data comes from TestPyPI _not_ from PyPI so you need to compare to https://testpypi.python.org/pypi/django-excel-response3/ which also has broken display. My guess is they fixed some broken RST and didn't update TestPyPI with the fix.

[dstufft]

I'm going to drop this from "Become PyPI". We don't currently support Markdown and actually doing so properly is more effort than it's worth to try and get it into our initial launch.

[nlhkabu]

Ok, this makes sense... thanks for the follow up @dstufft

[boxed]

I've recently had a lot of problems getting the rST to work on pypi. Some problems were fixed by running the check command in setup.py, but one error I had didn't show up there: I had the string 'foo__' which is a reference in rST. This error didn't show up when running the check command so I had to do a binary search of the description to find it :P

Clearly some work on improving the error checking on this would be nice, but isn't strictly a part of warehouse although warehouse suffers from it.

[hadim]

Could you elaborate a bit more about why it's a lot of effort to support Markdown ?

A lot of pypi packages are hosted on github where they have a Markdown README file. Maintaining an alternative README file in .rst for pypi seems to be a lot of work for nothing... That is why most of developers (me included) don't bother converting the description field and let it as Markdown.

It follows that a lot of packages does not have a nicely formated page on pypi.

As the future platform for Python packages, don't you think it's worth trying to put some effort in it ?

[hadim]

I just see that some work is undergoing there https://github.com/pypa/readme_renderer/pull/3

[hadim]

Please see https://github.com/pypa/python-packaging-user-guide/pull/258 for a more complete overview.

[Lucas-C]

By reading this thread, it seems to me this feature is pending on the completion of the PR mentioned by @hadim ?
https://github.com/pypa/readme_renderer/pull/3
as this README rendereding lib is used by warehouse: https://github.com/pypa/warehouse/blob/master/requirements/main.in#L37

Is that correct ?

[alonisser]

+1 fo markdown.. I would like to help with coding if supporting markdown is an option

[9999years]

I’ll also help implement support for Markdown or other languages! I think one of Github’s strengths is strong markup-language support. (Github currently supports nine different languages!)

I think it would make sense to implement the documentation as a subset of HTML rather than any one markup language; one of RST’s big problems is that there’s no support for nested inline styles, such as code links or bold italics, and we could avoid the inflexibilities of any particular markup language by supporting several or a superset of all of them.

[pradyunsg]

pypa/readme_renderer#51 implements Markdown Support and making it easier to add support for more languages in the future (I guess?).

It still needs to be reviewed AFAICT.

[gemerden]

Ehm, while understanding that feature requests can lead to discussion which can lead to better solutions, i am a bit lost on why this is complicated, especially compared to the value it adds. Seems that find a way to indicate what format to render and render it that way does not have to take 3 years.

I have seen loads of suggestions for solutions (e.g. add long_description_format to setup) that might for some reason or other not be perfect, but all are better than the current state.

My personal experience:

• oh, pypi does not render markdown,
• hm, google, maybe there is an automatic converter,
• read about pypandoc,
• install pypandoc,
• change an run setup.py,
• re-upload to pypi,
• check doc, hm, mostly ok, but conversion is not perfect,
• change markdown,
• 2 or 3 versions further: all good (i think, getting tired of rereading the readme).

And i am pretty sure that pypandoc is the easiest way to go and this took me about 3 hours for a simple project. Multiply by number of uploaders from github ...

Anyway, bit ranty, sorry about that, is there an eta yet?

[dstufft]

It's Open Source Software, the ETA is that it'll get done whenever someone has to motivation to do it. Someone took the effort to go through and get agreement on the mechanism to indicate what format to render things in, now someone needs to go through and add support for that to setuptools, twine, and Warehouse.

[ghost]

@dstufft What about pypa/readme_renderer#51?

[yohanboniface]

@dstufft @nlhkabu seems this is the last bit to have markdown support, can you give some pointer to what needs to be done here so we can give a hand?

Thanks :)

[di]

@yohanboniface Currently Markdown support in Warehouse is blocked by the following things:

• [x] A PR to Warehouse to use the correct renderer depending on the description_content_type field
• [x] An update to setuptools (Open PR is here)
• [x] An update to twine (There is a PR here)
• [x] A PR to Warehouse to accept the new fields (There is a PR open here); which is blocked on:
• [x] An update to pkginfo (pkginfo==1.4.2 is out); which is blocked on:
• [x] A PEP to specify the Version 2.1 Metadata (PEP-566 has been accepted)

~I'm currently working on the PEP, and hope to have it finished soon, to unblock the rest of the changes necessary.~
~The PEP is currently in draft status.~
~The PEP has been accepted, currently working on pkginfo updates.~
~Currently waiting on a merge & release forpkginfo~
~Waiting on a new release of pkginfo~
~Waiting on a merge/release in twine and setuptools~
~Just waiting for a pending release of twine and the final PR to Warehouse~
Markdown Descriptions on PyPI are live! 🎉

[yohanboniface]

Great news :)
So if I summarize, once the PEP is out and blessed, everything else is ready?
Is there anything we can do to help on the PEP side?

[dstufft]

I don't think we need a PEP, we did an adhoc spec for it already: https://packaging.python.org/specifications/core-metadata/#description-content-type-optional

The primary blockers at this point is just plumbing that new value through the various tools.

[yohanboniface]

The primary blockers at this point is just plumbing that new value through the various tools.

According to our tests, if you add the long_description_content_type argument to setup, and if you use a setuptools >= 36.4.0, the Description-Content-Type ends up in the PKG-INFO, see for example this package in test.pypi

So it seems like it's a matter of passing this metadata to readme_renderer, which is already capable since https://github.com/pypa/readme_renderer/pull/55 AFAIK.

So is https://github.com/pypa/warehouse/pull/2373 that missing bit? Or is there something else that needs to be done to consume that metadata when available?

[di]

@yohanboniface The issue isn't getting it _into_ the PKG-INFO, it's getting it out. Twine uses the pkginfo package to extract the metadata and pass it to Warehouse. The maintainer(s) of pkginfo didn't want to add the Version 2.1 metadata without a PEP.

Or is there something else that needs to be done to consume that metadata when available?

You're right, there will also need to be a small PR to use the fields added in #2373 to determine the renderer readme_renderer uses, but that will have to come after #2373 is merged.

[9999years]

Great! @di, please let me (or others!) know if there’s anything you need help with in getting the PEP ready; I know several of us are more than happy to help.

[boxed]

Why can't PyPI try to render as markdown if the normal renderer fails? The default now to fall back to plain text is almost always wrong.

[dstufft]

I've mentioned it elsewhere, but the reason is because it's just a crappy user experience.

There exists documents that are both valid markdown and valid rST but which render completely differently in the two systems. So if you have a description that is written in Markdown, you have to be careful to ensure that not only is it valid markdown, but it is invalid rST. This is likely to lead to having to have some silly little syntax quirk laying around somewhere in your README that exists entirely to fool the parser. This problem gets worse for each additional parser we add into the chain.

There is another, related problem that for many people they don't know that their description isn't going to render correctly until after they've uploaded to PyPI. It would be great if PyPI could validate this field like it validates other fields, and reject the upload if it fails. However we can't do that with the fallback behavior as it currently stands, because the fallback behavior will always succeed in the end. Now this isn't strictly related to allowing markdown rendering, but since the solution to the first problem is best served by allowing a project to indicate what language their description is written in, that means that same solution unblocks the ability to start validating descriptions on upload. Thus making it a sort of kill two birds with one stone kind of issue.

Ultimately we've already decided on the mechanism that this will work with, somebody just needs to do the work to plumb it through the various projects.

[jleclanche]

@dstufft The main problem seems to come down to pip/pypi only accepting a single description text field. The user experience shouldn't have to be crappy because of that internal detail. The user experience should be, user has README.rst -> renders in rst, user has README.md -> renders in markdown.

Legacy behaviour can be retained, but to be honest, I wonder if that's even a good idea. RST stops rendering on the tiniest of issues which, compared to Github is a very poor user experience.

Personally, I've given up on using RST and just use a README.md on all my projects. PyPI is just a file host at this point and I point users to Github so they can read a documentation intro document that doesn't require me to run a test suite on...

[dstufft]

I mean, the agreed upon solution is quite literally to add a second field that indicates if it's rst, markdown, or plaintext. We're not going to read a file from the tarball (that file doesn't even exist inside of a wheel for instance) over using the structured metadata that already exists.

Like I said, the mechanism to support that has already been agreed upon, all that is required now is someone to put in the effort to plumb it through the various tools. The people working on these tools are largely volunteers or have other priorities at this moment, so you can either wait for this to bubble up to the surface of those people, or well:

[ghost]

I don't understand what's needed:

1. The pkginfo PR isn't blocked, it's just waiting for @tseaver to hit the merge button.
2. twine already monkeypatched support for METADATA 2.0, and given the pain this is causing, I don't see a reason not to monkeypatch pkginfo in twine (especially since launchpad/bzr is difficult to use).
3. Surely the PR for warehouse is backwards-compatible with older twine versions? Meaning that it could be merged now (if it does what it claims to do).

[di]

A quick update: the PEP is in draft status here: https://www.python.org/dev/peps/pep-0566/

There is an RFC here: https://mail.python.org/pipermail/distutils-sig/2017-December/031788.html

[ncoghlan]

PEP 566 has been accepted, so tool authors are now likely to accept the PRs adding Description-Content-Type support :)

[sayanarijit]

Hi All

Is there any reason we are not using client-side JavaScript to automatically detect the format and render it?

[JulienPalard]

@sayanarijit see https://github.com/pypa/readme_renderer/issues/1 and https://github.com/pypa/readme_renderer/pull/51, TL;DR: "Since we now have a way for the user to explicitly describe the content-type of their README, being able to infer it isn't necessary.".

Also, infering is hard, and can fail. FWIW see https://github.com/JulienPalard/mdorrst

[sayanarijit]

@JulienPalard Thanks for the information. I was not aware of this. However I just tried updating one of my packages listed on pypi: pidlock after mentioning 'Description-Content-Type' as 'text/markdown' using setuptool's long_description_content_type argument. But it seem it failed to convert the markdown.

[JulienPalard]

@sayanarijit It's still WIP (note this issue is still opened) but as the PEP has been accepted we're now sure it's the right way to do it, it's a good first step.

[di]

With https://github.com/pypa/warehouse/pull/3296 merged and a pre-release of twine available, it is now technically possible to publish descriptions on PyPI using Markdown. 🎉

For example, see https://pypi.org/p/markdown-description-example/

Here's a step-by-step guide with everything necessary to make this work for your new or existing project: https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

I'll leave this issue open until @brainwane makes the official twine release is made on Monday. Please feel free to comment here if you try this and are having issues.

[brainwane]

I'd love for people to start preparing PRs for documentation improvements for the sample project used in the official package distribution guide, and for that guide itself.

[brainwane]

Also, per Dustin's guide, if you're testing this, make a source distribution (sdist), not a wheel. Wheel support may depend on pypa/wheel#231 being resolved.

[sfdye]

@di Will this work now if we deploy to pypi from travis?
https://github.com/PyGithub/PyGithub/blob/master/.travis.yml

[9999years]

I’ve created a PR for the sample project (pypa/sampleproject#66) and I’m working on one for the guide as well. For everyone’s reference (from @di’s blog post) twine>=1.11.0 and setuptools>=38.6.0 are required for Markdown support.

[brainwane]

Thanks for your PR, @9999years! Versions: pkginfo 1.4.2, Twine 1.11.0 (right now 1.11.0rc1 will work), setuptools 38.6.0, and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0. Dustin's blog post mentions some of those. I am not sure about Python versions (distutils being part of the Python standard library). Twine 1.11.0 requires the relevant version of pkginfo.

There's an open issue for the packaging docs.

@sfdye I see you're creating both a wheel and a source distribution but it looks like you only upload an sdist so I think you'll be fine, but of course I defer to @di here. (I'm experimenting with Markdown support with this test project and Markdown support works if I upload only an sdist, but if I upload a wheel, or an sdist and a wheel, then Markdown support does not yet work.)

Those of you who want to use GitHub-flavored Markdown (beyond what's in the CommonMark specification), please go over to pypa/packaging-problems#126 to find out what's necessary, again, in a few different codebases.

[9999years]

Okay, the guide PR is at pypa/python-packaging-user-guide#457.

[ncoghlan]

For the time being, we don't have any plans for distutils itself to support metadata versions beyond 1.1 - the guidance for a number of years has been to use setuptools instead (regardless of Python version), and this is just one more reason to continue that recommendation.

However, I think the existence of the standard library distutils API implementation is a genuine problem given that we don't really want people to be using it, so I filed https://github.com/pypa/packaging-problems/issues/127 to discuss how we might manage the process of making setuptools the reference implementation of the traditional distutils API.

[di]

and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0

Yes, per https://github.com/pypa/wheel/issues/231 we will need to instruct users wanting to build wheels to use an as-of-yet-unreleased version of wheel (working on this now).

[brainwane]

Twine 1.11.0 is now out and includes some fixes that'll stop users from trying stuff that won't work anymore.

We're leaving this issue open till @di's pypa/wheel#232 is merged & released, so users can upload Markdown READMEs in their wheels.

(You can thank Dustin in person at PyCon in May, where he's speaking and sprinting.)

[brainwane]

@di does https://github.com/pypa/wheel/compare/0.30.0...0.31.0 and https://pypi.org/project/wheel/0.31.0/ mean we can close this issue?

[di]

Yep! Thanks everyone for your patience w/ this issue.