Go: proposal: go/doc: Support for bulleted lists

[robfig]

Presently, the best practice when enumerating something in godoc is to use a
pre-formatted section.  For example:

"""
This library has the following caveats:
  * It requires the caller to invoke the method
    only on odd-numbered days
  * It may crash your computer.
"""

I think it would improve godoc (HTML) readability to recognize and format bulleted lists
as <ul>'s instead.  The recognition could be narrowly defined to avoid mistakenly
formatting blocks that were not intended to be lists.

I propose a list is recognized as:
- An indented (pre-formatted) block, 
- .. that consists of 2 or more list items
- .. and that consists only of list items

and a list item is defined as:
- Consecutive lines
- .. where the first line begins with "- " or "* "
- .. that terminate on blank lines.

This can be implemented with minimal disruption to the existing code.  Here is a CL:
https://golang.org/cl/91830044

Here are some before/after examples from popular package docs: 

http://godoc.org/code.google.com/p/gorilla/mux
http://192.241.149.161:8080/pkg/code.google.com/p/gorilla/mux

http://godoc.org/code.google.com/p/gorilla/sessions
http://192.241.149.161:8080/pkg/code.google.com/p/gorilla/sessions

http://godoc.org/code.google.com/p/gorilla/schema
http://192.241.149.161:8080/pkg/code.google.com/p/gorilla/schema

http://godoc.org/code.google.com/p/gogoprotobuf/gogoproto
http://192.241.149.161:8080/pkg/code.google.com/p/gogoprotobuf/gogoproto

http://godoc.org/github.com/davecgh/go-spew/spew
http://192.241.149.161:8080/pkg/github.com/davecgh/go-spew/spew

http://godoc.org/code.google.com/p/go.text/collate/colltab#pkg-constants
http://192.241.149.161:8080/pkg/code.google.com/p/go.text/collate/colltab#pkg-constants

[bradfitz]

Comment 1:

Previous proposals similar to this have been rejected on grounds that it's a slippery
slope from this to Markdown or worse.

[robfig]

Comment 2:

I agree wholeheartedly with comments as plain text, free of presentation markup. 
This CL reflects how developers are already writing documentation and is a strict
improvement in the presentation of it.  I don't believe it has any cost to in-code
comment readability.

[ianlancetaylor]

Comment 3:

The way to discuss a proposal like this is to raise it on the public lists.  Especially
if you have a patch.  See http://golang.org/doc/contribute.html .  Thanks.

_Labels changed: added repo-main, release-none._

[frankandrobot]

Are you kidding me? You guys have an issue with markdown?

[bradfitz]

@frankandrobot, that is not a constructive comment. We happily use Markdown on Github and elsewhere. We just don't want it to be Go's documentation format.

[frankandrobot]

@bradfitz i'm surprised that you guys decided to go with plain text. In practice in a large, complex app, you'll end up approximating a markup language for readability anyway (ex: caps for sections), and well you might as well pick a markup language for comments.

Alternatively, since you guys hate markdown, is there the ability to pick whatever markup you want (like a plugin for markdown)?

[bronger]

While I sympathise with not supporting Markdown in godoc, I also think that bulleted lists are important enough to have them properly displayed in godoc. A PR is still welcome? Or has this been rejected somewhere else already?

[bradfitz]

I think @adg and @robpike want to see a design before code.

[robpike]

And @griesemer.

[theherk]

I understand and support the strive for simplicity. In this case, it seems, simplicity could be improved going either direction. Either have it be plain text or support a preexisting parser. To do it the way it is currently seems unnecessarily complex. As it stands, a simple parser is implemented (to heading sections and for preformatted text; documented here).

It seems that just supporting an already existing parser, or at least parser specification like Markdown, would be duck soup. Maybe it would hurt the elegance of the output, but a list is a key ingredient of documentation.

[griesemer]

Moving to 1.9Maybe to raise visibility. No guarantee we're getting to it for 1.9, though.

[awnumar]

@bradfitz

Previous proposals similar to this have been rejected on grounds that it's a slippery slope from this to Markdown or worse.

This is an example of the slippery-slope fallacy and is not a valid argument.

[bradfitz]

@libeclipse, thanks.

[KernelDeimos]

In case anybody is interested, I made a modified version of Godoc that adds ordered and unordered lists with a fast and simple custom parser I wrote called slippery-slope-markdown.

Here's a link to my modified version: godoc-custom-fork

Admittedly, the way I threw it in there is kinda hacky, but it could be a starting point.

The parser follows this convention:

• A line starting with - and a space is a list item
• Any non-blank line following a list item belongs to that list item
• A blank line terminates the list

A similar convention is applied to ordered lists. As an aside, it also allows bold text.

[cespare]

Is it okay to turn this into a proposal so that it goes through the normal review process? Otherwise I think it will continue to stagnate.

[cespare]

To chime in on the merits:

I find the lack of list support to be a persistent annoyance when writing documentation, and I think there are at least two strong reasons for adding this feature:

1. The obvious way to write lists in doc comments yields mangled HTML. It's a common mistake to write a normal bullet/numbered list like you would in any plaintext document and not notice that the HTML output is wrong. This happens in the standard library (example 1, example 2, example 2 rendered). Third-party libraries make this mistake more frequently.

   This mismatch between expectations and output goes against the godoc philosophy. The godoc blog post says:

   Godoc comments are just good comments, the sort you would want to read even if godoc didn't exist.

   and

   Note that none of these rules requires you to do anything out of the ordinary.

   Unfortunately, when it comes to lists, you need to write comments that are different from what you'd write if godoc didn't exist, and the lack of list handling does require you to do something out of the ordinary.

2. The workarounds are ugly. The standard workaround is to use a pre-formatted text block for doc comments. Some examples:

   • encoding/json.Marshal
   • encoding/xml.Marshal
   • go/build
   • go/doc.Examples
   • html/template.CSS
   • net/http/cookiejar.PublicSuffixList
   • net/mail
   • path/filepath.Clean

   The HTML output looks bad: it's better than a mangled list, but there's no reason that all the list text should be pre-formatted.

   In the case of net/http.Client, @bradfitz noted that the pre-formatted style of list is ugly in #20043 and we ended up changing that list to use unicode bullets. This workaround looks a little better than having large <pre> blocks, but it's hard to type, it still looks bad (the indentation and spacing is wrong), and you have to remember to separate each bullet item with a blank line to force a new paragraph or else you're back at square one with a mangled list.

As far as specific proposals go, I prefer something like @KernelDeimos's suggestion rather than @robfig's original one because you shouldn't have to indent text for it to be considered a list.

[robpike]

You call them ugly, I find them easy to read. Yes, it would be nice if they were prettier but nice enough to merit another step down the slippery slope? I am not convinced either way.

I very much like the lack of fuss in Go documentation.

All that said, I agree that a proposal is a good way to resolve this.

[oelmekki]

Markdown always stroke me as something that was looking as good in plain text than in html (except maybe for the links part, I prefer to add numbered notes, like [1] and add the link at the bottom of content).

Could someone elaborate on why it is disliked? It is about avoiding embedding too big a parser in the toolchain, or do you have some fundamental problems with the format? I'm asking, because it can make the difference between trying to build a more efficient markdown parser, drafting a subset of markdown format or discarding it completely.

[ianlancetaylor]

I don't think anybody has strong feelings about markdown when it is used appropriately, it's more a sense that we want our comments to be ordinary text, not text with embedded formatting commands.

[frankandrobot]

If making it official is part of the problem, why can't markdown for documentation be a plugin? Is go currently built that way that it can support custom comment formats? This way if an organization finds it useful, then they can enable it or even use something else...

[ianlancetaylor]

You can fork the godoc program to do as you please. (There is no plan to add plugins to godoc, and I don't see any real need for them.)

[frankandrobot]

(There is no plan to add plugins to godoc, and I don't see any real need for them.)

Well it would obviate the need for this issue right? And the need to create a proposal...

[oelmekki]

Maybe it's worth for us who want markdown or markdown-like support to describe our need rather than a solution.

I reached this issue page googling for "godoc lists". The reason for that is that I try to replace my usual markdown documentation with godoc.

The documentation question is something we encountered previously in ruby. Initially, we were using rdoc, with its rdoc markup. To build docs, we would use something like rake doc to build the documentation for a project. When github added markdown support, it felt way better to organize documentation : instead of expecting users to generate the doc, we could just drop .md files in the repos, and it would automatically generates good looking documentation.

Now, in go world, there is this great service that is godoc.org. We can just look up a github repos, and it builds a doc for it. Awesome. I'm ready to replace my usual markdown documentation for that, especially given all the added bonus of automatically parsed packages and functions index. We can launch godoc locally, have a local website serving it, or even read it as pure text in IDE. This is a great documentation tool.

Problem is, if we want to add exhaustive documentation within it, we quickly feel limited. Bullet list are a thing - although, as Rob is pointing it, using code block still looks good (until you hit line wrap). But there are other things needed to organize a pleasant to read global documentation, like having several level of headings.

I'm not especially needing markdown by itself, my concerns are about having enough expressiveness to build a full blown documentation, rather than "simply" an API doc, completed by a more high level documentation hosted somewhere else.

Do you guys think we should only use godoc for api doc, or do you envision godoc as a complete self sufficient documentation tool? (or do you solve the same problems an other way?)

[ianlancetaylor]

@oelmekki Thanks for the explanation. I think the main point is that we don't particularly want to start writing annotated comments in Go source code. We think those comments can be ordinary text, and so far we're willing to stick to that.

However, I think you are describing something slightly different: some packages do require long comments, such as https://golang.org/cmd/go, and that text, though currently written as a long comment in a Go source file, could in principle be found somewhere else. For example, perhaps godoc could look for a doc.md file and treat that as the package comment.

Would you like to write that up as a separate proposal? I can't promise that it would be accepted, but I think it's worth discussing.

[oelmekki]

For example, perhaps godoc could look for a doc.md file and treat that as the package comment.

Good point, the only place where I ever felt the need for more advanced formatting is in doc.go files, where I add this kind of documentation. My first thought to your idea was "well, maybe we could parse the README.md file, which is already there for most projects hosted on github", but it's probably a good thing to be able to keep the README file small (explaining the why's, the install process and a short example usage section) and have a full documentation somewhere else.

Would you like to write that up as a separate proposal? I can't promise that it would be accepted, but I think it's worth discussing.

With pleasure. Although I was already doing what I called "documentation driven development" before, I've started only a few months ago to draft fully detailed specs before writing any code following the talk from Peter Bourgon at gophercon2017. This could be a good training and follow up for that, accepted or not.

I have no experience drafting proposals for big open source projects, though. Do you have any guidelines about writing a good proposal? Or at least, could you point out a few proposals that are considered best ones?

Thanks!

[robfig]

The argument against supporting bullet lists seems logically inconsistent to me. I 100% agree that comments should be pure text and as readable in source code as it is in the browser. I think godoc is a far better documentation system than what's come before.

However, these things are written naturally / today by documenters:

• Headers (an extra newline before / after)
• Embedded code blocks (a block indented with spaces)
• Bullet lists

In my mind, there is literally no difference in kind or caliber between those three examples, and yet 2 of them have special formatting treatment in godoc and the third does not. This is despite the fact that it requires an explicit workaround to avoid broken browser formatting, where the list is joined into a single paragraph with items separated by hyphens. Additionally, even when the workaround is taken, IMO it has poor appearance and readability compared to a native bullet list due to the line-break issue and random visual distinction. I think there are many examples where the author forgets to use a code block and the experience for a reader in those cases is very bad.

godoc can be easily extended to recognize the bullet lists patterns that exist today and fix this problem, and I see no downside to doing so. You can make a slippery slope argument, but many things in engineering involve selecting a spot on a continuum, so I don't find that too persuasive. Regardless, addressing that argument, I'm not aware of anything else that godoc formats flat-out incorrectly. Any other feature requests from markdown would be for adding something new, not fixing something that already happens poorly.

[ianlancetaylor]

@oelmekki Go's proposal process is documented over at https://github.com/golang/proposal/blob/master/README.md (using Markdown!) and there are several examples of proposals in that directory.

[ianlancetaylor]

@robfig This issue kind of spun off into a discussion of markdown. I don't see any strong argument in this issue against better support for bullet lists, other than Brad's comment about a possible slippery slope. What I see in this issue is people asking for a design, but I don't see a design. Perhaps I missed it.

[dsnet]

Here's a proposed design.

The current rule is:

A span of indented lines is converted into a <pre> block, with the common indent prefix removed.

My proposed adjustment is:

A span of indented lines is converted into a block with the common indent prefix removed. If the first line has a * prefix, then the block is converted into a <ul> block with <li> sub-blocks created for each line with a star-prefix and subsequent lines. If the first line lacks a star-prefix, then the entire block is converted into a <pre> block.

This piggy-backs on the parser that identifies pre-blocks, and has the benefit that use of bulleted blocks still look decent on older versions of godoc.

There is no support for sub-bullets or pre-formatted blocks within a bullet list.

---

For example:

// This is a paragraph
// on two lines in the source code.
//
//  func main() {
//      fmt.Println("Hello, world")
//  }
//
// Another paragraph:
//  * Bullet point 1.
//  This is still part of the above point.
//      Even if you indent again.
//  * Bullet point 2.
//  * Bullet point 3.
// Another paragraph.
//  This is a preformatted block since it has no star-prefix.
//  * Regardless of whether star-prefixes occur on subsequent lines.
// End paragraph.

This gets rendered as:

This is a paragraph on two lines in the source code:

func main() {
  fmt.Println("Hello, world")
}

Another paragraph:

• Bullet point 1. This is still part of the above point. Even if you indent again.
• Bullet point 2.
• Bullet point 3.

Another paragraph.

This is a preformatted block since it has no star-prefix.
* Regardless of whether star-prefixes occur on subsequent lines.

End paragraph.

Another example (from unsafe.Pointer):

// Pointer represents a pointer to an arbitrary type.
// There are four special operations available for type Pointer
// that are not available for other types:
//
//  * A pointer value of any type can be converted to a Pointer.
//  * A Pointer can be converted to a pointer value of any type.
//  * A uintptr can be converted to a Pointer.
//  * A Pointer can be converted to a uintptr.
//
// Pointer therefore allows a program to defeat the type system and
// read and write arbitrary memory. It should be used with extreme care.

This is rendered as:

Pointer represents a pointer to an arbitrary type. There are four special operations available for type Pointer that are not available for other types:

• A pointer value of any type can be converted to a Pointer.
• A Pointer can be converted to a pointer value of any type.
• A uintptr can be converted to a Pointer.
• A Pointer can be converted to a uintptr.

Pointer therefore allows a program to defeat the type system and read and write arbitrary memory. It should be used with extreme care.

[cespare]

@dsnet I disagree that we should have to use indentation to get bulleted lists. People should be able to write normal, plain-text bullet lists and have it just work instead of getting mangled as it does today.

I also think that numbered lists should work (i.e., give you <ol>s.).

[dsnet]

@cespare, I don't feel strongly about not requiring indentation, as long as the rule handles the common situation where people often add spaces to align a bulleted paragraph up with the text after the bullet point.

Example:

// * Bullet point 1.
//   Still part of the above bullet.
// * Bullet point 2.
//
// * Bullet point 3.

Rendered as:

• Bullet point 1. Still part of the above bullet.
• Bullet point 2.
• Bullet point 3.

Numbered lists are nice too::

// 1. Ordered point one.
//
// Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
// sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
//
// 2. Ordered point two.
//
// Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
// sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
//
// 3. Ordered point three.
//
// Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
// sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Rendered as:

1. Ordered point one.

Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

1. Ordered point two.

Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

1. Ordered point three.

Intervening paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

(The CSS for godoc can strip left-padding so that the bullets align with the paragraphs).

[theherk]

My preference would be to consider bulleting for each paragraph, but with a more simple implementation. I recommend the rules change as follows.

My addition is italicized:

• Subsequent lines of text are considered part of the same paragraph; you must leave a blank line to separate paragraphs.
• _Paragraphs beginning with * are rendered as unordered lists. Following lines are considered part of the same list item unless beginning with *. The list continues until a blank line is found and there is no support for nested or ordered lists._
• Pre-formatted text must be indented relative to the surrounding comment text (see gob's doc.go for an example).
• URLs will be converted to HTML links; no special markup is necessary.

Regarding the recommendations just preceding, I believe they over-complicate the truly simple requests of having simple lists. Additionally, in dsnet's comment, how would you know when the list had ended?

I believe the only way we can get wide support, especially from the core team, is for the request to be very simple, and not turn into a host of new rendering rules. I can understand the desire to have more, but I likewise understand the elegance of simplicity. It is either markdown, or as far from it as we can stay while still having a satisfactory list.

---

In the specification here, this text:

Each span of unindented non-blank lines is converted into a single paragraph. There is one exception to the rule: a span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation is formatted as a heading.

would need to change to:

Each span of unindented non-blank lines is converted into a single paragraph. There are two exceptions to the rule. First, a span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation is formatted as a heading. Second, a span that begins with * is rendered as an unordered list. Following lines withing that span are considered part of the preceding list item unless beginning with *. There is no support for nested or ordered lists.

[ianlancetaylor]

In designing lists remember that comments turn into plain text, not just HTML, via go doc. Thanks.

[dsnet]

I hacked up an implementation of this (~25 lines of change on top of CL/72890) and I feel more strongly now that lists be preceded by indents since there are some cases where users have preceding "1. " or "* " where their intention is not to have a list, but rather a heading (which is a separate concept from what this issue is about).

Secondly, lists are typically rendered with an intent themselves, thus, it is natural to expect that how they appear in the comment also be indented.

[theherk]

If their intent is to have a heading, they should have the span be a single line, begin with a capital letter, and contain no punctuation. Indentation has a meaning already, <pre>; no reason to confuse that. How lists are commonly rendered seems irrelevant. How are they most intuitively read and written in plain text?

[gopherbot]

Change https://golang.org/cl/113915 mentions this issue: crypto/x509: reformat template members in docs

[cespare]

I previously discussed turning this into a proposal but never did. Adding the label now, so that this gets discussed.

Current proposal is to go with @dsnet's suggestions in https://github.com/golang/go/issues/7873#issuecomment-341206350 and https://github.com/golang/go/issues/7873#issuecomment-341254252. Essentially, we'll take the current "standard" workaround for bulleted lists (indent so as to create a code block) and recognize it, turning it into a nice bullet list in HTML. We could optionally do something special for text (go doc) output but it's okay to leave that alone for now.

[bcmills]

we'll take the current "standard" workaround for bulleted lists (indent so as to create a code block) and recognize it

In addition to the asterisks in @dsnet's sketch, we should probably also accept at least Unicode bullets (because they express clear semantic intent) and hyphen-minus characters (because that's what we use throughout the Go codebase) as the leading symbol to recognize a list.

[mwmahlberg]

We think those comments can be ordinary text, and so far we're willing to stick to that.

Isn't that the whole point of markdown? That it can be read both as ordinary text but can also be rendered to a more eye-friendly form? I can see some problems here, for example on how to deal with markdown links (both of the hyper and image flavor), but those could be easily circumvented, couldn't it?

I personally would love to have markdown in my comments, since that would basically prevent me from redoing myself having both godoc comments and a more detailed explanation elsewhere. So basically, all documentation could be dealt with and would be present in one place.

[cespare]

@mwmahlberg in the past year this thread has been converging on a small, concrete proposal for adding lists. That's also the title of the issue. It previously devolved into a markdown discussion, but I'd like for it to stay focused on just lists at this point in hopes that it will finally be reviewed and approved.

At this point, let's put new requests for markdown into a new issue or else add details to #16666. Thanks.

[cespare]

@golang/proposal-review is there something more to be done to move this along? (I understand that I can't expect much at this point during the last few weeks of the year, but I added the proposal label in August.)

[mwmahlberg]

@cespare I understand that and did not mean to escalate the topic at hand. But my point of view is that the question at hand is basically a landmark decision: Open up for more formatted godocs or not? Because when it is started, more feature requests will come in: named links, tables, etc. Therefor, the reluctance is understandable. What I was trying to bring across that markdown|rst|foobar markup could be seen as a language feature with quite some value (DRY) for developers.

[rsc]

Generally, we'd like to allow writing lists in docs. Deciding the exact syntax is still unresolved. It's not a high priority right now.

[gopherbot]

Change https://golang.org/cl/167403 mentions this issue: builtin: make len's godoc less ambiguous

[rwxrob]

I find it extremely disturbing that this discussion has not once referenced the amazing efforts of the Pandoc Markdown project (which is the standard documentation method use by the R language as well as many text books). It is objectively the most widely adopted knowledge source standard.

[fosskers]

Can this move forward at all? It seems like many people are looking for it, and miss this docstring feature that is present in many other languages.

[dmitshur]

/cc @julieqiu This is a longstanding proposal (with lots of discussion above) to modify how Go packages can be documented. Perhaps it should be considered as part of other future work that's being done to improve documentation rendering on pkg.go.dev.

[tooolbox]

Just stumbled across this.

Current proposal is to go with @dsnet's suggestions in #7873 (comment) and #7873 (comment). Essentially, we'll take the current "standard" workaround for bulleted lists (indent so as to create a code block) and recognize it, turning it into a nice bullet list in HTML. We could optionally do something special for text (go doc) output but it's okay to leave that alone for now.

In addition to the asterisks in @dsnet's sketch, we should probably also accept at least Unicode bullets (because they express clear semantic intent) and hyphen-minus characters (because that's what we use throughout the Go codebase) as the leading symbol to recognize a list.

Sounds great! Would be neat to address this in pkg.go.dev.