Hypothesis: Overhaul documentation structure for distinct kinds of docs

[Zac-HD]

At PyConAU, there were several good talks about documentation - including this one by @EvilDMP. For readers and later reference, he's also written it up as an article.

In short: there are four kinds of documentation (tutorials, how-to, discussion, and reference; plus about-the-project). Each has a different aim and style, and the structure of the docs should reflect this.

I intend to experiment with this when I have time, but if anyone wants to jump in feel free - just leave a comment (as I will) so we avoid duplicating work.

[DRMacIver]

I'm definitely keen to improve our documentation structure.

One thing I'm not so sure about is the move to more docstrings. The problem with doing that is that changes to docstrings currently require a release. I think if we're going to to move to move to more docstrings then we need to make releases slightly less frequent and streamline the process first.

(I don't have any objection to docstrings per se, so if and when we do that anyway I'd be fine with more docstrings, I just don't think they're sufficiently compelling in their own right to force that)

[Zac-HD]

I and other data-y people find docstrings really useful in interactive environments, but I'm happy to defer that part until after (if, sometime) we change the release process.

For other parts, after my current round of open pulls are merged I can look at a few specific options for structure, and after that the improvements can usefully be done on an incremental basis. No timeline for any of this, though.

[DRMacIver]

I and other data-y people find docstrings really useful in interactive environments

Hmm. Which I guess is an argument that we do want to release docstring changes immediately.

I guess after we've got #784 I don't really have any objections to that. Frequent releases are mostly only a problem for us because the release process is annoying (and making them slightly more frequent isn't going to be substantially more annoying for users than it already is).

[Zac-HD]

Honestly, if we have so many people desperate to improve our documentation that it impacts our release process I'll be pretty happy!

[Zac-HD]

@hwayne, this is probably the issue for you to get excited about 😄

[hwayne]

Do we have a sense of what we mean when we say "We want to improve the documentation?" Is that more tutorials? Better references? Clarification on specific things you can do? And what are people most in need of? SO seems like it has some questions, to start. Do we have a list of people's questions? IRC logs?

[Zac-HD]

These are all good questions! This specific issue was inspired by a talk at PyConAU (link + article form above) about how to structure different kinds of documentation - I'm not convinced about the specifics, but we definitely need a clearer structure than we have. I suggest something like:

• Tutorials and how-to: a reasonably linear introduction to how to use the library.
• Reference documentation: comprehensive API docs for all the public bits (which are not all documented yet), preferably autogenerated from docstrings
• Stuff about the project: changelog, release notes, development policy, supported versions, etc.

But if you're willing to write something better than what we have, your proposal will have significant weight 😄

[DRMacIver]

which are not all documented yet

I don't agree. If it's not documented it's not a public bit.

There is definitely an non-uniformity to the structure of the public documentation and how concepts are introduced, but that's not the same thing.

In particular it's worth noting that a lot of the stuff that is undocumented is undocumented for the very specific reason that it's not part of the API we're committing to. SearchStrategy is a bit weirdly documented in large part because most of its API isn't public and it's supposed to be treated as a black box.

[DRMacIver]

One thing I'd like to think about in terms of improving our documentation structure is better integration between hypothesis.works articles and the documentation.

[Zac-HD]

I don't agree. If it's not documented it's not a public bit.

I agree entirely in principle and mostly agree in practice. Some things don't have much of an API reference - for example the Django extra is mostly narrative, and I think it would be useful to have a concise reference page for the core things (what you get if you from hypothesis import *). Also our errors seem to be undocumented, but IMO we should at least outline the exception hierarchy - it's useful to know 'catch this thing to handle that kind of error' and listing for eg InvalidArgument would help with that.

One thing I'd like to think about in terms of improving our documentation structure is better integration between hypothesis.works articles and the documentation.

Indeed! I'd also like to have a tutorial that takes you through writing useful tests for a small project - similar to the training exercises, but with an explicit progression through several steps of using Hypothesis.

[hwayne]

I think one area of improvement is that the official docs act more right now like reference than discussions and how-tos. Things are divided by logic rather than use-case. For example, assume and example are in "advanced features", while mapping and filtering are under "what you can generate". I'd argue that their use-case is similar enough - basic tweaks to your state space - that they should be grouped together in a how-to.

I'll mock up a rough how-to of what that'll look like, see how y'all feel about that approach.

[Zac-HD]

I'm closing this issue as it seems to have been a bit to vague to drive useful outcomes 😕

We still see docs as an important part of the project of course - it just seems that more specific and incremental issues have led to bigger improvements because they actually get worked on!