Readthedocs.org: Add travis-ci style pull request builder

[alex]

It'd be _amazing_ for helping review documentation if RTD has a travis-ci pull request builder. It could watch for PRs/pushes, build docs for them, and then link them in the build status.

/cc @reaperhulk

[reaperhulk]

We do this with a jenkins builder right now on pyca/cryptography but if we could get it straight through RTD that would be wonderful.

[agjohnson]

This is on our radar, but we need to make some design decisions around how to handle arbitrary HTML uploads without allowing completely arbitrary HTML uploads.

Closing this as a duplicate of some bug I know exists and I'll find later :)

[ericholscher]

This isn't an arbitrary HTML upload ticket, it's a request to auto-build branches from PR's of projects that we know about. It's definitely something we should be doing.

[agjohnson]

The ticket description here seemed to describe building docs on Travis. Reopening this as we are describing a Github/Bitbucket pull request integration.

There are some design decisions that would be required here. Namely, how to handle pull requests from branches on forks given our currnet data model. We could easily support branches pushed to the repo we know about by expanding the webhook to handle PR open/update.

[alex]

This ticket isn't about building docs on travis; it's about building docs _in the style of travis_. That is, on every push for a PR, build the docs and link them so people can browse.

[bukzor]

If we could set up webhooks to tell readthedocs to build docs for pull requests that touch documentation, that would be ideal.

Currently when a pull request comes in that touches documentation, we have two distinctly distasteful options:

1. Eyeball the changes and merge it if it _looks_ right, then try to remember to double-check rtfd when it builds the merged master.
2. Tell the requestor to set up rtfd for their branch. Even users that are intimately familiar with sphinx/readthedocs would often balk at this request, since it would have to be re-done each time the pull request used a unique branch name.

[ericholscher]

I agree this makes a lot of sense. We need to implement some kind of "temporary" version that would map to the PR, and clone from the remote repository sending the PR. Then we'd need to delete the version when the branch got merged, but I agree this would be quite useful.

[bukzor]

ericholscher: I think those can all be done based on data from the github webhooks, but I've no idea where the implementing code would live.I suppose it would be a new endpoint for the rtfd site?

[agjohnson]

Blocked on #2465

[stsewd]

I don't know how the GitHub API works for this case, but I think this feature #872 could be used for this

[ssbarnea]

Any news on this? I do build sphinx docs with Travis on multiple projects but it seems that even if building docs with travis may succeed, it may not succeed on RTD, causing more trouble.

As I find much easier to build docs myself, fully integrated into CI pipeline, I am starting to wonder what are the real benefits of using RTD in this case, as it seems to be a PITA.

Did anyone managed to find a workaround that would prevent people from merging PRs that would break building RTD documentation?

[stsewd]

@ssbarnea this is what you get from rtd for free :wink: https://docs.readthedocs.io/en/latest/features.html

I'm not sure how this feature would be integrated (rtd would have a lot of _temporal_ docs), you can have something like this for checking your builds

https://github.com/rtfd/readthedocs.org/blob/e73b266558af84745dd1cfc9e9dec7aa3e091dde/tox.ini#L21-L24

And maybe using rstcheck (something like this https://github.com/rtfd/readthedocs.org/pull/3624)

[thorade]

I would like to use this in combination with protected branches and status checks, so that PRs can only be merged if they do NOT break the building process on RTD.
https://help.github.com/articles/about-protected-branches/
https://help.github.com/articles/about-required-status-checks/
https://developer.github.com/v3/repos/statuses/

[flying-sheep]

if i could at least activate all tags and branches as rtd versions, i’d be happy.

right now there’s no way to test a doc change without releasing a beta version on github (i.e. using a tag that looks like a version)

[stsewd]

@flying-sheep

right now there’s no way to test a doc change without releasing a beta version on github (i.e. using a tag that looks like a version)

You can actually build a branch/tag that hasn't a _version format_. If you create a branch/tag named test, that would be listed under the versions section, then you can activate and built it. If that isn't what you mean, sorry I did not understand it.

[flying-sheep]

This is what I mean, but I don’t see the tag appear in the versions section.

[stsewd]

@flying-sheep rtd doesn't pull automatically tags via webhooks (#3546), so you need to build any current version (like latest for example) in order to rtd pull the new tags/branches.

If that doesn't work, please open a new issue with your rtd project url.

[flying-sheep]

Thanks!

[ericholscher]

I just had a realization for this, which is that GH lets us access the PR's for a project as a remote:

https://gist.github.com/piscisaureus/3342247

You can then checkout a PR on the repo in a normal fashion:

git reset --hard origin/pr/95

This would let us treat PR's as just another type of Version (Tag/Branch/Pull Request) and use the existing RTD modeling to support them. We could then have a custom syncer for PR's that pushes the content into an S3-type bucket instead of keeping them around forever, and perhaps get this feature done without too much work.

[jakirkham]

Yeah there is a /merge and /head under that. The first is a generated merge commit with the base branch (presumably master). The second is the latest commit on the PR.

Can also get both using git fetch and optionally attach branches to them locally. We use this a lot in conda-forge. Happy to answer questions.

[gaborbernat]

@ericholscher what you say sounds like a plan. Is this still in blocked given that, or has moved over to needs help/work phase? :+1:

[ericholscher]

I believe this should be unblocked.

[ericholscher]

There are still some design decisions, but I think at least for GitHub, we have a path forward.

[KengoTODA]

to whom it may concern,

I created a GitHub Probot app that enables RTD build and share its document URL in PR. It may cover a part of your requirements, please check.

[ericholscher]

Interestingly, travis does it using the git aliases also: https://travis-ci.org/rtfd/readthedocs.org/jobs/482572527#L418

[davidfischer]

This feature is accepted and we plan to do it. Our plan is to do this after all build artifacts are stored in blob storage (see https://github.com/rtfd/readthedocs.org/pull/5549 for a start on that) because this change will significantly increase our storage requirements as RTD would be storing build output for every PR docs build indefinitely. This will become higher priority in the latter half of 2019.

[alex]

That's wonderful to hear! If you ever want a beta tester, don't hesitate to holler!

[stsewd]

We have this feature in beta https://blog.readthedocs.com/building-docs-for-pull-requests/

[ericholscher]

Going to call this one closed, as it's now in the codebase. We will slowly roll it out, but it's done 👍

[jakirkham]

What are the steps for setting this up for a repo currently?

[stsewd]

@jakirkham you mean locally or on the site? On the site you need to request a beta access https://blog.readthedocs.com/building-docs-for-pull-requests/

For a local installation you need to enable this flag for your projects EXTERNAL_VERSION_BUILD

https://docs.readthedocs.io/en/stable/guides/feature-flags.html

You can do this from the admin of Features

[jakirkham]

Thanks for the detailed answer. Was meaning the site. Didn't know if things had changed in the past few months.

[flying-sheep]

Since I find this issue faster than the docs: https://docs.readthedocs.io/en/stable/guides/autobuild-docs-for-pull-requests.html