Go: doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository

[andybons]

With the website slated for a refresh, we should move the static content out of the main go source tree (https://go.googlesource.com/go/+/master/doc/) and into its own repository (web? website?).

This will reduce binary and source code download sizes (#27151) and allow us to apply changes unrelated to our release schedule without having to cherry-pick them to a release branch. Hopefully we can also implement continuous delivery as part of this as well.

We also won’t have to worry about bloating the main go source tree as we add more assets and content for the refresh.

We still only want to display package documentation and source files from a GOROOT pinned to the current stable release (and not show content only relevant for future releases like release notes). So there are elements that still require logic that ties content to a specific Go version. What we consider fluid vs tied to a release is to be determined and can be fleshed out here.

/cc @bradfitz @ianlancetaylor @julieqiu @katiehockman @dmitshur @FiloSottile @cnoellekb

[bradfitz]

"website" SGTM. I like the repo name golang/website.

Inside that I'd like something like:

• README.md
• codereview.cfg
• content/ directory
• cmd/ directory
• cmd/goweb or cmd/golangweb or cmd/golang-org-server (name TBD)

We could start with mirroring stuff over from the existing golang/go repo (and any necessary code from x/tools/cmd/godoc) but without modifying either for now. And then serve the new one on something like https://new.golang.org for a bit and when we're happy then switch over and start deleting code & assets from the old locations.

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

If they're the same, then do we develop it in x/tools or in x/website? I kinda like having the code next to the assets but I could go either way and haven't thought about it much.

[andybons]

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

My preference is for x/tools/cmd/godoc to become a binary for viewing local $GOPATH/etc documentation but not anything else (which would be more in line with its name anyhow). A common go-source-to-html rendering library could be pulled out (if it’s not already general enough in x/tools/godoc) and used in places like godoc.org, x/tools/cmd/godoc, and the new website. This way most of the logic specific to new.golang.org is right next to the content.

[dmitshur]

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

Yes, we need to figure that out. At this time, I strongly suspect that we’ll want to make the new cmd/golang-org-server be strictly the website and not a tool users run locally, other than to contribute to the golang.org website development.

[bradfitz]

Pulling the render-godoc-as-HTML handler out and using it on both places SGTM. That can happen first probably.

And making x/tools/cmd/godoc be super minimal also SGTM.

[cnoellekb]

We discussed 4 main milestones for moving the website:

1. Separate out rendering logic (converting code -> HTML) from x/tools/cmd/godoc. There should be no behavior change at this point.
2. Creating website binary and serving templates (the static content) from go/doc to x/website
3. Using the common rendering library in x/website.
4. Reduce x/tools/cmd/godoc to be the minimal version (since the website can now be run out of x/website).

Right now I'm focusing on the second step because I was able to get a prototype up and running. I copied tools/cmd/godoc, tools/godoc, and go/doc into a local directory that will eventually become the root for the main website. The plan is to just copy everything for now to maintain x/tools functionality. Can someone help me set up an x/website repo?

Also, I was looking into how the rest of the tools directory would be affected by removing any godoc files and I found that x/tools/cmd/present/play.go uses the tools/godoc/static package. Does anyone think this may be an issue?

Lastly, is the final version of godoc going to need an app engine configuration? If not, then tools/internal/memcache can be moved to the x/website repository.

[bradfitz]

@dmitshur, @andybons, where are the notes from last time we created a new repo? I'd rather not do it myself. It'd be nice if we can have somebody else run through the steps from the docs. (gerrit, github, maintner, gitmirror updating, etc)

[dmitshur]

where are the notes from last time we created a new repo?

@bradfitz The steps are documented at golang.org/wiki/CreatingSubRepository.

It'd be nice if we can have somebody else run through the steps from the docs.

Yep, that's a good idea. If anyone on the team wants to go through the process to learn it, I'm happy to assist and fill in any gaps in docs. Note that most of the bottlenecks will be having the right permissions to all components involved. I can also do it myself.

[bradfitz]

I created https://github.com/golang/website and https://go.googlesource.com/website (Gerrit).

@cnoellekb, can you send CLs for the maintner and dashboard and gitmirror bits? And gopherbot if needed.

[cnoellekb]

Sure, I'll take care of it!

[gopherbot]

Change https://golang.org/cl/156217 mentions this issue: maintner/maintnerd, cmd/gerritbot, cmd/gitmirror: add golang/website to

[gopherbot]

Change https://golang.org/cl/156337 mentions this issue: cmd/godoc: add x/website redirect

[gopherbot]

Change https://golang.org/cl/156321 mentions this issue: x/website: add code and static files for website

[gopherbot]

Change https://golang.org/cl/156326 mentions this issue: x/website: add code and static files for website

[cnoellekb]

Some decisions that have been made over the last few days:

1. I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?
2. Deployment of the website will come later when godoc has been properly stripped down with all non-documentation stuff ready to be deleted from x/tools
3. In the long term @dmitshur plans to expand the package to support rendering documentation for modules

[katiehockman]

1. I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?

This sounds good. Perhaps under x/tools/godoc/unstable since it will be for go documentation.

[bcmills]

I wouldn't bother with the /unstable suffix: just put a prominent note in the package comment explaining that it's unstable.

Actually, better still, could we start a new repo with the path golang.org/x/internal? Then the Go tool's visibility rules would enforce that it is only used within other golang.org/x modules.

[dmitshur]

We already have the golang.org/x/tools/godoc package, which is documented as:

Package godoc is a work-in-progress (2013-07-17) package to begin splitting up the godoc binary into multiple pieces.

This package comment will evolve over time as this package splits into smaller pieces.

I think this existing package is a good candidate to move code that we want to share between x/tools/cmd/godoc (for viewing documentation) and x/website/cmd/golangorg (for serving golang.org) in the short term.

I like the idea of x/internal in general, because if it existed, we could use it in other instances to do similar refactors between subrepos. However, I think that should be a separate proposal. Also, golang.org/x/internal would preclude golang.org/dl from being able to use it, so maybe it should be golang.org/internal.

[gopherbot]

Change https://golang.org/cl/159737 mentions this issue: x/website: remove godoc package/make module aware

[gopherbot]

Change https://golang.org/cl/159897 mentions this issue: website: use modules

[gopherbot]

Change https://golang.org/cl/159917 mentions this issue: internal: add copy of packages in x/tools/godoc

[gopherbot]

Change https://golang.org/cl/159918 mentions this issue: website: fix import paths

[gopherbot]

Change https://golang.org/cl/160137 mentions this issue: cmd/golangorg: add xerrors to x.go

[gopherbot]

Change https://golang.org/cl/160238 mentions this issue: content: delete content/doc

[gopherbot]

Change https://golang.org/cl/160237 mentions this issue: cmd/golangorg: use go/doc instead of x/website/content/doc

[gopherbot]

Change https://golang.org/cl/161198 mentions this issue: cmd/golangorg: make deployment work for x/website

[gopherbot]

Change https://golang.org/cl/162157 mentions this issue: cmd/golangorg: deploy to default servive

[gopherbot]

Change https://golang.org/cl/162158 mentions this issue: content/static: inherit textarea color to avoid illegible text

[gopherbot]

Change https://golang.org/cl/162160 mentions this issue: content/static: update static.go

[gopherbot]

Change https://golang.org/cl/162377 mentions this issue: internal/short: point to new tracking issue for shortlink creation

[gopherbot]

Change https://golang.org/cl/162400 mentions this issue: cmd/godoc: remove golang.org serving code

[gopherbot]

Change https://golang.org/cl/162401 mentions this issue: godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

[cnoellekb]

Update!

x/website has been successfully deployed to https://new-website-dot-golang-org.appspot.com and all of the changes to x/tools have been applied to x/website (as of 2/13/2019) so now we just need manual testing before we make the switch to serve x/website on golang.org. So if you have a chance to try out the website and let me know what problems you find it would be much appreciated, I'll keep a running checklist of bugs here in this post.

Thinking ahead past manual testing and golang.org deployment, there is also the question of what we want to do with go_spec.html (language specs), release.html (release notes), and go_mem.html (memory model). Should they move with the rest of the files in go/doc to x/website/content/doc or should they stay in the go repo? An option is moving them into new go/ref directory.

[gopherbot]

Change https://golang.org/cl/162908 mentions this issue: cmd/tip: start building website from x/website repository

[gopherbot]

Change https://golang.org/cl/162909 mentions this issue: cmd/tip: use bootstrap version of Go to build golangorg

[dmitshur]

So if you have a chance to try out the website and let me know what problems you find it would be much appreciated

I've spotted a minor difference in the HTML that the new website renders compared to the current golang.org:

 <head>
   ...
-  <link rel="stylesheet" href="/lib/godoc/jquery.treeview.css">
   ...
-  <script src="/lib/godoc/jquery.treeview.js" defer></script>
-  <script src="/lib/godoc/jquery.treeview.edit.js" defer></script>

This difference is not a problem; on the contrary, it's an improvement to exclude treeview CSS/JS when it's not used on the page. It was implemented in CL 115875, which was merged to master branch of x/tools 4 months ago. That CL hasn't been backported to x/tools release-branch.go1.11 branch, which is what the current golang.org website has deployed.

x/website is currently using a 24-day-old version v0.0.0-20190124215303-cc6a436ffe6b of x/tools module, a commit only present on its master and release-branch.go1.12 branches:

https://github.com/golang/website/blob/829c1b065c5390c7259b9175d86d6148f13906a8/go.mod#L11

Which is why it's picking up this improvement ahead of Go 1.12 release.

---

We should consider the strategy of whether the x/website should stick to using release-branches of sub-repositories like x/tools/cmd/godoc has in the past, or if it's okay to use newer versions in advance.

I think a good approach would be to _start_ with using the current Go release release-branch (i.e., release-branch.go1.11 now) initially for consistency with the current website, but explore being able to use newer versions in the future, especially as x/website gains more independence from the Go release schedule.

This also intersects with x/build/cmd/tip. We may want to modify it so that it updates go.mod of x/website to use newer sub-repository versions, to test them in advance.

[gopherbot]

Change https://golang.org/cl/162834 mentions this issue: website: update subrepos to use release-branch.go1.12

[dmitshur]

My comment above has been resolved in CL 162834. As described in the commit message, we chose to make the upgrade from release-branch.go1.11 to release-branch.go1.12 a little earlier, given the proximity to Go 1.12 final release. It was a smaller and safer change to make to the x/website that's been in testing since 5 days ago.

By now, all tasks for making x/website the canonical source code of the golang.org website have been completed, and we have not discovered any issues on the staging instance currently deployed at https://new-website-dot-golang-org.appspot.com/.

We plan to make the final switch to deploying the golang.org website from the new x/website codebase tomorrow (Wednesday) morning EST time. There should not be any disruption, and we'll be watching the website closely after making the switch.

If you're planning to send CLs that change the website at x/tools/cmd/godoc, please hold until after the switch has been completed, and redirect them to x/website after that.

[gopherbot]

Change https://golang.org/cl/164818 mentions this issue: cmd/golangorg: add GODOC environment variables to prod

[cnoellekb]

Another update!

Just wanted to summarize the current state of the transition of golang.org from x/tools to x/website. As of last week, golang.org is now deployed from x/website and serving all traffic so all CLs for updating the website should be filed against the x/website repo.

The following shows how files were moved:

"x/tools/cmd/godoc" --> "x/website/cmd/golangorg"
"x/tools/godoc/static" --> "x/website/content/static"
"x/tools/godoc" --> stays put for now, referenced via module
"x/tools/godoc/{dl,env,proxy,redirect,short}" --> "x/website/internal/{dl,env,proxy,redirect,short}"
"x/tools/internal/memcache" --> "x/website/internal/memcache"
"GOROOT/doc/..." --> stays put (to move in the future, possibly leaving go_mem.html and go_spec.html)
"GOROOT/favicon.ico" --> "x/website/favicon.ico"
"GOROOT/robots.txt" --> "x/website/robots.txt"

The internal release notes and deployment scripts have all been updated so that deployment from x/website should go smoothly in the future. Also all changes to x/tools/godoc/{dl,env,proxy,redirect,short} as of Feb 13 were applied to x/website/internal/{dl,env,proxy,redirect,short}.

In addition, Dmitri's also merged the following CLs:

• removed the website serving code from x/tools/godoc here
• deleted godoc/dl, godoc/proxy, godoc/short, and internal/memcache from x/tools here
• updated tip to deploy from x/website here

Short term tasks:

• apply this CL to x/website/cmd/golangorg
• moving all the files in GOROOT/doc (excluding go_mem.html and go_spec.html - location TBD) to x/website/content/doc
• figuring out where exactly to put favicon.ico and robots.txt (currently in the root directory of x/website, need to remove a small chunk of code that was made before x/website used modules and relies on the GOPATH)

Long term tasks:

• fully stripping down the godoc binary so that it only displays local code documentation
• extracting the ability to render code documentation html into its own library that can be used between godoc and x/website (in the short term) and possibly the discovery site and godoc.org (in the long term)

[agnivade]

Is this bug tracking the short term tasks or the entirety of the long term tasks ?

[cnoellekb]

This issue will continue to track the short term tasks but I think the long term tasks are big enough to be tracked in their own issues. Does anyone disagree?

[gopherbot]

Change https://golang.org/cl/170000 mentions this issue: [release-branch.go1.12] cmd/godoc: remove golang.org serving code

[gopherbot]

Change https://golang.org/cl/170001 mentions this issue: [release-branch.go1.12] godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

[gopherbot]

Change https://golang.org/cl/165837 mentions this issue: internal/env: use golangorgenv package from x/tools

[gopherbot]

Change https://golang.org/cl/177217 mentions this issue: cmd/tip: use proxy.golang.org when building x/website

[gopherbot]

Change https://golang.org/cl/180959 mentions this issue: cmd/golangorg,content/static: render homepage using local root.html

[dmitshur]

Edit: This comment was factored out into issue #41102.

The website has been factored out into the cmd/golang command in the x/website repository. At this time, there remain viable simplifications that were not possible back when it was a shared godoc binary. Primarily, it's about reducing the number of unused configurations and code paths.

Simplification Opportunities

Currently, there are these independent configuration axes:

1. cmd/golangorg can be built with golangorg build constraint or without
2. the content/static directory can be read from disk or via the static package that needs to be go generate
   
   
   
   • new files need to be manually added to gen.go
     
     
     
     
     
     • it is possible to make a mistake of forgetting to do this step (e.g., see CL 213937)
     
     
     
   
   
3. the GOROOT directory can be provided via a .zip file (-zip flag) or directly on disk (-goroot flag).

For 1, the build constraint can be removed. It was needed back when the website was the same binary as godoc, but that is no longer the case. However, need to consider how memcache and datastore clients will be configured for local testing vs prod (currently, they're always turned on when golangorg build constraint is satisfied, always off otherwise).

For 2, as far as I can tell, the static package was needed to be able to run godoc binary without its source code on disk. When the golangorg binary is deployed, the repository filesystem is already included, and it should be always available during development.

For 3, the ability to provide GOROOT contents via a zip instead of a directory on disk is also not being used at all right now, and I don't foresee a need for this. I think it was needed back when the website was deployed to an App Engine without filesystem access. If there is a need in the future, we can re-introduce this feature.

Proposed State

If we agree to remove all 3 aforementioned options, the end result will be a single golang command without build constraints. It will read GOROOT directory from disk, and it will be provided the x/website/content directory on disk.

Deployment to production (both to golang.org, and to tip.golang.org) will continue to work. It'll be reading content from filesystem instead of the generated static package. It's already reading the favicon.ico and robots.txt files from the filesystem, as well as the entire GOROOT directory.

This is becoming a larger problem now that there is more content being moved for issue #33637, slowing down development. /cc @jayconrod

@andybons Do you think any of those unused configuration options are worth keeping? If we're in agreement they can be removed, I can send CLs.

[gopherbot]

Change https://golang.org/cl/197638 mentions this issue: cmd/golangorg, content: serve /doc from content/static/doc before GOROOT/doc

[andybons]

@dmitshur remove anything and everything you can to simplify the code. I have no strong opinions.

[gopherbot]

Change https://golang.org/cl/197721 mentions this issue: cmd/golangorg: re-add documentation for flat presentation mode

[gopherbot]

Change https://golang.org/cl/197722 mentions this issue: cmd/golangorg: remove -url flag

[gopherbot]

Change https://golang.org/cl/198061 mentions this issue: all: cleanup repository root

[gopherbot]

Change https://golang.org/cl/199058 mentions this issue: content/static/doc: add code.html

[gopherbot]

Change https://golang.org/cl/199057 mentions this issue: doc: remove code.html

[gopherbot]

Change https://golang.org/cl/199117 mentions this issue: doc: remove docs.html

[gopherbot]

Change https://golang.org/cl/207262 mentions this issue: content: update recompiling static assets directions

[gopherbot]

Change https://golang.org/cl/207777 mentions this issue: cmd/godoc, godoc/static: remove remnants of golang.org website

[gopherbot]

Change https://golang.org/cl/208259 mentions this issue: cmd/godoc: delete unused autocert hooks

[gopherbot]

Change https://golang.org/cl/210797 mentions this issue: content/static/doc: copy Release History pages from Go repo

[gopherbot]

Change https://golang.org/cl/210798 mentions this issue: doc: remove Release History pages (moved to x/website)

[gopherbot]

Change https://golang.org/cl/211300 mentions this issue: content/static/doc: copy Go Security Policy page from Go repo

[gopherbot]

Change https://golang.org/cl/211301 mentions this issue: doc: remove Go Security Policy page (moved to x/website)

[gopherbot]

Change https://golang.org/cl/213157 mentions this issue: doc: remove root.html and conduct.html (moved to x/website)

[gopherbot]

Change https://golang.org/cl/213937 mentions this issue: content/static: add gopath_code.html file and regenerate

[gopherbot]

Change https://golang.org/cl/229081 mentions this issue: cmd/golangorg: generate release history page from internal/history

[gopherbot]

Change https://golang.org/cl/229483 mentions this issue: cmd/golangorg: generate major version list on Go project page

[gopherbot]

Change https://golang.org/cl/229482 mentions this issue: content/static/doc: copy The Go Project page (contrib.html) from Go repo

[gopherbot]

Change https://golang.org/cl/229485 mentions this issue: doc: remove The Go Project page (moved to x/website)

[gopherbot]

Change https://golang.org/cl/243118 mentions this issue: internal/dl: add CORS support to JSON endpoint

[dmitshur]

I think this can be closed. The golang.org website and most of its content has been factored out into the x/website repository, and it's been in production for a long while. There's more that we can optionally deal with in the future, but there's nothing critical left.

I've factored out some simplification opportunities from https://github.com/golang/go/issues/29206#issuecomment-536099768 into a separate issue #41102.

If there's anything more that needs to be done, let's open individual issues.

Thanks to everyone who contributed to this successful move!