Go: x/tools/cmd/godoc, gddo: module support

[agnivade]

I wanted to open an umbrella issue discussing the changes that we might need to do to adapt godoc now that we will start to have modules.

• ~For starters, now that we have $GOPATH/src/mod folder containing versioned snapshots of various libraries, all of them show up in the /pkg/ html output.~

~What is the recommendation to handle this case ? To simply dump all versions of all packages does not seem like the right thing to do. It also gives a sense of exposing internal details of storing "@vX.Y.Z" along with folder names.~

(This is taken care of by moving from src/mod to pkg/mod)

• Also, for new systems that are completely working outside of GOPATH, how do we decide what to show in the output of /pkg/ page ? In the current workflow, it won't show anything because $GOPATH/src is empty. Everything is inside $GOPATH/pkg/mod.

Can we improve this ? Should we add an option to show godoc for a specific module root ? Or some notion to show documentation of only a given package.

I have not been following the progress on modules. So please let me know if there are some obvious fixes to these.

@rsc @bradfitz

[kr]

@agnivade note that in b8f42d74e87aeec189f53e9fdb2a7e6026c099b1, the module cache moved to $GOPATH/pkg/mod, so this particular "for starters" issue seems like it's been addressed. If you still have anything in $GOPATH/src/mod, you can delete it.

[agnivade]

Indeed. Thank you.

[thepudds]

@gopherbot please add modules label

[agnivade]

I don't think this is a modules issue. It is about changes in godoc due to modules.

[acln0]

I have been directed here from the #modules channel on the Gophers slack. Hopefully this is the right place to give a report.

I'm writing a new package using modules, outside GOPATH. I have not pushed this package anywhere public just yet. I'd like to know how the HTML documentation renders before I do. I am writing examples and adding code blocks to godoc comments as well, so doing go doc acln.ro/foo is not enough.

So far, I've written a little shell script that copies the entire tree to GOPATH so that godoc can pick it up. As I iterate, I run the shell script. This doesn't feel quite right. I have also tried using symlinks, which godoc (rightfully) ignores.

Should we add an option to show godoc for a specific module root ?

That would certainly be nice. I would like to be able to point godoc at some directory outside GOPATH, where I am currently working.

[go101]

[moved from https://github.com/golang/go/issues/28720]

A suggestion: the godoc.org page for a package containing the go.mod file should also show an import/replace path specified in the go.mod file.

For example, this package https://godoc.org/github.com/go101/tinyrouter specified its import path in go.mod as go101.org/tinyrouter instead of its hosing path. The the doc page should also list the import path which is specified in go.mod.

It would be better if the doc page can suggest a replace line for package users
if the godoc program finds that the hosting path and the module path specified
in the go.mod file are not consistent.

An example for the TinyRouter doc page:

====

package tinyrouter

import "github.com/go101/tinyrouter" // for GOPATH based projects

import "go101.org/tinyrouter" // for modules based projects

Please add the following line in your go.mod file to use this package
if your project is modules based:

replace go101.org/tinyrouter => github.com/go101/tinyrouter v1.0.1

[galxy25]

+1 for @go101 A suggestion: the godoc.org page for a package containing the go.mod file should also show an import/replace path specified in the go.mod file. updating a public GO sdk to be modules based and now the published Godoc page for the module has an incorrect module import 🤦🏾‍♂️

[mattouille]

If godoc could support doing previews for an individual package (in the context of a directory rather than through package discovery) I think that might help a lot of use cases in the near term. Even if this were a temporary feature I think it'd go a long way.

If godoc can already do this I'd love to hear about it.

[thepudds]

Workaround from thread (similar to
https://github.com/golang/go/issues/26827#issuecomment-429315761):

The solution I came up with is to have a bash script like this in my Makefile:

mkdir -p /tmp/tmpgoroot/doc
rm -rf /tmp/tmpgopath/src/github.com/username/packagename
mkdir -p /tmp/tmpgopath/src/github.com/username/packagename
tar -c --exclude='.git' --exclude='tmp' . | tar -x -C /tmp/tmpgopath/src/github.com/username/packagename
echo -e "open http://localhost:6060/pkg/github.com/username/packagename\n"
GOROOT=/tmp/tmpgoroot/ GOPATH=/tmp/tmpgopath/ godoc -http=localhost:6060

Karel

[thepudds]

Reopening. (Sorry, fat fingered on my phone).

[mattouille]

Yeah, this is more or less what I've been doing. Instead of dumping it in /tmp I actually put it where it should go in the GOPATH, do my changes, then move it back to my normal directory. I wonder if this can be done with symlinks? I don't know of any potential side effects either. I'll report back.

[alexaandru]

Yeah, this is more or less what I've been doing. Instead of dumping it in /tmp I actually put it where it should go in the GOPATH, do my changes, then move it back to my normal directory. I wonder if this can be done with symlinks? I don't know of any potential side effects either. I'll report back.

No go with symlinks, but sudo mount -o bind does the trick :) Now I can keep my project in only one place and let godoc see it where it wants to see it... :)

[izgeri]

We are working on a workaround with Docker - mounting our code in as a volume and running godoc from the container. But really, it would be nice if we could run it locally as intended.

[dmitshur]

But really, it would be nice if we could run it locally as intended.

There is planning work being done with the goal of making that the case. I will be posting more updates in the coming weeks. Thanks for your patience on this.

[alexaandru]

There is now https://github.com/neilotoole/gohdoc :-) which does exactly that: show the docs for your project and only your project.

[odiferousmint]

There is now https://github.com/neilotoole/gohdoc :-) which does exactly that: show the docs for your project and only your project.

This doesn't seem to work for a project outside of GOPATH. It tells me Error: failed to find in server pkg list: myproj.

[shuLhan]

@acln0 @thepudds

This comment is a note for anyone who want to run godoc web on local system.

GOPATH actually allow multiple directory separated by ":", much like PATH. My source code is structured like GOPATH, for example, inside $HOME directory I have,

├── github.com
│   ├── github
│   ├── Go-ID-community
│   ├── golangci
│   └── shuLhan
├── gitlab.com
│   └── shuLhan
└── golang.org
    └── x

and then in another directory $HOME/work I have Go repositories for work related.

Combining this two directories, I can run a local godoc using user systemd service,

master ms 0 % cat .config/systemd/user/godoc.service
[Unit]
Description=godoc

[Service]
Type=simple
Environment=GOROOT=/home/ms/git/go
Environment=GOPATH=/home/ms:/home/ms/work
ExecStart=/home/ms/bin/godoc -v -http=:6060 -goroot=${GOROOT} -play -index -index_files=/home/ms/.cache/godoc.index

[Install]
WantedBy=default.target

Then I can run the service with systemctl --user start godoc and view the local godoc on web browser.

[trivigy]

@dmitshur

There is planning work being done with the goal of making that the case. I will be posting more updates in the coming weeks. Thanks for your patience on this.

Just checking in to see if there has been any development on this front?

[trivigy]

After using @thepudds workaround from above I decided to add another that eliminates the need for creating special tmp folders. This one requires docker installed. It's a simple linux bash alias that runs a function which starts docker, attaches the current directory as a volume, and configures the necessary setup inside the container. Just one liner. Tested it only on debian.

If you need a different version of golang just make sure to change the docker image tag.

# start godoc inside docker (https://github.com/golang/go/issues/26827)
alias godoc=$'function _godoc() { [ ! -f "$(pwd)/go.mod" ] && echo "error: go.mod not found" || module=$(awk \'NR==1{print $2}\' go.mod) && docker run --rm -e "GOPATH=/tmp/go" -p 6060:6060 -v $(pwd):/tmp/go/src/$module golang:1.12.6 /bin/bash -c "awk \'END{print \\\"http://\\\"\\$1\\\":6060/pkg/$module\\\"}\' /etc/hosts && godoc -http=:6060"; unset -f _godoc; }; _godoc'

The alias will print the url where you need to navigate. Make sure to run it from project directory where go.mod can be found.

[dfang]

it would be great to have mod support when develop locally so that godoc can view unpublish docs

[mvdan]

Please remember to not leave +1 comments; see https://github.com/golang/go/wiki/NoPlusOne.

[dfang]

@mvdan it's not off-topic. i mean when i develop a package outside of $GOPATH. you want show it's doc before publishing it to github, you have to ln to $GOPATH for now.

[schrej]

@mvdan it's not off-topic. i mean when i develop a package outside of $GOPATH. you want show it's doc before publishing it to github, you have to ln to $GOPATH for now.

The issue is about adding support for go modules to godoc.
You saying that it would be good to have go module support in godoc without adding any additional information is essentially just a longer version of writing +1.

[stapelberg]

cc @bcmills — this issue is open for almost a year.

Is it particularly hard to make godoc work with modules (at least making the module in $PWD available seems pretty straight-forward?), or did this fall through the cracks?

Thanks,

[mvdan]

Just to give some context - this issue has been discussed as part of golang-tools multiple times in the past year, so it's definitely not forgotten.

My understanding is that this is indeed a hard problem. Perhaps the team can share details (or failed experiments) to keep us updated, or to specify how the community can help.

[robpike]

This strikes me as a very important issue to resolve. I had to resort to moving my modules-driven projects into GOPATH, depending on the kindness of the go tool's handling of go.mod, in order to run godoc on my own projects. The issue needs a fix, soon.

[bcmills]

CC @ianthehat for x/tools, although @andybons is listed as the primary owner for tools/cmd/godoc in particular.

[dmitshur]

This strikes me as a very important issue to resolve.

@robpike I agree. This is important in order to provide a good developer experience in module mode and needs to be done soon.

There has been progress made, but I haven't posted updates recently (the last one was https://github.com/golang/go/issues/26827#issuecomment-466507591).

For golang.org/x/tools/cmd/godoc (what @robpike's comment was referring to), the current status is by now I have investigated and prototyped many of the implementation details that are needed to add full module support to it. What's left is to make some code organization decisions, and after that I'll start sending CLs for review.

For gddo, the latest update was shared in https://github.com/golang/gddo/issues/567#issuecomment-503322119.

[stapelberg]

After using @thepudds workaround from above I decided to add another that eliminates the need for creating special tmp folders. This one requires docker installed. It's a simple linux bash alias that runs a function which starts docker, attaches the current directory as a volume, and configures the necessary setup inside the container. Just one liner. Tested it only on debian.

Thanks for sharing your solution!

Unfortunately, it didn’t work for me in zsh as-is, so I’ve taken the liberty to rewrite it in a way that works and that I find more clear. I tested this in bash and zsh:

function godoc() {
  if [ ! -f go.mod ]
  then
    echo "error: go.mod not found" >&2
    return
  fi

  module=$(sed -n 's/^module \(.*\)/\1/p' go.mod)
  docker run \
    --rm \
    -e "GOPATH=/tmp/go" \
    -p 127.0.0.1:6060:6060 \
    -v $PWD:/tmp/go/src/$module \
    golang \
    bash -c "go get golang.org/x/tools/cmd/godoc && echo http://localhost:7070/pkg/$module && /tmp/go/bin/godoc -http=:6060"
}

[julieqiu]

This issue is the umbrella issue for all “godoc”-like projects that need to support modules. It’s not easy to discuss progress on individual projects here. We have created smaller tracking issues for the individual projects:

33654 - discovery: Go discovery site tracking issue

33655 - x/tools/cmd/godoc: add module support

[petersamokhin]

Wrote a small workaround for go doc & go1.13 & go modules project outside of a GOPATH:

https://github.com/nightstory/go-mod-godoc

[TheDiveO]

@stapelberg in newer Go images, at least :1.13 and :latest, godoc isn't installed by default. Thus, the bash command should start first with go get golang.org/x/tools/cmd/godoc && ... and then proceed as before. Oh, and if you're behind a corporate proxy, then this needs to be passed into the container too, such as with -e "http_proxy=$http_proxy".

Thank you very for this useful script!

[stapelberg]

Thanks, I updated my earlier comment to work with Go 1.13.

[stevegt]

Here's a generalized version of @stapelberg's code -- this one serves all of the packages in your dev tree regardless of whether they have a go.mod yet or not. Modify $devbase (or pass it in as $1) to point at the base of your tree:

#!/bin/bash 

set -x  # optional

devbase=$HOME/gohack
port=6060

docker run \
    --rm \
    -e "GOPATH=/tmp/go" \
    -p 127.0.0.1:$port:$port \
    -v $devbase:/tmp/go/src/ \
    --name godoc \
    golang \
    bash -c "go get golang.org/x/tools/cmd/godoc && echo http://localhost:$port/pkg/ && /tmp/go/bin/godoc -http=:$port"

[romandvoskin]

@stevegt Thanks so much for your script. To make things even more dead simple I replaced devbase=$HOME/gohack with devbase=$(pwd) and placed the script at the base of my repo.

[nextkitt]

Only in Go1.13, look this:

ln -s $GOPATH/pkg/mod  $GOPATH/src

and

godoc -http=:6060

[dmitshur]

For those who are following this issue because you are waiting for module support in the godoc command (golang.org/x/tools/cmd/godoc), please note that it has been implemented in the latest version. See issue #33655.

I've also sent a mail to golang-nuts to announce this.

[StarpTech]

@dmitshur I faced another issue. In my organization, we host on Gitlab and projects are namespaced with their "groups". My module and VCS name is git.$organization.se/$group/my-project.

When I run godoc -http=:6060 in root and click on the package git.$organization I see the following error in the terminal:

using module mode; GOMOD=/mnt/ssd/repositories/golang-dynaday/go.mod
2020/03/07 00:03:31 cannot find package "." in:
        /src/$organization

This results in a broken UI and thus my packages aren't listed correctly.
go list -m all list correctly all packages and modules but godoc doesn't display them.

The workaround is to remove $group name from the module name and update all places accordingly. I'm not aware of that constrain.

go.mod

module git.$org.com/internal/golang-dynaday

go 1.14

replace git.$org.com/internal/golang-dynaday/submodule => ./submodule

require (
    git.$org.com/internal/golang-dynaday/submodule v0.0.0-00010101000000-000000000000
)

[dmitshur]

@StarpTech Thanks for reporting. Do you mind opening a new issue with a "x/tools/cmd/godoc: " prefix? That will make it easier to track that specific problem.

I have a guess about what the problem may be. I have a WIP CL 205661 which I suspect may resolve the issue. If you can make a smaller reproduce case, that would be very helpful to confirm whether or not it it'll solve your problem. (Alternatively, you can try that CL yourself and let me know.)

[StarpTech]

@dmitshur thanks for the quick response. I created #37729 and there you find the feedback for the patch.