Go: cmd/go: track tools/tooling updates to support modules

[flibustenet]

Many Go tools that previously worked with GOPATH need to be upgraded in order to work with modules.

Note: Many of these tools have been deprecated with modules, and their features have been incorporated into gopls. This is indicated below.

This is a tracking issue for the conversion of tools to work with modules.

• [x] aligncheck - Merged - opennota/check#43
• [x] bundle - (#32031)
• [x] counterfeiter - https://github.com/maxbrunsfeld/counterfeiter/issues/90
• [ ] deadcode
• [x] delve - Working
• [ ] dupl
• [ ] eg
• [x] errcheck - Beta - kisielk/errcheck#148
• [x] fillstruct - available through gopls, or https://github.com/davidrjenni/reftools/issues/31
• [ ] gas
• [x] goa/goagen - goadesign/goa#1914
• [x] go-langserver - use gopls instead
• [x] go-outline - also available through gopls
• [ ] go-symbols
• [x] gocode - available through gopls or https://github.com/stamblerre/gocode
• [ ] goconst
• [x] gocyclo - doesn't need an update, see https://github.com/golang/go/issues/24661#issuecomment-715614373
• [x] godef - available through gopls or github.com/rogpeppe/godef
• [x] godoc - #33655
• [x] godoctor - available through gopls
• [x] gofmt - working and also available through gopls
• [x] gogrep - working (https://github.com/mvdan/gogrep/issues/39)
• [x] go-fuzz - (https://github.com/dvyukov/go-fuzz/pull/274)
• [x] gogetdoc - working (zmb3/gogetdoc#40)
• [x] goimports - working and also available through gopls
• [ ] golint - Beta - https://go-review.googlesource.com/c/lint/+/127695
• [ ] gometalinter - Deprecated - https://github.com/alecthomas/gometalinter/issues/590
• [ ] gomobile
• [x] gomodifytags
• [ ] gomvpkg
• [x] gopkgs - unnecessary with gopls
• [ ] goplay
• [x] gorename - available through gopls
• [x] goreturns - available through gopls
• [x] gosimple
• [x] go-sumtype - (https://github.com/BurntSushi/go-sumtype/pull/9)
• [x] goswagger - go-swagger/go-swagger#1991
• [x] gotags
• [ ] gotests
• [ ] [gotext](https://godoc.org/golang.org/x/text/cmd/gotext)
• [ ] gotype (@ramya-rao-a confirmed this was not required for vscode-go)
• [x] gotype-live - available through gopls
• [x] govet - also available through gopls
• [x] gqlgen - (99designs/gqlgen#514)
• [x] ~guru~ - replaced by gopls
• [ ] impl
• [ ] ineffassign
• [ ] keyify
• [x] lll - Working
• [x] maligned
• [x] megacheck
• [ ] misspell
• [x] mockery - (https://github.com/vektra/mockery/pull/218)
• [ ] motion
• [ ] nakedret
• [ ] safesql
• [x] staticcheck - also available through gopls
• [x] stringer
• [x] structcheck - Merged - opennota/check#43
• [x] unconvert - merged - mdempsky/unconvert#32
• [x] unparam - https://github.com/mvdan/unparam/commit/0cc8df20029c4cc92b4af5da062d32dc41b21c89
• [x] unused
• [x] varcheck - Merged - opennota/check#43

[rsc]

Yes, this is an issue we're very aware of. Will leave this open to track it.

[myitcv]

@rsc - just to check this issue effectively encapsulates what we discussed in golang-dev?

[myitcv]

Changed the title so that this issue matches on use of the words "tool", "tools" or "tooling"

Will keep a list of tools here too (please edit):

• Requiring build output:
  
  
  
  • guru
  
  
  • gocode
  
  
  • stringer
  
  
• Requiring source code
  
  
  
  • goimports
  
  
  • godef
  
  
  • godoc.org (https://github.com/golang/gddo/issues/567)
  
  

[gopherbot]

Change https://golang.org/cl/105855 mentions this issue: x/vgo: add deplist command to get build information about (test) deps

[rsc]

See https://go-review.googlesource.com/c/tools/+/116359 for a sketch of a start.
/cc @alandonovan

[gopherbot]

Change https://golang.org/cl/125296 mentions this issue: go/build: invoke go command to find modules during Import, Context.Import

[myitcv]

@alandonovan @ianthehat - is it worth creating a number of issues, specific to each tool that's being worked on?

I keep directing people to this umbrella issue... but maybe there's merit in getting a bit more granularity on things? Equally, maybe not.

So I'm definitely wouldn't push back if you said "let's keep things here" 👍

[sam3d]

How about the case of goimports importing packages that are both:

• Within the module's local directory
• Outside of the system scoped $GOPATH

---

Assuming the following directory structure:

./
├── go.mod
├── main.go
└── test
    └── test.go

With ./go.mod containing:

module example.com

goimports will happily leave alone and even format ./main.go:

package main

import (
    "log"

    "example.com/test"
)

func main() {
    log.Print("Hello, world!")
    test.Start()
}

With ./test/test.go:

package test

import "fmt"

func Start() {
    fmt.Println("Started properly")
}

---

However, if the "example.com/test" import is removed from main.go, it wont get added or found again. If its import path is changed in any way to render it incorrect (say to "example.com/tes"), goimports simply deletes the line.

It's odd - goimports can clearly tell that package test exists when it's been provided already, but it can't actively seek and add it if it hasn't been.

Is this something that requires looking into the module cache for, or is this a different issue?

[ianthehat]

goimports will assume that the import line "example.com/test" imports a package called test if it cannot find any sources for that package, and it knows the test symbol is used, so it won't delete the line. It does not mean that it knew the package existed...

goimports scans your GOPATH and GOROOT, anything not there it will not find right now.

It will be made to use x/tools/go/packages plus some extra stuff that is beyond the scope of that package. When it does, this case will work (does not involve the module cache, just the list of packages in the current module, which you can already get with go mod -packages

[sam3d]

Thank you for explaining @ianthehat! I've misunderstood how goimports handles symbols.

[metakeule]

I have two questions:

1. will x/tools/go/packages move to the standard library with Go 1.12?
2. I can't see any information about the version of a plugin, so how is a tool that assists in importing (like GoSublime) supposed to get a list of possible imports (with versions)?

[metakeule]

Also: How to prevent that import completion will get really slow with larger projects? (see https://github.com/mdempsky/gocode/issues/32)

[myitcv]

@metakeule

will x/tools/go/packages move to the standard library with Go 1.12?

The plan is for it to move to the standard library at some point, yes, once it has stabilised (ref).

I can't see any information about the version of a plugin

This is similar to the goimports "problem"; see https://github.com/golang/go/issues/26717

How to prevent that import completion will get really slow with larger projects?

Many of the problems describe in https://github.com/mdempsky/gocode/issues/32 relate to speed problems brought about by the use of the -source flag, which drives the use of the source importer. go/packages will work using the underlying build cache and so should not suffer these problems.

But in case lag does become an issue for gocode _with its current architecture_, go/packages will be offering support for long-running programs. So it's conceivable that gocode's cache will switch to be module-based, where it effectively "watches" for relevant changes and refreshes its cache accordingly. Its cache will then be primed for fast responses to ad-hoc queries.

cc @dominikh (in case there's any overlap of interest here with your work)

[DisposaBoy]

@myitcv

go/packages will be offering support for long-running programs

Is this documented anywhere? I've been AFK for a most of the last couple weeks so I'm somewhat OOTL, but from this message on golang-dev https://groups.google.com/d/msg/golang-dev/Pb8L70Ps1EQ/LBvl_KFTCwAJ I got the impression that this would not be the case.

quote:

the x/tools/go/packages API is really intended for direct use by a command line tool, not repeated use by a long running tool or underlying use for a library.
[...]
We have plans for another level of API for people writing tool libraries

Is this quote wrong/outdated/did I mis-understand/....?

It's all very confusing and frustrating because many of the current packages like go/build rely on global variables (go/build.Default, os.Environ, etc.) making it very difficult and error-prone to use them in long-running servers and it seems like the plan is to start shelling out to external commands, partially defeating the whole purpose of having a long-running server that can do caching, etc.

[alandonovan]

go/packages does not have any special support for long-running programs. It does not cache the results of metadata queries made to the underlying build system, nor does the API permit a sequence of calls (or concurrent calls) for additional packages, though we did consider supporting this in earlier drafts.

The reasons go/packages executes an external command are (a) to enable portability across build systems and project environments (go, vgo, bazel, buck, etc) , and (b) to avoid duplicating build system logic inside every application, which would necessitate recompiling each application each time the build system changes. Unfortunately these two forms of flexibility do come at the cost of increased query time, though I will note that the go list command is impressively fast.

We are conscious of the risk of excessive reliance on global variables; go/packages does use the process environment and working directory, but also exposes them as parameters to give clients the control they need.

[myitcv]

@DisposaBoy

go/packages will be offering support for long-running programs

Is this documented anywhere?

It seems I saw the early draft 😄 - quoting @alandonovan

though we did consider supporting this in earlier drafts.

[DisposaBoy]

I understand the reasoning for using external commands, but I'd guess that many of these tools will need re-compiling from time-to-time anyway so that's not a big deal. On, the downside, at least IME, go list is quite slow.

With a weak (~1.10ghz) CPU and shitty m2 ssd (still much faster than hdd), go list -export -f '{{.Export}}' golang.org/x/tools/go/packages takes ~190ms. It drops to around ~120ms without -export (which is still very slow) but IIUC, I now need -export (with its own downsides) because of the package cache.

I also tried go/packages but it took ~2 seconds with config Mode: packages.LoadTypes (most likely PEBKAC). A nil config took ~210ms but contains no types and also don't tell me where the compiled package is.

I don't have any numbers, but I will note that on a much more powerful machine, things did also become noticeably slower when the package cache was introduced.

All this just to find out where a compiled package is. What am I missing?

In the past (and today with go install -i), supporting just the go tool meant that given an import path, I could find the compiled package with little more than string concatenation.

Sorry for all the doom and gloom, but this release is giving me flashbacks of Python3.

[rsc]

Pushing tracking issue off to Go 1.12.

[flibustenet]

@rsc could you clarify (here or elsewhere in a faq ?) how will continue the work on integration of tools with modules, and also upgrade of go mod commands after go1.11 final release ?

• For go mod commands, i believe we will need to works on go tip to follow improvements ?
• For go tools will they be released independently to can works with modules before go1.12 or will we need to wait or works on go tip also ?
  thx

[ianthehat]

Many of the tools are not owned or released by the go team, and their release is not tied to the go release cycle. We are continuing to work on them and put up pull requests to the owners as they are ready.
Our main focus is on the support library (x/tools/go/packages) and the editor experience, most of the tools invoked by editors are common across all the editors.
We are also happy to advise and help anyone who wants to help with tool conversion or has a tool they think we will not do for them.

[dominikh]

What's worth noting is that

1. the plan is to get go/packages usable by the time Go 1.11 releases
2. go/packages doesn't work all that great with Go 1.10 yet, and may depend on Go 1.10.4 to work fully.

It seems unlikely that most tools will (or should) move to go/packages before the second point is resolved. At least that's true for my tools, and errcheck (c.f. https://github.com/kisielk/errcheck/issues/150)

[myitcv]

@dominikh regarding:

go/packages doesn't work all that great with Go 1.10 yet, and may depend on Go 1.10.4 to work fully.

I'm clearly doing the wrong search, but please can you provide a link to this issue?

[dominikh]

@myitcv A combination of https://github.com/kisielk/errcheck/issues/150, https://groups.google.com/d/msg/golang-dev/Pb8L70Ps1EQ/Z8bwJrOjBgAJ, https://go-review.googlesource.com/125616

Alas, I've looked at Go 1.10.4 and don't see backports of any of the go list features required to make go/packages work...

[dominikh]

Also relevant:

• https://go-review.googlesource.com/c/tools/+/131015
• https://go-review.googlesource.com/c/tools/+/130576

[ianthehat]

I have modified this issue to track the full list of tools, and assigned it to me.
I did this rather than file a new issue because it seems this issue is already referenced in quite a few places. I apologies to @flibustenet for deleting all your words, but this seems like the most practical approach.
Feel free to file individual tracking issues for each tool where needed, and to add any tools not in the list.
We will continue to update this issue moving forwards.

[agnivade]

@ianthehat - Could you take a look at https://github.com/golang/go/issues/26827 and close it if you think it falls under this issue ?

[aarzilli]

dlv

Delve 1.1.0 works with Go 1.11 and as far as I know doesn't have any problem with modules. I just tried with a toy example and everything seemed to work. If you are aware of something I'd like to know.
It just calls go build and uses the output executable, btw.

[ianthehat]

@agnivade : godoc is a special beast with all sorts of module issues.
We intend to fix godoc (or an alternative) so that the usage an IDE makes of it works as part of this issue, and possibly fixing the single session local web server variant, but fully fixing godoc in all it's flavours to be module and version aware is a whole separate task, and probably needs its own issue outside of this one.

[e-dard]

I have been using a next branch on staticcheck that seems to be working very well with modules. Maybe link it on the list on the PR description?

[mvdan]

I ported over unparam last week and noone has complained so far, so I've ticked it off from the list above.

[jirfag]

hi!
golangci-lint now works with go modules (uses go/packages), add it please

[solumos]

Are there any updates on issues related to go/packages or similar? I see that stringer is still unchecked above, and there's no activity on the linked PR (I'm planning a similar refactor from build to packages for https://github.com/percolate/charlatan/issues/29).

Apologies if this isn't the right place for a question like this.

[ianthehat]

Stringer is a bit special, it requires us to vendor go/packages into the standard library, which we are not quite ready to do.
In general, go/packages is ready for use but still slower than the systems it is replacing, we are waiting on speed improvements to go list before merging some of the tools. Others that might be slightly less speed sensitive or not trigger the slow paths have just been converted already.
I would think for something like charlatan it should be fine, let us know how you get on.

[ianlancetaylor]

Minor point, but why is stringer special? It doesn't live in the standard library, it lives in x/tools/cmd/stringer.

[myitcv]

In general, go/packages is ready for use but still slower than the systems it is replacing, we are waiting on speed improvements to go list before merging some of the tools.

Could we not put the modules-enabled changes behind a "standard" build tag, e.g. go111module (for some symmetry with the GO111MODULE env variable)?

~_It's a shame that the go tool's build-related commands don't allow the -tags flag to be repeated, otherwise it might have been possible to make use of build-tag guarded tools/code in more transparent way by setting GOFLAGS=-tags=go111module in one's profile/.bashrc/wherever._~

[myitcv]

Updated my previous comment; my testing with multiple -tags flags was clearly bogus. The GOFLAGS approach _does_ work. Which, to my mind, makes maintaining a modules-aware version of a tool/package alongside the non-module-aware code in x/tools and friends a more attractive option.

The overhead will be similar to maintaining a separate branch/fork (of each tool), but the cost in terms of people now not needing to switch references to another repo will be much lower, which is a win.

[ianthehat]

I don't think that would really help.
The interesting cases right now are the IDE integrations, and asking them to build two copies from the same repository with different tags is probably not any easier than asking them to build two copies from different repositories. There is also a certain amount of overhead in having the person writing the fix have to maintain code in a repository they are not a maintainer of.
Maybe we should discuss this in the meeting tomorrow?

[myitcv]

Maybe we should discuss this in the meeting tomorrow?

Just to note here, we concluded that a build-tags based approach is likely no better, indeed potentially worse, than the current planned approach. So no need to diverge from that for now.

[atombender]

@flibustenet Can you add Mockery to the list? Fixed.

[atombender]

@flibustenet Also, go-sumtype. Fixed.

[mtibben]

Another tool: gqlgen

[powerman]

golangci-lint now works with go modules (uses go/packages), add it please

This is great as it let us work around incompatibility of many individual lint tools embedded in golangci-lint.

Maybe it's worth to re-prioritize work on tools in the list and give refactoring tools like gorename higher priority, because absence of such tools is a real blocker for moving projects out of GOPATH.

[myitcv]

@powerman - gorename came up on the on the last golang-tools call. What editor do you use out of interest?

[keegancsmith]

I use gorename a lot. I do it from emacs. As a workaround I just run go mod vendor (also fixes other tools on that list, eg I use guru references a lot).

[myitcv]

Thanks @keegancsmith - I've added an agenda item for the next call on 27 Nov to make sure we cover this more thoroughly. In the meantime, @ianthehat / @alandonovan might be able to provide some more detail.

[powerman]

@powerman - gorename came up on the on the last golang-tools call. What editor do you use out of interest?

This is good news, thanks! (I've checked #27571 but there is no activity in last two months, that's why I've asked about priorities.) I'm using vim-go.

[groob]

Is anyone tracking eg yet?
We use eg for a number of rewrites, but unfortunately it is not able to find dependencies without GOPATH and/or vendor/.

I end up with something like this in our Makefile:

 eg is not module aware, so we add the vendor/ directory, run generate without modules
# and then remove the vendor directory.
eg:
    go mod vendor
    GO111MODULE=off eg -w -t some.template github.com/groob/pkg
    rm -rf vendor/

[bcmills]

Does the godoc entry above cover cmd/doc, or should there be a separate entry for it?

[agnivade]

It doesn't actually. godoc is tracked separately here - https://github.com/golang/go/issues/26827. It is just there on the list for completeness.

[casualjim]

adding go-swagger to this list

[mvdan]

I just finished porting gogrep over from go/loader, which was fairly straightforward. My only gripe, which I've mentioned before, is that there's no LoadSyntax or LoadAllSyntax without type info. Which I realise can be done with LoadImports and go/parser, but that seems like unnecessary boilerplate.

[tbarbugli]

errcheck added support for go modules on v1.2.0 🎉

[mdempsky]

FYI, something that caught me by surprise in the transition of mdempsky/unconvert from go/loader to go/packages was that the go/token.File.Name() for cgo-rewritten sources now points into the go-build cache instead of pointing to the pre-cgo source file, which meant that unconvert -apply could corrupt the build cache instead of rewriting the original source files.

Heads up to any other authors of automated-rewrite tools to be careful about this.

On the upside, I can confirm that unconvert -apply will fully work with cgo-processed source files starting with Go 1.12, thanks to the fix for #26745.

[andig]

Is there any update on the stringer issue? The patchset is from august?

[npgm]

Noticed that golang/mock is not being tracked here: https://github.com/golang/mock/issues/230

[F21]

The latest master of stringer works with modules now.

[josharian]

I’m working on migrating go-fuzz to use go/packages. It is slow going. :)

Once that is done, there will probably be some further steps to properly support modules; I’m not sure yet.

[josharian]

go-fuzz PR for go/packages: https://github.com/dvyukov/go-fuzz/pull/211. As anticipated, I believe there’s still more to do for module support. I don’t currently plan to work on that.

[ptone]

adding a reference to https://github.com/src-d/proteus/issues/107 to track proteus

[ramya-rao-a]

@ianthehat Can we add https://github.com/godoctor/godoctor to the list being maintained here?

[atombender]

go-sumtype works with modules now.

[kush-patel-hs]

Any update for goreturns? so VSCode can work out of box way it used to in $GOPATH mode instead of having to change the config to point to goimports

[wingyplus]

How about gomvpkg? Is there still support on module?

[tschellenbach]

this is starting to feel like a python 2 to 3 moment for the go language.

[mirtchovski]

this is starting to feel like a python 2 to 3 moment for the go language.

i don't think so. mostly everyone is now using modules. this reflects only the fact that a lot of the tooling that propped up in Go's early life is hobby level and generally unsupported/abandoned.

[caarlos0]

gometalinter was deprecated...

https://github.com/alecthomas/gometalinter

[baor]

Hi all,
does anyone know if there is any work in progress for guru and gorename?
is it expected from the community to take part in fixing tools or all hope for go core team? :)

[mtibben]

gqlgen: done in https://github.com/99designs/gqlgen/pull/514

[dominikh]

@baor guru and gorename will be superseded by gopls. And at least for guru, there are no plans to maintain it further, though patches would be accepted from what I heard.

[marco-m]

@dominikh (or anybody who can edit this issue): for all poor souls arriving to this ticket via a google search, would it be possible to edit the very first post in this issue and mention, for each listed tools concerned, the ones that are meant to be superceeded/replaced by gopls, such as for example guru ? This would greatly reduce the frustration :-) Thanks for your help

[myitcv]

edit the very first post in this issue and mention, for each listed tools concerned, the ones that are meant to be superceeded/replaced by gopls, such as for example guru

@ianthehat @stamblerre might be best placed to do such an edit/provide commentary.

[ssoroka]

Please add golang.org/x/text to the list; thanks!

[mirtchovski]

Looks like "fillstruct" got support for modules 5 days ago:

https://github.com/davidrjenni/reftools/tree/master/cmd/fillstruct

tested and it works, which is great news for me: i'm using it a lot on menacingly large structs.

[casualjim]

go-swagger now has support for modules, it was released a few weeks ago

https://github.com/go-swagger/go-swagger/pull/1991

[preslavmihaylov]

gorename will no longer be supported as gopls already implemented renaming functionality.

https://github.com/golang/go/issues/27571

[flibustenet]

Is goreturns going to be implemented in gopls and deprecated ?

[ianthehat]

That would make sense, there are two ways to do what it does better, one is with completions triggered on return statements, so they are not wrong in the first place, the other is with an analyzer that detects broken return statements and supplies a suggested edit to fix them.
It is a perfect starter project for a gopls contributor if anyone wants to pick it up...

[gopherbot]

Change https://golang.org/cl/189818 mentions this issue: cmd/bundle: use caller's module version via go/packages

[thepudds]

dvyukov/go-fuzz was listed above as not supporting modules, but updated list to indicate it does now support modules (https://github.com/dvyukov/go-fuzz/pull/274).

The current approach for go-fuzz works reasonably well for fuzzing modules, but it is not the "forever" solution for go-fuzz. In particular, the current approach will stop working if or when GOPATH mode goes away completely (because a portion of the implementation still relies on being able to disable module mode temporarily, though that is more-or-less transparent to the user).

[gopherbot]

Change https://golang.org/cl/255053 mentions this issue: cmd/bundle: expand test coverage to all modes

[powerman]

Maybe it's time to update this list? I'm pretty sure most of mentioned linters works just fine today with modules (at least within golangci-lint). Probably other things has changed too since Feb.

[stamblerre]

Just went through and updated a bit. A lot of the tools in this list have been replaced by gopls as well.

[fzipp]

I am the author of gocyclo. It does not need an upgrade to support modules, it works on a file-by-file basis.

[mdempsky]

maligned (github.com/mdempsky/maligned) is updated to support modules. (It's now implemented as an x/tools/go/analysis.Analyzer.)