Go-tools: go-langserver: implement a Language Server for Go

[dominikh]

Implement a Language Server for Go that supports all the common features as well as hooks into our linters. Will use the new loader package to provide incremental rebuilds, which will be much faster than calling standalone tools repeatedly.

Where possible we should implement our own logic. More complex matters may be delegated to vendored and patched versions of guru.

Desired features

• [x] Report diagnostics for build errors
• [ ] Report diagnostics for linters (may want to vendor golint and errcheck)
  
  
  
  • [ ] megacheck
  
  
  • [ ] go vet
  
  
  • [ ] golint
  
  
  • [ ] errcheck
  
  
• [x] Support the hover request, displaying documentation
  
  
  
  • [ ] display struct field alignment(?)
  
  
  • [ ] display type size(?)
  
  
  • [ ] display VRP-based information
  
  
• [x] Support signature request, displaying function signatures
• [ ] Support references request. Will likely depend on guru.
• [x] Support documentHighlight request, highlighting instances of an identifier in the same file
  
  
  
  • [ ] Use the read and write highlight kinds instead of text
  
  
  • [x] Support highlighting more things than just identifiers. Struct fields, imports and so on.
  
  

• [x] Support documentSymbol request, listing all symbols in the file

  • [ ] Support the same for workspace queries
  • Due to Go's use of GOPATH, and the stdlib, would it make sense to offer combinable configurable modes of workspace-wide symbol queries?
  1. Only search in the VS Code workspace, i.e. the opened folder
  2. Search in the entire GOPATH
  3. Search in GOROOT

  If we do support such large scopes, we will definitely need good caching and also not rely on having all packages in memory at once.




• [ ] Support formatting request, using go/format to gofmt Go documents



• [x] Support definition request, a la godef, but with our own implementation


• [ ] Support code actions. Most of gosimple's warnings can be turned into automatic fixes, maybe some of staticcheck, too.


• [ ] Support rename request, by deferring to gorename.

Maybe features

• [ ] Code completion by delegating to gocode (or writing our own)
• [ ] Support document links. Maybe linkify imports, if that fits into the vscode ecosystem. Maybe mentions to RFCs in comments, a la godoc.org. Anything else?

Experimental extensions

There are a number of Go-specific features that don't map to standard LSP commands. LSP 3, however, supports experimental extensions, which act as vendor-specific extensions. We may be able to offer some extra functionality that is currently handled by editors and external tools.

• [ ] Return a list of importable packages, for example for vscode-go's Go: Add Import feature.
  
  
  
  • Caching combined with fsnotify could make it extremely fast to get the list of packages
  
  
• [ ] https://code.visualstudio.com/docs/editor/editingevolved mentions Go To Type Definition and Go To Implementation. Can we add support for that? Similarly, can we support Open symbol by name? Update: Apparently there might be methods for that functionality in LSP 4.0, c.f. https://github.com/Microsoft/language-server-protocol/issues/173

Required architectural changes

• [ ] The current implementation of loader only accepts packages that type-check fully, because we can only create SSA programs out of well-formed code, which is required by the linters. As a language server, however, we will get a lot of invalid code as the user is typing. While we can't lint or do any complex analysis on that code, we still want to support signature information, Go To definition and other features that can work on partially type-checked packages.

  This change will require loader to not build SSA form for broken packages or any of their reverse dependencies. This will likely be achieved by setting some flag and not building SSA form for those packages, as well as not pass them to the linters.

• [ ] The current loader prototype needs to retain all compiled packages in memory. That may be okay when only compiling packages we work on + their forward dependencies. It is, however, not acceptable once we compile all reverse dependencies, for queries such as References.

  We should design a serialisation format, akin to object files, that allows us to store the AST, type information and SSA form on disk, and load these on demand. We have to use our own format, as opposed to gc object files, because the latter doesn't store all the information we need.

Code rebuilding

The long-term goal for (re)compiling packages, caching and memory usage looks something like this:

1. When the user opens a file, compile the surrounding package

2. When the user modifies code, mark the changed package, and all reverse dependencies, as out of date.

3. After an idle timeout, or when we require type information (e.g. when hovering), we rebuild the package that we require. This will not automatically rebuild reverse dependencies unless required (such as by rename and references)

4. If the package can't be type-checked fully, we'll do as much work as possible and mark the package and its reverse dependencies as incomplete. For incomplete packages we will offer a reduced set of features (no renaming, for example.)

If a package failed to compile, and neither its own code, nor that of any of its dependencies changed we shouldn't try to recompile it.

As far as memory usage and caching go, there are several incremental approaches of increasing complexity. The one we will implement first is as such:

• All packages that we require for a computation will be loaded into memory
• Additionally, the packages that the user is currently actively editing will be pinned in memory
• Additionally, we won't immediately unload unused packages. Instead, we will use a LRU cache with a configurable maximum memory usage. When we need to load new packages and exceed our memory limit, cached packages will be evicted.
• When loading a package, we will use information cached on disk, unless the sources changed, in which case we build the package from scratch

A more advanced approach, requiring modifications to the go/types package, would allow us to load and unload packages in a more fine-grained fashion. Instead of loading all packages for a query into memory, we could stream packages in and out as the algorithms are inspecting the packages.

An even more complex approach could forego the internal go/types representation and instead use a disk-backed database, likely a graph database. Some queries, such as references could be solved entirely with graph queries if the graph is up to date.

Preview

Some previews of features using just our language server:

Screenshots

As-you-type type checking

Identifier highlighting

Hover

/cc @ramya-rao-a in case she has any ideas/requests.

[szuecs]

To browse larger projects, for example Kubernetes, it would be super helpful to integrate guru queries and build an index for the calls to "implements interface function". I am not sure if this should be in this project or you integrate guru queries here and it should be somehow added in guru.

[ramya-rao-a]

Desired features

That's a good set of desired features to begin with. One other thing I can think of is workspace symbol search feature. See https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#workspace-symbols-request

Support references request. Will likely depend on guru.

On large codebases, guru is known to be slow. Can we have our own implementation here?

Support code actions. Most of gosimple's warnings can be turned into automatic fixes

That would be awesome

Maybe features

Code completion by delegating to gocode (or writing our own)

For code completion, it would be better to write our own than delegating to gocode. See https://github.com/nsf/gocode/issues/307. There is so much more we can do if we can have our own completion provider.

Support code lenses (what do we want to do with them?)
Can we use them to integrate with goimports? Could we add imports as you type code?

Codelens are the below. I don't see how you could add imports using these. Are you referring to code actions instead?

Experimental Extensions

Open symbol by name

If we implement https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#textDocument_documentSymbol, then we get Open symbol by name for free as it internally calls the documentSymbolProvider

Go to implementation and Go to Type definition

I don't see LSP support for these yet, but should be easy enough to add. I'll get in touch with the VS Code team and see when it can be done. https://github.com/Microsoft/language-server-protocol/issues/173 and https://github.com/Microsoft/language-server-protocol/issues/156

Return a list of importable packages

Not just importable packages, it is also useful to have an internal implementation to get list of non vendor packages. It is useful when trying to build/test workspace

List of importable packages will also help in adding imports when auto completing symbols from unimported packages

[dominikh]

On large codebases, guru is known to be slow. Can we have our own implementation here?

Guru is slow because it has to parse and type-check all of the code every time. We'd create a fork that can use our cached type information. The references query is simple enough to just write ourselves, too, though.

For code completion, it would be better to write our own than delegating to gocode

That will be a long-term goal. Writing code completion based on
go/types is a bit tricky, because it isn't as forgiving to
incorrect code as we'd want from code completion.

Codelens are the below. I don't see how you could add imports using these. Are you referring to code actions instead?

I suppose I am.

[dominikh]

@szuecs Caching of type information, and making guru (or our own implementation of guru) work with it is part of this project, sort of. It will be part of my tools repository, and will also be used by the standalone tools. The goal for the language server is to have nearly instant feedback on queries, like you get with IDEs like Gogland.

[tylerstillwater]

This is absolutely brilliant. I love this. Thanks so much for making this happen, @dominikh!

It looks like basically all the things I was interested in having are already implemented or planned, so I can't think of much to add in terms of requests or ideas.

One thing that I do miss from other development environments is powerful refactoring tools. I don't know if that is something that would be in scope for this project, or what would be required to make it happen, but it would be wonderful to have.

[ramya-rao-a]

Yes indeed. Refactoring would be very cool.

There is no explicit plumbing in vs code for refactoring, but I believe we can make use of commands instead.

[myitcv]

Very much like the sound of this.

I'm sure you've seen the thread @dominikh but this conversation just started in golang-dev: https://groups.google.com/forum/#!topic/golang-dev/mLsGwoFwqoI

[tylerstillwater]

Been a while since I checked in on this. How's it coming along? Is this something the community could help with?

[dominikh]

@tylerb I had to focus on other changes first. I'm predicting the language server release for the 2nd quarter of 2018, but don't quote me on that please.

There's not really much the community can help with (other than feature requests). An efficient implementation of the language server requires some rather complex internal changes that I have to do myself. On top of that, I have to balance the LS server, paid work and other (partly commercial) staticcheck features. LS got a bit of the short end. Luckily enough, some of the commercial functionality will require the same foundation as the language server.

[tylerstillwater]

Makes sense! Thanks for working on this project. If there is any way I can help support it, let me know! Do you have a patreon or gofundme or some other way I can contribute financially?

[dominikh]

@tylerb There's a Patreon at https://www.patreon.com/dominikh but be sure to read https://www.patreon.com/posts/thank-you-future-13530779 first!

[tylerstillwater]

Will do! Thanks again!

[hickeng]

@dominikh Is the language server in a remotely viable early form? I'm putting together a hosted development environment for vmware/vic but run into the same issue as Microsoft/vscode-go/issues/1188 - I want vscode to run the tools in the remote VM/container.

While I could easily throw together script to relay the tools invocations remotely I'd rather build on something that's less kludged together.

[dominikh]

@hickeng It's not in a usable form yet, no. It will probably be another 2-3 months.

[dgryski]

Also https://github.com/sourcegraph/go-langserver

[szuecs]

@dgryski as far as I can see https://github.com/sourcegraph/go-langserver does not support interface to implementation lookup, right?

[keegancsmith]

@szuecs FYI the language server protocol doesn't have that sort of query as a first class method. However, I believe it should be possible via a codeAction or a custom extension.

The references implementation in https://github.com/sourcegraph/go-langserver is forked off go guru. We would accept a PR which implements that code action :) But I look forward to see the progress on this server, since as mentioned the way references work isn't performant enough in go guru and our code is based off that. Our project in general is just a hodge podge of different bits from the golang tools ecosystem, so this project which is tackling everything with a more holistic approach is exciting to me.

[szuecs]

Thanks @keegancsmith for pointing it out. I will not have the time to implement it, but thanks that you accept PRs. :)

[AlekSi]

See also: https://github.com/golang/tools/tree/master/internal/lsp

[dominikh]

Google is working on their own language server. Unless they mess it up big time, there's no point in us working on our own language server. Closing this issue.

[dgryski]

Don't forget delete the code you wrote in support of this.

[dominikh]

Branches are a wonderful thing. No trace left behind.