Go: x/tools/gopls: reduce memory usage

[stamblerre]

This is a general tracking issue for memory usage in gopls. We need to collect data and set goals for usage for a given project size. We also need to work towards reducing the memory usage when staticcheck is enabled (though that may be a separate issue).

If you have a specific problem with gopls using too much memory on your machine, please create a new issue so that we can investigate it separately.

Please include your editor settings and memory debug information with your issue. These are automatically written to the temporary directory (/tmp for Unixes) with names like gopls.1234-5GiB-withnames.zip. If you are uncomfortable sharing the package names of your code, you can share the -nonames zip instead.

[nghialm269]

I just upgrade gopls to v0.3.0 today so that I'm able to rename symbols / find reference across package, but the memory usage is very high:

Edit: It's a small private project, the only dependency is beego

I'm using Ubuntu 16.04

gopls version:

golang.org/x/tools/gopls v0.3.0
golang.org/x/tools/[email protected] h1:l9KKK1/n6CIbfgaUvHBWAvCfOxcl1N+KSOK79OlPIao=

go version:

go1.13.4 linux/amd64

[patrikeh]

As the commenter above, I've have had problems with massive memory usage since upgrading to 0.3.0 yesterday, sometimes upward of 12 GB.

macOS Mojave 10.14.6
go version go1.13 darwin/amd64
golang.org/x/tools/[email protected] h1:l9KKK1/n6CIbfgaUvHBWAvCfOxcl1N+KSOK79OlPIao=

[guiguan]

same here :(

[myitcv]

@guiguan there is absolutely no need for language like that. Please, keep the feedback constructive.

FWIW this has already been fixed on master via https://go-review.googlesource.com/c/tools/+/217677

[stamblerre]

I'm sorry that you all encountered this bug - we had a memory leak that was a bit tricky to reproduce, but as @myitcv said, we believe that we've fixed it. You can try it out by downloading the pre-release for the next version of gopls (GO111MODULE=on go get golang.org/x/tools/[email protected]). Please report here if you still see the excessive memory usage with that version.

[vincepri]

@stamblerre On the latest master, gopls is using more than 30GB when developing with some kubernetes sub-projects

[heschik]

@vincepri per the bold section in the original issue, please file a new issue with repro instructions, even if they're just "check this repository out and open it."

[trajber]

@stamblerre @heschik

I'm using the latest master and the memory usage is relatively large for a small/medium project. (~1.5GB)
Unfortunately I cannot share the project, but I can send you the pprof heap file for gopls, if it helps :man_shrugging:

Build info

golang.org/x/tools/gopls master golang.org/x/tools/[email protected] h1:yNTWrf4gc4Or0UecjOas5pzOa3BL0WDDyKDV4Wz5VaM= github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ= golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E= golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU= golang.org/x/[email protected] h1:BtAc3ytYUhbzaL+TpDGZSBVecmz6ra8/VYiIwtasXXc= golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA= honnef.co/go/[email protected] h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Here you can find the pprof heap file

Let me know if you need any more info from my side that can be useful for you.

[heschik]

@trajber Unfortunately I don't see any signs of a bug in your profile. That doesn't mean there isn't one, but 1.5GiB is not completely out of line for a project with a lot of dependencies, and without access to the source it's very difficult to judge. Sorry, but I don't think there's going to be much we can do in the short term. Hopefully you will benefit from general optimizations we make in the future.

[Jacalz]

When not using a huge monorepo and slowly running out of memory, I usually see gopls hanging out at around 800MB to 1GB of memory when browsing and editing in smaller repositories.

[stamblerre]

@Jacalz: Responded to you on https://github.com/golang/go/issues/37670#issuecomment-614732451, so let's continue the conversation on that issue.

[gopherbot]

Change https://golang.org/cl/236397 mentions this issue: internal/lsp/cache: intermediate test variants aren't workspace packages

[heschik]

I think the commits I pushed today have improved things significantly for large projects. I'll keep looking for a little bit.

[itzmanish]

The memory uses is not as much users commented before. Not even close but yet giving half a GB is big. However it provides better and fast autocomplete.

go version go1.14.4 linux/amd64

gopls latest

[heschik]

From the first comment:

If you have a specific problem with gopls using too much memory on your machine, please create a new issue so that we can investigate it separately.

There will be zip files in /tmp with additional memory diagnostic information. Please include one of them. If you are uncomfortable sharing the package names of your code you can pick one with a -nonames suffix, otherwise please upload a --withnames` file.

[stamblerre]

There have been a number of memory improvements in gopls at master. Please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions, or wait for the next release.

[raghav-clumio]

@stamblerre just switched to master and it dropped my ram usage from 10 gigs to 1.5gb! Great work guys!

[gopherbot]

Change https://golang.org/cl/253577 mentions this issue: gopls/doc: add memory usage info to troubleshooting guide