Go: x/tools/gopls: high memory consumption when used with a monorepo

[trapgate]

Please answer these questions before submitting your issue. Thanks!

What did you do?

Our code is in a fairly large monorepo containing about 26K files, including vendored packages.
I use VSCode, with gopls enabled, and launched from the root of our monorepo, so the workspace includes the whole repo.

What did you expect to see?

gopls using 1-2GB of memory. That's a wild guess that I can't really validate, but 8-10 seems excessive.

What did you see instead?

The gopls process is using 8-10GB of memory. This number goes up and down with editor usage, but it never goes below about 6GB, and it generally hovers around 8GB. This is a 16GB laptop, so there's quite a bit of memory pressure when gopls is running.

Nearly all the code in our monorepo is intended for linux, and this is a mac, so I've experimented with adding this setting:

    "gopls": {
        "env": {"GOOS": "linux"}
    },

This does affect the number of errors reported by gopls, but the memory consumption is approximately unchanged.

I also tried building the latest master of gopls using the patch mentioned in this bug: https://github.com/golang/go/issues/37223. This didn't help either. I'm attaching an SVG heap profile from that version.
pprof.zip

Build info

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/[email protected] h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=
    github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/[email protected] h1:OX66ZzpltgCOuBSGdaeT77hS2z3ub2AB+EuGxvGRBLE=
    golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/[email protected] h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
    mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Go info

go version go1.14 darwin/amd64

GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/geoffhickey/Library/Caches/go-build"
GOENV="/Users/geoffhickey/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*.internal.digitalocean.com,github.com/digitalocean"
GONOSUMDB="*.internal.digitalocean.com,github.com/digitalocean"
GOOS="darwin"
GOPATH="/Users/geoffhickey/do/cthulhu/docode"
GOPRIVATE="*.internal.digitalocean.com,github.com/digitalocean"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v7/w5lfw5qs1bd9np31z9vv04h80000gn/T/go-build021923438=/tmp/go-build -gno-record-gcc-switches -fno-common"

[gopherbot]

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

[heschik]

There should be some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

[trapgate]

gopls-heap.zip

[heschik]

Thanks.

Unfortunately I don't have good news for you. Everything you've posted points to expected behavior -- I don't see any signs of a memory leak or work pileup. It looks like the monorepo is simply too large to work with all at once.

If you're only interested in one folder, you can start VS Code in that folder. If you're interested in multiple, you can add them piecemeal to your workspace, but note that most gopls features, e.g. find references, will only work within one folder.

[Jacalz]

I am running in to the same issue described here and it is really annoying because my Linux machine with 8GB memory always runs out of memory and I have to pull out the power plug to be able to actually restart the computer.

This often occurs when i am using vscode and gopls with the https://github.com/fyne-io/fyne repository. I usually see memory slowly but steadily rising up to around five or six gigabytes until the memory in my computer is gone and everything stops functioning as expected. Killing of gopls just makes it rise steadily again.

[stamblerre]

@Jacalz: That sounds like a bug with older versions of gopls - can you confirm that you are using v0.4.0 (gopls version)?

It's also possible that you have multiple versions of gopls installed and VS Code is using an old one - you can confirm the path to the gopls binary that VS Code is using the Go: Locate Configured Go Tools command. Once we confirm you are using the correct gopls version, we can continue investigating the memory usage.

[Jacalz]

@stamblerre: I verified that vscode is using the correct installation and this is the path of it: gopls: /home/jacob/go/bin/gopls installed. Looks good to me. The version should be v0.4.0 as long as vscode doesn't update to the wrong release. I tried running Go: Install/Update tools and I still have enormous amounts of memory usage.

[stamblerre]

You can confirm the version by running /home/jacob/go/bin/gopls version.

If the memory usage is high with gopls/v0.4.0, you should have some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

[Jacalz]

Thanks. It indeed does appear like I am using the latest version:

jacob@pacman ~/go/src/sparta $ ~/go/bin/gopls version
golang.org/x/tools/gopls 0.4.0
    golang.org/x/tools/[email protected] h1:G4+YP9kaV4dJb79J5MobyApxX493Qa6VoiTceUmxqik=

I tried to find the PID files, but I could not find any. I looked in /tmp, but nothing with the name gopls stood out at all. Am I doing something wrong?

[stamblerre]

What OS are you on? Take a look at the https://pkg.go.dev/os?tab=doc#TempDir docs to see what directory gopls will write temporary files to.

[Jacalz]

I am on Linux. I looks like I just wasn't getting into regions of memory usage that was concidered too much. I did not open the huge monorepo (https://github.com/fyne-io/fyne) and only saw 800MB memory usage due to that. Below are the files that you requested:

gopls.3169-5GiB-goroutines.txt
gopls.3169-5GiB-heap.pb.gz

I must admit that they were quite hard to get. When I tried for the first time, the memory usage jumped to 4GB and then gopls hugged 82% of my cpu and the computer totally froze and that meant that a force restart pruged the temporary files. This was by opening two files from https://github.com/fyne-io/fyne and doing small edits and a two saving of those files.

I restarted, had the system monitor open and ready to kill gopls when I had the chance. My system only has 8GB memory so when gopls starts using over 6GB (I saw a total of 6.7 GB before it was starting to get laggy and I killed it), things get really out of control and in to unplug power adapter territory. Thus I could only get the PID files for 5GB heaps at most. The PID files for 1, 2, 3, 4 GB heaps are saved too in case you need those too.

[heschik]

I was able to reproduce a similar profile by simply opening the root of the Fyne project. Most of the memory is being used by the type checker:

Showing nodes accounting for 853.95MB, 88.04% of 969.98MB total
      flat  flat%   sum%        cum   cum%
  620.02MB 63.92% 63.92%   620.02MB 63.92%  go/types.(*Checker).recordTypeAndValue
   87.50MB  9.02% 72.94%    98.44MB 10.15%  go/parser.(*parser).parseOperand
   43.13MB  4.45% 77.39%   153.17MB 15.79%  go/parser.(*parser).parseElementList

That's a little surprising for Fyne which is a relatively small project. My best guess is that the problem is the GL libraries, which generate tens of thousands of lines of cgo.

Regardless, I don't see a gopls bug here; it appears to be general go/types memory usage as usual.

[huapox]

+1 with version gopls 0.4.0

[huapox]

get worse with gopls 0.4.1

sam @ debian in /_ext/gopath/bin |03:24:50  
$ ./gopls version
golang.org/x/tools/gopls 0.4.1
    golang.org/x/tools/[email protected] h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=
sam @ debian in /_ext/gopath/bin |03:32:22  
$ uname -a
Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux

sam @ debian in /_ext/gopath/bin |03:43:33  
$ go version
go version go1.13.7 linux/amd64

[ggilley]

I have 6 projects open in vscode. A gopls process for each is running ranging from 465MB to 900MB. Crazy!

[hunterlxt]

gopls eat 22G memory within just one project. Do you have any experience with large project

using VSCode with the default config

[egorenar]

syzkaller is another Go project which causes gopls on my machine to consume 27GiB RAM.

[Luffy110]

i have the same issue .
my gopls version is
golang.org/x/tools/gopls 0.4.1
golang.org/x/tools/[email protected] h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=

uname -a

Linux shenjie-VirtualBox 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

go version :
go version go1.13.4 linux/amd64

[tommady]

golang.org/x/tools/gopls 0.4.1
go version go1.14.2 darwin/amd64
34G too crazy 🤪

[agnivade]

Hi everyone, just a quick reminder that just mentioning your Go version and gopls version is not sufficient information to help us debug this further.

If you notice issues with gopls consuming high memory,

• please include your VS Code settings
• the repo that is causing this (if it's open source)
• and a heap profile to go with it.

Thanks.

[heschik]

@trapgate, @Jacalz I believe some changes I landed today have improved memory usage on large projects substantially. Could you try master using the instructions at https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions?

For fyne specifically the problem is the theme directory, which contains massive byte slice literals that are very expensive to parse. Representing them as strings might make a huge difference.

[trapgate]

Thanks @heschik, I gave that a try with our monorepo, but it didn't make much difference. Running gopls version 0.4.1 the language server used 7.24GB; with the current tip of master it dropped to 7.07GB.

I've been working around this problem since I reported it by launching VSCode on a subdirectory of our monorepo. For my purposes this works well enough for now - gopls uses roughly 2GB running this way. It's still surprising, as there are only 2500 or so .go files in this subdirectory.

[heschik]

Somewhat disappointing.

There's a new debug section now. If you follow the troubleshooting instructions to start the debug server, then look at /cache/1, there are some cryptic per-package stats. Can you attach those, sanitized if you so desire?

[Jacalz]

Hi @heschik. Sorry for the late answer, I haven't had the time to look at it until now. I gave your changes a little spin with https://github.com/fyne-io/fyne and the difference is staggering. I will need to do more testing but initial results seem to indicate that memory usage caps out on around 4.2 GB instead of growing indefinitely until my computer gets starved of memory and crashes.

In short, it is finally usable even though memory usage still is absurdly high in my opinion. Will do some more testing before committing fully though :)

[Jacalz]

Alright, it still seems to like to steal all my memory, but it seems to take more time before that happens. A good improvement definitely, but I still seem to need to upgrade the amount of ram in my computer to 16 GB Ram...

[trapgate]

Hi @heschik, the output from the debug server's /cache/1 endpoint would include a complete map of the directories in our repo, which I can't comfortably reveal. Here are some of the vendored packages from the top of the list though. This is from a gopls process that's currently using 9.5 GiB. I can probably share more of the information from the cache if you let me know what you're looking for.

vendor/github.com/swaggo/files (1)  35.68 = 35.63 + 0.02 + 0.01 + 0.02
vendor/github.com/digitalocean/go-netbox/netbox/client/dcim (1) 10.83 = 1.46 + 4.83 + 1.44 + 3.10
vendor/k8s.io/api/core/v1 (1)   10.12 = 2.24 + 3.92 + 0.97 + 3.01

[heschik]

@trapgate At high heaps, very recent gopls will write zip files to /tmp. The ones with -noname have the package names removed.

[trapgate]

Gotcha, here's one:

gopls.449451-7GiB-nonames.zip

[RayfenWindspear]

Same thing here. Up to 9gb, but managed to trim it down to around 3.5gb. Pretty sure it was just less than valid paths and cleaning actual paths and cache files that brought it down.

Anyway, to the question at hand. What would be the viability of running a single instance of gopls against master of the repo and having everyone in the organization connect to it? There would be drift as work on your local branch progressed, but would it be able to handle, say, 50 people connecting to one instance remotely?

[heschik]

@RayfenWindspear Even if that were feasible I'm reasonably certain you'd just end up with a gopls instance using 50x the memory. There are bugs here, but massive server-side deployments are not going to be the fastest way to address them.

[RayfenWindspear]

@heschik While I appreciate the reply, and forgive my ignorance if this is totally bogus, but would it not be able to reconcile and consolidate the memory for the indexed files? Or would it treat each individual Session as having its own separate set of indexed files? I do understand that it's technically pre-release and is listed as unstable, so perhaps that isn't part of its feature set (yet?).

Basically, we have several devs who already use VSCode to run remote to our dev server, which will each therefore run their own separate instance. In theory, even consolidating that to a single instance would at least allow it to not have to reel in the go.mod deps (disk space) and then index that all (memory) per instance.

So considering that, it seems like it could possibly be feasible and incur less overhead than one instance per dev. Sure it would still be O(n*repo), but (in theory, if it works like this), that's better than O(n* (repo+deps+instance_overhead) ).

Thoughts?

[waynr]

So while tinkering around with go tool pprof and gopls on Friday evening, I was able to significantly reduce gopls memory usage in the same repo @trapgate is referring to in this issue by disabling the *debug.Instance when no -debug flag value is specified by the user. I estimate about a 15% reduction in memory consumption.

Another of my coworkers told me that emacs will prompt them for a working directory to pass to gopls which results in it consuming even less memory. Unfortunately I am a vim-go user and I'm not sure how to configure it to pass this kind of working directory hint to gopls. Is it something that can be configured by the editor plugin via Language Server Protocol?

I also experimented with moving the monorepo in question away from a monolithic module approach (currently our monorepo is comprised of several dozen projects all within the same go.mod) to a module-per-project approach and in combination with my abovementioned -debug "hack" saw my gopls memory consumption reduced by a total of roughly 70%.

[heschik]

alloc_space is total bytes allocated. Allocating memory isn't free, but if your goal is to reduce the overall heap size of gopls then you are looking at the wrong number.

Without looking carefully, I believe the primary result of your PR is to disable logging. My guess is that either you have verbose logging enabled, or something is going very very badly wrong in a way that results in log spam. Either way, please share the profile.

[waynr]

@heschik thanks for the quick response! If I understand what you are saying correctly alloc_space isn't short for "currently allocated space" but instead for "total allocated during the process's lifetime" and at the point in time when the memory profile is captured that space may have been deallocated.

Without looking carefully, I believe the primary result of your PR is to disable logging.

The debug.Instance struct is what is used to keep track of package statistics; when a user passes for example -debug=localhost:6060 they can then view package statistics gathered by what i assumed to be hooks on the main language server processing bits that are passed to debug.WithInstance. I didn't realize that it also contributes to logging.

[waynr]

I estimate about a 15% reduction in memory consumption.

After looking more carefully at the same profile in the PR I linked above, I see now what @heschik was talking about. Using go tool pprof -png -sample_index=inuse_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz instead of go tool pprof -png -sample_index=alloc_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz shows that the debug.Instance consumes an almost negligible amount of memory.

[stamblerre]

There have been a number of memory improvements in gopls at master. Please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions, or wait for the next release.

[Raghav-intrigue]

Great improvement guys. My memory usage dropped from 9gigs to almost 2gb.

[trapgate]

For me at least, gopls 0.4.4 has been much more parsimonious than previous versions - I'm seeing 1-2GB of memory usage on my monorepo, where before I was seeing 8-10GB. I haven't yet tried running with master; I will try to check that out in the next couple of days and report back.

[stamblerre]

Glad to hear that things are working better! There will be more improvements coming in gopls/v0.5.0, so I think I will go ahead and close this issue for now. If you encounter memory usage issues again, please take a look at the instructions in https://github.com/golang/go/issues/36943 and file a new issue.

[egorenar]

I also confirm that it got much better, it became usable with large projects, thank you very much.

[rmerry]

Using gopls at master hasn't fixed this issue for me.

I'm working on a monorepo (~40GB in size) and v0.4.4 would regularly cause my remote machine to become unresponsive by using all available memory. I'm now using master which has bought me a little more time but not fixed the issue.

$ gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/[email protected] h1:/32sBC1LOo43X5JHPUZT+hbLpyamXpL4FgR5eMEYb7w=

Gopls now takes quite a little while longer to eat up all my memory but if left unchecked it will do just that. I'm still having to kill it every 3-4 minutes. My solution now is to use a systemd watchdog to kill gopls when memory usage gets too high---not ideal but far better than losing connection to my remote box.

[Jacalz]

I am having the roughly the same issues as @rmerry, but on a different repository. Using gopls with https://github.com/fyne-io/fyne results in massive memory consumption that easily climbs up to over 8 gigabytes of memory consumption. It climbs a lot more slowly with the latest releases, but it is still largely unusable sadly :(

[heschik]

@rmerry @Jacalz

Please file new issues following the steps at https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#memory-usage. Simply reporting that gopls uses a lot of memory, or describing the workarounds you use to kill it, doesn't help us solve the problem. The memory diagnostics might.