Go: x/tools/gopls: using 1GB RAM with Mattermost server

[ygworldr]

What version of Go are you using (go version)?

$ go version
go version go1.13.9 linux/amd64
$ gopls version
golang.org/x/tools/gopls 0.4.0
    golang.org/x/tools/[email protected] h1:G4+YP9kaV4dJb79J5MobyApxX493Qa6VoiTceUmxqik=

Does this issue reproduce with the latest release?

Unknown.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE="on"

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/yann/.cache/go-build"

GOENV="/home/yann/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/yann/go"

GOPRIVATE=""

GOPROXY="direct"

GOROOT="/usr/lib/golang"

GOSUMDB="off"

GOTMPDIR=""

GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/home/yann/worldr/github/server/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build075671369=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running gopls as part of vim-go. This is is one go file open.

What did you expect to see?

A small increase in memory.

What did you see instead?

; procs gopls
 PID:▲  User │ State Nice TTY CPU  MEM   VmPeak   VmSize   VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                [%]  [%]  [bytes]  [bytes] [bytes]         [B/s] [B/s] │                            │
 212834 yann │ S        0     0.0 27.5   5.203G   5.203G  4.285G []  []  0.000 0.000 │ 00:01:58 2020/05/06 15:51  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-e4935e-daemon.yann -listen.timeout 1m0s -logfile auto
 215897 yann │ S        0     0.0  0.1 767.355M 767.355M 18.465M []  []  0.000 0.000 │ 00:00:00 2020/05/06 15:53  │ /home/yann/go/bin/gopls -remote=auto
 216038 yann │ S        0     0.0  0.1 831.230M 767.293M 18.000M []  []  0.000 0.000 │ 00:00:00 2020/05/06 15:53  │ gopls -remote=auto

I have to constantly close/open neovim since otherwise, my machine dies because gopls takes all the 16G of RAM I have.

[gopherbot]

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

[stamblerre]

If you are using gopls/v0.4.0, you should see heap profiles in your temporary directory. If you find them, please upload them.

Edit: Sorry, I see you are using v0.4.0. Is this on a public or private repository? If it's public, do you mind sharing a link to it?

[ygworldr]

The repository is the Mattermost server.

Here is a heap profile file (I hope!): gopls.149275-1GiB-heap.pb.gz

; procs gopls
 PID:▲  User │ State Nice TTY CPU MEM   VmPeak   VmSize   VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                [%] [%]  [bytes]  [bytes] [bytes]         [B/s] [B/s] │                            │
 149275 yann │ S        0     0.0 8.4   2.269G   2.269G  1.312G []  []  0.000 0.000 │ 00:00:12 2020/05/07 12:18  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-cbe194-daemon.yann -listen.timeout 1m0s -logfile auto
 150361 yann │ S        0     0.0 0.1 839.297M 839.297M 16.805M []  []  0.000 0.000 │ 00:00:00 2020/05/07 12:18  │ /home/yann/go/bin/gopls -remote=auto
 150503 yann │ S        0     0.0 0.1 695.227M 695.227M 15.723M []  []  0.000 0.000 │ 00:00:00 2020/05/07 12:18  │ gopls -remote=auto

[stamblerre]

Thank you for sharing this. It looks as we'd expect, with most of the memory allocated during type checking. By the way, it looks like this heap is only 1GiB, not 16 as you saw. What are your gopls settings - do you have staticcheck enabled by any chance?

Thank you for sharing the link to the repository - we'll try to reproduce. There are number of repos that seem to cause problems with gopls (https://github.com/golang/go/issues/38752 is a similar issue, for example).

[ygworldr]

@stamblerre Thank you so much for looking into it! Much appreciated.

Yes, it's not the one that generated it. I fiddled a little with dependencies and now I only have the version that vim-go specifically request: 0.4.0, without any additional tags. It seems to behave better but still gets to 4/5G. I will upload another large file when I get one.

do you have staticcheck enabled by any chance?

I do not think so, at least not explicitly. It might be turned on by default in vim-go.

[stamblerre]

Yes, it's not the one that generated it. I fiddled a little with dependencies and now I only have the version that vim-go specifically request: 0.4.0, without any additional tags. It seems to behave better but still gets to 4/5G. I will upload another large file when I get one.

Thank you!

I do not think so, at least not explicitly. It might be turned on by default in vim-go.

I don't believe vim-go turns on staticcheck by default, and based on the profile you attached that seems to be the case.

We'll follow up here when someone from the gopls team has a chance to take a closer look.

[ygworldr]

gopls.176223-3GiB-heap.pb.gz

Here's another heap, under gopls 0.4.1…

[heschik]

@ygworldr I believe some changes I landed today have improved memory usage on large projects substantially. Could you try master using the instructions at https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions?

[ygworldr]

@heschik I updated the version I have and opened two files in the Mattermost project: channel_test.go and channel.go in api4. This is what I got:

√ ; gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/[email protected] h1:+BNEvY32Dn2PtaE6qifJQQshGNIuJvcXAvYUDe6+/gc=
√ ; procs gopls
 PID:▲  User │ State Nice TTY   CPU MEM  VmPeak  VmSize    VmRSS TCP UDP  Read Write │ CPU Time Start             │ Command
             │                  [%] [%] [bytes] [bytes]  [bytes]         [B/s] [B/s] │                            │
 167625 yann │ S        0       0.0 0.1  1.242G  1.181G  16.816M []  []  0.000 0.000 │ 00:00:00 2020/06/12 13:17  │ /home/yann/go/bin/gopls -remote=auto
 167651 yann │ S        0     265.5 6.1  2.168G  2.168G 969.145M []  []  0.000 0.000 │ 00:00:08 2020/06/12 13:17  │ /home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-c6d4b6-daemon.yann -listen.timeout 1m0s -logfile auto

I am afraid there was no heap file in /tmp… Do I need to do something special to generate it?

[heschik]

That looks like pretty good success, no? Your most recent report was 1.3G, and now it's down to 970M. Not a bad improvement.

The heaps are only generated once gopls crosses 1G so you won't have any yet.

I'm inclined to close this -- I don't see any sign that there's a specific problem with mattermost, and we've made significant progress since it was filed. Do you agree?

[heschik]

...assuming it doesn't grow up to 4 or 5 G again.

[ygworldr]

@heschik

I'm inclined to close this -- I don't see any sign that there's a specific problem with mattermost, and we've made significant progress since it was filed. Do you agree?

Yes, I think the improvements are great! Thank you so much for that. Your hard work is much appreciated!

[ygworldr]

@heschik, I am afraid I have some bad news… But they are not more than 3G so, maybe not so bad?

• gopls.146849-1GiB-heap.pb.gz
• gopls.146849-2GiB-heap.pb.gz
• gopls.146849-3GiB-heap.pb.gz

[heschik]

That's disappointing. Some progress but we should be able to do better.

I added a new diagnostics section to the debug server that may be helpful. If you follow the troubleshooting guide to start it, /cache/1 will have a table of per-package memory usage. Could you attach it here from a gopls that's using a lot of memory?

[ygworldr]

That's disappointing. Some progress but we should be able to do better.

No worries whatsoever. I am really grateful for the work you are doing.

I added a new diagnostics section to the debug server that may be helpful. If you follow the troubleshooting guide to start it, /cache/1 will have a table of per-package memory usage. Could you attach it here from a gopls that's using a lot of memory?

I shall do as soon as is feasible.

[ygworldr]

• gopls.195510-1GiB-heap.pb.gz
• gopls.195510-1GiB-goroutines.txt

Sadly, the log file is empty. The server command looks like:
/home/yann/go/bin/gopls serve -listen unix;/tmp/gopls-c6d4b6-daemon.yann -listen.timeout 1m0s -logfile auto and /home/yann/go/bin/gopls -remote=auto -rpc.trace…

What am I doing wrong here?

[stamblerre]

There have been a number of memory improvements in gopls at master. If you can, please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions and let me know if that's any better.

[ygworldr]

@stamblerre I have updated just been doing lots of DevOps stuff and not much coding. This will change soon and I shall update. Thank you for your hard work!

[stamblerre]

Closing, as this has been in the WaitingForInfo state for over a month. gopls/v0.5.0 has been released, and it contains these memory improvements.