Go: runtime: programs compiled by 1.11 allocate an unreasonable amount of virtual memory

[HouzuoGuo]

What version of Go are you using (go version)?

go1.11.1 linux/amd6

Does this issue reproduce with the latest release?

It was first discovered under go 1.11, and the issue remains in the minor upgrade to 1.11.1.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/howard/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/howard/gopath"
GOPROXY=""
GORACE=""
GOROOT="/home/howard/go"
GOTMPDIR=""
GOTOOLDIR="/home/howard/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build718493987=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compile the following trivial program:

package main

func main() {
        for {
        }
}

go build -o usagetest; ./usagetest and observe its memory usage as monitored by host operating system, for example via ps aux | grep usagetest.

What did you expect to see?

Compiled by 1.10.4, the program uses a very reasonable amount of virtual (2MB) and resident (600KB) memory:

howard 10838 101 0.0 2384 688 pts/2 Rl+ 05:57 0:59 ./usagetest

But when compiled by 1.11 (or 1.11.1), the virtual memory usage dramatically increases to 102MB:

howard 11326 99.5 0.1 101820 4776 pts/2 Rl+ 05:59 1:58 ./usagetest

The significant increase in virtual memory usage is usually not an issue, however security sensitive programs often lock their memory, causing a far greater performance degradation on low-spec computer hosts.

[randall77]

Related: #28081 (large core files)

[ianlancetaylor]

CC @aclements

[ianlancetaylor]

Presumably related to https://golang.org/cl/85888 and #10460.

[aclements]

Interesting. I had thought that mlocking memory would only lock pages from being paged out, but wouldn't eagerly fault the pages in. But looking more closely at the manpage, it looks like that's only true with mlock2 with the MLOCK_ONFAULT flag or mlockall with the MCL_ONFAULT flag. I assume you're using mlockall? MCL_ONFAULT has only been available since Linux 4.4, but would that be a reasonable workaround, at least for now?

The solution I proposed in #28081 should also fix this.

[networkimprov]

More discussion: https://news.ycombinator.com/item?id=18184541

[HouzuoGuo]

Thanks for the hint @aclements.

Indeed, couple of my system programs are using mlockall, often with both syscall.MCL_CURRENT | syscall.MCL_FUTURE. For this issue report, MCL_CURRENT alone is sufficient:

package main

import "syscall"

func main() {
        if err := syscall.Mlockall(syscall.MCL_CURRENT); err != nil {
                panic(err)
        }
        for {
        }
}

As observed by host OS, it appears to have indeed copied all of the allocated virtual memory into main memory:

root 29281 156 2.5 101836 101736 pts/0 RLl+ 05:27 0:01 ./usagetest

Hehe also I was wondering why this issue attracted so many emojis, it turns out someone put it on hackernews..

[kumarharsh]

Yeah, I had just upgraded to 1.11 while not noticing anything significant in the release notes (at that time). And just after that, I opened hacker news, and this was the top result. Imagine the panic! (pun intended)

[aclements]

@HouzuoGuo, yes, that's what I would expect. Thanks for confirming.

I don't really understand when it makes sense to use mlockall instead of mlock. There's a ton of stuff in memory that certainly doesn't have to be locked, and using mlock for just the things that need to be locked wouldn't have the problem with virtual memory. What's the reason for using mlockall instead of mlock? And if you do need mlockall, is there a reason not to use MCL_ONFAULT?

Even if we do modify arena mapping to be incremental (which I believe should be relatively easy to do), I don't want to change how the arena index is mapped because that would impact both performance and simplicity. So mlockall would still fault in about 32MB, even with the change to arena mapping.

[HouzuoGuo]

Thanks @aclements , TIL about MCL_ONFAULT, apparently available since Linux 4.4. I shall use it from now and onward.

Beyond the scope of this issue report, would you please offer some hints on the proper invocation of mlock in protecting a sensitive instance of structure, especially to determine its memory address range? Take this structure for example

type Daemon struct {
    ListenAddress string
    ListenPort int
    InternalData map[string]interface{}
}

[aclements]

Folding this issue into #28081, which I've re-titled, since the root cause is the same.