Go: cmd/compile: out of memory with large slice and map composite literals

[h12w]

What version of Go are you using (go version)?

go version go1.9 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.B05GgQUqzW/go"
GORACE=""
GOROOT="/usr/local/go1.9"
GOTOOLDIR="/usr/local/go1.9/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build029171386=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

go get h12.me/ua
go build -v h12.me/ua

What did you expect to see?

go build finishes as go1.8.3.

What did you see instead?

h12.me/ua
# h12.me/ua
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xb62665, 0x16)
    /usr/local/go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc45a290000, 0x2f0000, 0xc420060100, 0xf05378)
    /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0xee3f40, 0x2f0000, 0xc42006fe20)
    /usr/local/go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0xee3f40, 0x172, 0x0)
    /usr/local/go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0xee3f40, 0x172, 0xf05388, 0x7fe3c6a9fa18)
    /usr/local/go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0xee3f40, 0x172, 0xffffffffffff0100, 0xc420060180)
    /usr/local/go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:733 +0x4d
runtime.systemstack(0xc42006ff08)
    /usr/local/go/src/runtime/asm_amd64.s:360 +0xab
runtime.(*mheap).alloc(0xee3f40, 0x172, 0x7fe3c6010100, 0x7fe3c6a9fa18)
    /usr/local/go/src/runtime/mheap.go:732 +0xa1
runtime.largeAlloc(0x2e2ba0, 0x7fe3c8220001, 0x7fe3c6a9fa18)
    /usr/local/go/src/runtime/malloc.go:827 +0x98
runtime.mallocgc.func1()
    /usr/local/go/src/runtime/malloc.go:722 +0x46
runtime.systemstack(0xc420020600)
    /usr/local/go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1125

goroutine 1 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc43ea636e8 sp=0xc43ea636e0 pc=0x455170
runtime.mallocgc(0x2e2ba0, 0xb2bf20, 0x17101, 0xc43c652228)
    /usr/local/go/src/runtime/malloc.go:721 +0x7b8 fp=0xc43ea63790 sp=0xc43ea636e8 pc=0x410638
runtime.makeslice(0xb2bf20, 0x1715d, 0x1715d, 0x1715d, 0x0, 0x80)
    /usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc43ea637c0 sp=0xc43ea63790 pc=0x43f157
cmd/compile/internal/ssa.makeLCArange(0xc43c652140, 0x0)
    /usr/local/go/src/cmd/compile/internal/ssa/lca.go:36 +0x8d fp=0xc43ea63920 sp=0xc43ea637c0 pc=0x5e783d
cmd/compile/internal/ssa.tighten(0xc43c652140)
    /usr/local/go/src/cmd/compile/internal/ssa/tighten.go:52 +0x2a6 fp=0xc43ea63a98 sp=0xc43ea63920 pc=0x8ff876
cmd/compile/internal/ssa.Compile(0xc43c652140)
    /usr/local/go/src/cmd/compile/internal/ssa/compile.go:70 +0x2bb fp=0xc43ea673d0 sp=0xc43ea63a98 pc=0x5cb19b
cmd/compile/internal/gc.buildssa(0xc43cae8b00, 0x0, 0x0)
    /usr/local/go/src/cmd/compile/internal/gc/ssa.go:216 +0xd89 fp=0xc43ea67530 sp=0xc43ea673d0 pc=0x9f6b29
cmd/compile/internal/gc.compileSSA(0xc43cae8b00, 0x0)
    /usr/local/go/src/cmd/compile/internal/gc/pgen.go:240 +0x3c fp=0xc43ea675a8 sp=0xc43ea67530 pc=0x9c363c
cmd/compile/internal/gc.compile(0xc43cae8b00)
    /usr/local/go/src/cmd/compile/internal/gc/pgen.go:219 +0x218 fp=0xc43ea67608 sp=0xc43ea675a8 pc=0x9c3568
cmd/compile/internal/gc.funccompile(0xc43cae8b00)
    /usr/local/go/src/cmd/compile/internal/gc/dcl.go:1049 +0xb7 fp=0xc43ea67660 sp=0xc43ea67608 pc=0x96a007
cmd/compile/internal/gc.fninit(0xc4203abe00, 0x28, 0x40)
    /usr/local/go/src/cmd/compile/internal/gc/init.go:207 +0xb59 fp=0xc43ea67880 sp=0xc43ea67660 pc=0x98fe39
cmd/compile/internal/gc.Main(0xb73f90)
    /usr/local/go/src/cmd/compile/internal/gc/main.go:592 +0x2c78 fp=0xc43ea67f08 sp=0xc43ea67880 pc=0x99e578
main.main()
    /usr/local/go/src/cmd/compile/main.go:49 +0x95 fp=0xc43ea67f80 sp=0xc43ea67f08 pc=0xabdfb5
runtime.main()
    /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc43ea67fe0 sp=0xc43ea67f80 pc=0x42bedd
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc43ea67fe8 sp=0xc43ea67fe0 pc=0x457cf1

[mvdan]

Both 1.9 and tip indeed seem to run out of memory. I have 7GB of free memory and it takes the build around 10s to use it all up, so it looks like it's stuck in an endless loop somewhere.

Will try to investigate and get a smaller reproduction program.

[mvdan]

Nope - removing the 23k elements from the handsets map fixes it on my machine. Removing all elements of deviceInfoMap also fixes it. But leaving both eats up my memory.

I think I saw issues about huge composite literals (especially really long slices and maps) using too much CPU time or memory. I wonder what could have caused the regression from 1.8. A bit beyond me, I'm afraid.

CC @mdempsky @randall77

[gopherbot]

Change https://golang.org/cl/66050 mentions this issue: cmd/compile: improve static map initialization

[randall77]

CL 66050 reduces memory use from ~15GB to ~0.5GB.

[ianlancetaylor]

That is only 96% smaller. Why stop there?

[mdempsky]

@ianlancetaylor Gotta leave some work for 1.11.

[randall77]

This is now fixed on tip.
What I don't understand is why 1.8 is better than 1.9.
1.8 uses mapassign and 1.9 uses mapassign_faststr, but I don't see how that would make much of a difference. The generated assembly is otherwise very similar.

Since this is a regression from 1.8 to 1.9, there's a possibility we could backport this fix (or another fix more directly related to the 1.8->1.9 regression, if we can understand it) to 1.9.1. But I'm leaning against doing this, it is a difficult bug to trigger (probably requires generated code) and can be worked around (doing the optimization in CL66050 manually).