Cockroach: cli/sql: SIGSEGV when the TERM env var is not set

[HeikoOnnebrink]

Version:
Server version: CockroachDB CCL v2.0.5 (x86_64-unknown-linux-gnu, built 2018/08/13 17:59:42, go1.10) (same version as client)
Cluster ID: b29eb50b-b671-494f-a8da-c662de51958b
Cockroach running inside Docker container cockroachdb/cockroach:v2.0.5
OS Container Linux by CoreOS 1520.8.0 (Ladybug)
Docker version 1.12.6, build d9ad3fc

Problem:
just upgraded cluster from 1.1.6 to 2.0.5
cluster got migrated fine and all looks good, but when connecting to cluster with built-in client I get SIGSEGV
Connections from other client apps seem to work fine
Problem reproduces on all nodes of the cluster

Steps to reproduce
exec into running docker container 2.0.5 and launch sql client

core@api-prd-2 ~ $ docker exec -it cockroach bash
root@api-prd-2:/cockroach# ./cockroach sql --certs-dir=/cockroach

Error

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b16407]

runtime stack:
runtime.throw(0x24936f8, 0x2a)
    /usr/local/go/src/runtime/panic.go:619 +0x81
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:372 +0x28e

goroutine 1 [syscall]:
non-Go function
    pc=0x1b16407
non-Go function
    pc=0x1b165d1
non-Go function
    pc=0x1b15b94
non-Go function
    pc=0x73045f
runtime.cgocall(0x1b15b50, 0xc4208ef820, 0x248eaec)
    /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc4208ef7d8 sp=0xc4208ef7a0 pc=0x6d8004
github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix._C2func_go_libedit_init(0x0, 0x7f7f10c2e000, 0xc42000e010, 0x7f7f10c0e000, 0x7f7f10c0e280, 0x7f7f10c0e500, 0x0, 0x0, 0x0, 0x0)
    _cgo_gotypes.go:200 +0x5c fp=0xc4208ef820 sp=0xc4208ef7d8 pc=0x1aab89c
github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix.InitFiles.func3(0x0, 0x7f7f10c2e000, 0xc42000e010, 0x7f7f10c0e000, 0x7f7f10c0e280, 0x7f7f10c0e500, 0x0, 0xc4208ef930, 0x16b5c7b, 0x27602e0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:93 +0x19e fp=0xc4208ef880 sp=0xc4208ef820 pc=0x1aaea4e
github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix.InitFiles(0x24497ad, 0x9, 0x1, 0xc4200bc000, 0xc4200bc008, 0xc4200bc038, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:93 +0x482 fp=0xc4208ef9c8 sp=0xc4208ef880 pc=0x1aacc52
github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit.InitFiles(0x24497ad, 0x9, 0x700f01, 0xc4200bc000, 0xc4200bc008, 0xc4200bc038, 0xc420000180, 0x2576d60, 0xc420334080)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/editline_unix.go:15 +0x63 fp=0xc4208efa20 sp=0xc4208ef9c8 pc=0x1ab05c3
github.com/cockroachdb/cockroach/pkg/cli.runInteractive(0xc4200f1980, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1062 +0x41a fp=0xc4208efbf8 sp=0xc4208efa20 pc=0x1ad966a
github.com/cockroachdb/cockroach/pkg/cli.runTerm(0x3670da0, 0xc42049ea70, 0x0, 0x1, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1179 +0x159 fp=0xc4208efc50 sp=0xc4208efbf8 pc=0x1ad9e79
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0x3670da0, 0xc42049ea70, 0x0, 0x1, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:40 +0x5a fp=0xc4208efcc8 sp=0xc4208efc50 pc=0x1af036a
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute(0x3670da0, 0xc42049ea40, 0x1, 0x1, 0x3670da0, 0xc42049ea40)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:698 +0x46d fp=0xc4208efd70 sp=0xc4208efcc8 pc=0x1a7cfbd
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x366d440, 0x6, 0x0, 0xc420703ee8)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:783 +0x2e4 fp=0xc4208efea0 sp=0xc4208efd70 pc=0x1a7d734
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute(0x366d440, 0x18, 0xc420703f00)
    /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:736 +0x2b fp=0xc4208efed0 sp=0xc4208efea0 pc=0x1a7d42b
github.com/cockroachdb/cockroach/pkg/cli.Run(0xc4200a6160, 0x2, 0x2, 0xc4200b8010, 0xc42023e000)
    /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:156 +0x6d fp=0xc4208efef8 sp=0xc4208efed0 pc=0x1ab802d
github.com/cockroachdb/cockroach/pkg/cli.Main()
    /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:51 +0x15d fp=0xc4208eff78 sp=0xc4208efef8 pc=0x1ab7bbd
main.main()
    /go/src/github.com/cockroachdb/cockroach/main.go:27 +0x20 fp=0xc4208eff88 sp=0xc4208eff78 pc=0x1b0c770
runtime.main()
    /usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc4208effe0 sp=0xc4208eff88 pc=0x703412
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4208effe8 sp=0xc4208effe0 pc=0x731791

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 22 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1161 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x110

goroutine 23 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:601 +0x105
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:593 +0x128

goroutine 6 [select, locked to thread]:
runtime.gopark(0x2577f98, 0x0, 0x244476f, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420477750, 0xc4204bc0c0)
    /usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

[knz]

Thank you for this report. This is an oversight on our side, but rather easy for you to work around: please set the TERM environment variable inside the container when you are running the cockroach sql command. Set it to the value corresponding to the terminal you are using.

If that doesn't work, please let us known. I'll be fixing the bug in the meantime.

[knz]

(For colleagues looking to know how this can be investigated:)

I saw the SIGSEGV was in C code, where the stack trace didn't contain information.

However the panic message included the PC of the instruction containing the segfault: pc=0x1b16407

The code is generated potentially at different address in every build, so I needed the exact same binary as was ran here, so I went and downloaded a fresh copy of the linux 2.0.5 binary from the web site.

I ran objdump -d cockroach to disassemble and searched for the offending address. I found this:

0000000001b163e0 <go_libedit_term_supports_bracketed_paste@@Base>:
 1b163e0:       48 8d 3d a5 74 cf 00    lea    0xcf74a5(%rip),%rdi        # 280d88c <_cgo_yield@@Base+0x3a4>
 1b163e7:       48 83 ec 08             sub    $0x8,%rsp
 1b163eb:       e8 b0 25 ae fe          callq  5f89a0 <getenv@plt>
 1b163f0:       48 8d 3d a4 de cf 00    lea    0xcfdea4(%rip),%rdi        # 281429b <term_bp_off@@Base+0x6793>
 1b163f7:       48 89 c6                mov    %rax,%rsi
 1b163fa:       b9 06 00 00 00          mov    $0x6,%ecx
 1b163ff:       48 89 c2                mov    %rax,%rdx
 1b16402:       b8 01 00 00 00          mov    $0x1,%eax
 1b16407:       f3 a6                   repz cmpsb %es:(%rdi),%ds:(%rsi)

And indeed looking at the source code of that function:

int go_libedit_term_supports_bracketed_paste(EditLine* el) {
    char* term = getenv("TERM");
    return strncmp(term, "screen", 6) == 0 ...

getenv returns NULL if the env var is not set, and strncmp only works on non-NULL pointers.

From then both the fix and the workaround were clear.

[HeikoOnnebrink]

workaround is fine ..

root@api-prd-1:/cockroach# export TERM=xterm-256color
root@api-prd-1:/cockroach# ./cockroach sql --certs-dir=/cockroach
# Welcome to the cockroach SQL interface.
# All statements must be terminated by a semicolon.
# To exit: CTRL + D.
#
# Server version: CockroachDB CCL v2.0.5 (x86_64-unknown-linux-gnu, built 2018/08/13 17:59:42, go1.10) (same version as client)
# Cluster ID: b29eb50b-b671-494f-a8da-c662de51958b
#

thanks for ultra fast feedback 👍

[knz]

Fixed in #28614