Nomad: 0.8.2 Panic - 3x since upgrading 12 hours ago

[djenriquez]

Nomad version

Output from nomad version

Operating system and Environment details

Amazon Linux 2017.09
3 Server deployment

Issue

Upgraded servers to 0.8.2 and have three times in the past 12 hours found the cluster servers down. Found this panic:

    2018/04/27 12:53:07.549286 [ERR] http: Request /v1/namespaces, error: Nomad Enterprise only endpoint
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfd3586]

goroutine 25399 [running]:
github.com/hashicorp/nomad/nomad.(*ClientAllocations).Stats(0xc42000e5d8, 0xc421253500, 0xc4208b9ca0, 0x0, 0x0)
    /opt/gopath/src/github.com/hashicorp/nomad/nomad/client_alloc_endpoint.go:174 +0x436
reflect.Value.call(0xc420055020, 0xc420be3ac8, 0x13, 0x159a46f, 0x4, 0xc421a99d48, 0x3, 0x3, 0x13f32e0, 0x19, ...)
    /usr/local/go/src/reflect/value.go:434 +0x905
reflect.Value.Call(0xc420055020, 0xc420be3ac8, 0x13, 0xc421a99d48, 0x3, 0x3, 0x16, 0x16, 0x16)
    /usr/local/go/src/reflect/value.go:302 +0xa4
net/rpc.(*service).call(0xc420c48100, 0xc42101e3c0, 0xc4214ec158, 0xc4204e9b80, 0xc420b36840, 0x149eac0, 0xc421253500, 0x16, 0x13e2500, 0xc4208b9ca0, ...)
    /usr/local/go/src/net/rpc/server.go:381 +0x142
net/rpc.(*Server).ServeRequest(0xc42101e3c0, 0x200fac0, 0xc4210ae340, 0x0, 0x0)
    /usr/local/go/src/net/rpc/server.go:496 +0x22c
github.com/hashicorp/nomad/nomad.(*Server).handleNomadConn(0xc420096280, 0x200f200, 0xc420ec4480, 0x2017ea0, 0xc42153a240, 0xc42101e3c0)
    /opt/gopath/src/github.com/hashicorp/nomad/nomad/rpc.go:236 +0x239
created by github.com/hashicorp/nomad/nomad.(*Server).handleMultiplex
    /opt/gopath/src/github.com/hashicorp/nomad/nomad/rpc.go:218 +0x2bf

The error to /v1/namespaces may be a red herring, it was just the last log line.

Restarting the servers seem to bring everything back up just fine, not sure whats causing this to die though.

I should also mention we have 3 other Nomad clusters on 0.8.2, 2 have a 5 server deployment, 1 is a 3 server deployment. Those have been completely stable since the upgrade.

[djenriquez]

Maybe state is null here? https://github.com/hashicorp/nomad/blob/master/nomad/client_alloc_endpoint.go#L174

[chelseakomlo]

Thanks for the report- we are looking into this. Are you running Nomad Web UI as well?

[djenriquez]

Hi @chelseakomlo, yes we are.

[dadgar]

@djenriquez Nomad 0.8.3 has been released fixing this! Thanks for reporting!