Nomad: Unable to start both the server and client on the same server

[akshatgit]

Nomad version

Nomad v0.10.2 (0d2d6e3dc5a171c21f8f31fa117c8a765eb4fc02)

Operating system and Environment details

PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Issue

Unable to start both the server and client on the same server. Consul is running on the host.

Reproduction steps

start nomad service

Config.hcl file

data_dir = "/etc/nomad.d"
datacenter = "xxx"

server {
enabled = true
bootstrap_expect = 3
}
client {
enabled = true
}

Nomad Error logs

==> Loaded configuration from /etc/nomad.d/client.hcl
==> Starting Nomad agent...
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host.PlatformInformationWithContext(0x27c9da0, 0xc000042098, 0x0, 0x0, 0x8f889027, 0x48b48f, 0x212e440, 0x212e478, 0xc000563250, 0x406f36)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host/host_linux.go:318 +0x199e
github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host.PlatformInformation(...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host/host_linux.go:280
github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host.InfoWithContext(0x27c9da0, 0xc000042098, 0x1f02ee0, 0xc0002312d8, 0xc000563318)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host/host_linux.go:48 +0x97
github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host.Info(...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/shirou/gopsutil/host/host_linux.go:35
github.com/hashicorp/nomad/client.(Client).nodeID(0xc000319900, 0x21d7360, 0x227c801, 0xc000357c00, 0xc000563450, 0xaef8a3, 0x21d7360)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1190 +0x50
github.com/hashicorp/nomad/client.(Client).setupNode(0xc000319900, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1257 +0x5b
github.com/hashicorp/nomad/client.NewClient(0xc00000c960, 0x27a29e0, 0xc0002ea0f0, 0x27d0b20, 0xc0001b2540, 0xc0004b6510, 0xc0005a26e0, 0x14)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:357 +0x8a7
github.com/hashicorp/nomad/command/agent.(Agent).setupClient(0xc00044ec60, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:756 +0x18e
github.com/hashicorp/nomad/command/agent.NewAgent(0xc0002af180, 0x280aac0, 0xc0004b6510, 0x277b3c0, 0xc00046b9e0, 0xc0003eae60, 0x272d9a0, 0xc000601a28, 0xec2944)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:116 +0x291
github.com/hashicorp/nomad/command/agent.(Command).setupAgent(0xc00016f490, 0xc0002af180, 0x280aac0, 0xc0004b6510, 0x277b3c0, 0xc00046b9e0, 0xc0003eae60, 0x5, 0x6f636f7475612d2d)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:452 +0xb0
github.com/hashicorp/nomad/command/agent.(Command).Run(0xc00016f490, 0xc00003c1a0, 0x1, 0x1, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:632 +0x466
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(CLI).Run(0xc00012e8c0, 0xc00012e8c0, 0xc00049b1c0, 0x38)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x1f1
main.RunCustom(0xc00003c190, 0x2, 0x2, 0xc0000460b8)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:138 +0x4a1
main.Run(...)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:83
main.main()
/opt/gopath/src/github.com/hashicorp/nomad/main.go:79 +0x63

[tgross]

Hi @akshatgit, this is definitely something we'll want to look into. At a first pass, I wasn't able to reproduce using this config file on buster, so maybe there's some more complex environment trigger.

Were you running Nomad as root in this case? And are you running from the Nomad released on releases.hashicorp.com or the Debian package?

[tgross]

Assuming the build is from releases.hashicorp.com the panic is bubbling up from https://github.com/hashicorp/nomad/blob/v0.10.1/vendor/github.com/shirou/gopsutil/host/host_linux.go#L316-L319. Is /etc/debian_version present?

[tgross]

@akshatgit I've been able to replicate the panic by removing /etc/debian_verion and replacing it with a zero-byte file. So you definitely will want to check that.

In the meanwhile, I've opened a PR to the upstream library: https://github.com/shirou/gopsutil/pull/822

[akshatgit]

Thanks for the prompt response, i am running it with sudo and /etc/debian_verion is empty

[tgross]

Ok, I'm going to open a new issue to track the dependency update and close this one.