Nomad: Nomad agent fails to start

[SergeyYakubov]

Hi!

I have problems running nomad.

Nomad version

Output from nomad version
Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f
also tried with fresh compiled 0.8.4-dev

Operating system and Environment details

bare metal CentOS Linux release 7.4.1708 (Core)

Issue

trying to start nomad client fails immediately. nomad server seems to work

./nomad agent -dev
==> No configuration files loaded
==> Starting Nomad agent...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcd3c7f]

goroutine 1 [running]:
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(Client).SetTimeout(...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:349
github.com/hashicorp/nomad/client/driver.(DockerDriver).newDockerClient(0xc420365800, 0x45d964b800, 0x20d23a8, 0x0, 0x38)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1103 +0x38f
github.com/hashicorp/nomad/client/driver.(DockerDriver).dockerClients(0xc420365800, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1047 +0x19c
github.com/hashicorp/nomad/client/driver.(DockerDriver).Fingerprint(0xc420365800, 0xc420429a60, 0xc4200f39e0, 0x6, 0x20d3d40)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:498 +0x43
github.com/hashicorp/nomad/client.(FingerprintManager).fingerprintDriver(0xc420716740, 0x159d192, 0x6, 0x2b8c8ce54000, 0xc420365800, 0x1, 0x0, 0x12, 0xc4206cd2b0)
/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:347 +0x17d
github.com/hashicorp/nomad/client.(FingerprintManager).setupDrivers(0xc420716740, 0xc420365780, 0x6, 0x8, 0x20d3d40, 0xc420365700)
/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:190 +0x1f1
github.com/hashicorp/nomad/client.(FingerprintManager).Run(0xc420716740, 0xc420716740, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:129 +0xb4f
github.com/hashicorp/nomad/client.NewClient(0xc420736000, 0x2006f00, 0xc4201d6010, 0x200f500, 0xc4200cc240, 0xc4205460f0, 0xc42000e030, 0x1107c65, 0xc420603888)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:273 +0xa8d
github.com/hashicorp/nomad/command/agent.(Agent).setupClient(0xc42045a150, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:633 +0x14a
github.com/hashicorp/nomad/command/agent.NewAgent(0xc4201fa780, 0x2002080, 0xc42044ef20, 0xc4201d8fa0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:90 +0x227
github.com/hashicorp/nomad/command/agent.(Command).setupAgent(0xc420212310, 0xc4201fa780, 0x2002080, 0xc42044ef20, 0xc4201d8fa0, 0xc42044ef20, 0xc420024000)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:382 +0x96
github.com/hashicorp/nomad/command/agent.(Command).Run(0xc420212310, 0xc4200101a0, 0x1, 0x1, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:494 +0x3ea
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc42046fa40, 0xc42046fa40, 0xc420543920, 0x37)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x1eb
main.RunCustom(0xc420010190, 0x2, 0x2, 0xc4204f1f60)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:127 +0x4c9
main.Run(0xc420010190, 0x2, 0x2, 0x11ce214)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:72 +0x3f
main.main()
/opt/gopath/src/github.com/hashicorp/nomad/main.go:68 +0x63

Reproduction steps

download nomad
$ nomad agent -dev

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Job file (if appropriate)

[preetapan]

@SergeyYakubov Thanks for reporting, that looks like a potential unsafe call - we will have a fix for it in the upcoming 0.8.4 release.

To help us determine the exact conditions that cause this panic - Do you have docker installed on the machine you ran the client on? Any other environment variables starting with DOCKER_?

[SergeyYakubov]

@preetapan I seem to see the reason - I have docker installed, but it configured to use TCP port instead of unix socket. Therefore I have DOCKER_HOST=localhost:2376 and DOCKER_TLS_VERIFY=1 and DOCKER_CERT_PATH. Is it possible to use nomad in such configuration?

[SergeyYakubov]

OK, I have configured the client with
client {
enabled = true
options = {
"docker.endpoint" = "0.0.0.0:2376",
"docker.tls.ca" = "path/ca.pem",
"docker.tls.cert" = "path/cert.pem",
"docker.tls.key" = "path/server-key.pem"
}
}
and it seems to work now.
But a nice(r) error message would be helpful :)

[preetapan]

@SergeyYakubov glad you figured that out

4303 fixes the panic. If the docker client can't be created, we emit node events and the node status CLI command should tell you if there's an issue with detecting docker.