Nomad: Nomad 0.8.4 panics when talking to vault 0.11.0

[groggemans]

Nomad version

Nomad v0.8.4 (dbee1d7d051619e90a809c23cf7e55750900742a)

Operating system and Environment details

Raspbian linux

Issue

Nomad panics when trying to access the vault api

Reproduction steps

Run nomad agent in server mode and connect with a vault 0.11.0 cluster

Nomad Server logs (if appropriate)

WARNING: keyring exists but -encrypt given, using keyring
==> Loaded configuration from /etc/nomad.d/base.hcl, /etc/nomad.d/client.hcl, /etc/nomad.d/server.hcl
==> Starting Nomad agent...
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 67 [running]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api.(*Sys).Capabilities(0x13bd8dc8, 0x13b20181, 0x24, 0x13bfd17, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api/sys_capabilities.go:38 +0x4c4
github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api.(*Sys).CapabilitiesSelf(0x13bd8dc8, 0x13bfd17, 0x15, 0x0, 0x13784780, 0x76d6a6c8, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api/sys_capabilities.go:6 +0x3c
github.com/hashicorp/nomad/nomad.(*vaultClient).hasCapability(0x13b821b0, 0x13bfd17, 0x15, 0x1f0d668, 0x2, 0x2, 0x22ac4, 0x13ac0320, 0x10, 0x10, ...)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:753 +0x38
github.com/hashicorp/nomad/nomad.(*vaultClient).validateCapabilities(0x13b821b0, 0x13b1c8a1, 0xd, 0x139c4100, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:696 +0x68
github.com/hashicorp/nomad/nomad.(*vaultClient).parseSelfToken(0x13b821b0, 0x1, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:668 +0x1a0
github.com/hashicorp/nomad/nomad.(*vaultClient).establishConnection(0x13b821b0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:428 +0x11c
github.com/hashicorp/nomad/nomad.(*vaultClient).(github.com/hashicorp/nomad/nomad.establishConnection)-fm()
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:247 +0x1c
github.com/hashicorp/nomad/nomad.wrapNilError.func1(0x0, 0x1399d040)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:1186 +0x1c
github.com/hashicorp/nomad/vendor/gopkg.in/tomb%2ev2.(*Tomb).run(0x139a8e40, 0x1370f718)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/gopkg.in/tomb.v2/tomb.go:153 +0x1c
created by github.com/hashicorp/nomad/vendor/gopkg.in/tomb%2ev2.(*Tomb).Go
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/gopkg.in/tomb.v2/tomb.go:149 +0xa4

I think this needs a fix in the vault code, but posting here for completeness and follow up.

[jefferai]

Nomad needs to pull in updated API libs

[jefferai]

This doesn't seem to be a 0.11 bug. It appears to be this: https://github.com/hashicorp/vault/pull/4553 which was fixed in Vault 0.10.2. This, however, appears to be after Nomad last pulled Vault API libs.

We'll put in a workaround for 0.11.1 but Nomad should pull in updated API libs.

[kak-tus]

May be problem is not only in https://github.com/hashicorp/vault/pull/4553?

I dumped request/response in this case.

Request:

(*http.Request)(0xc420c36900)({
 Method: (string) (len=4) "POST",
 URL: (*url.URL)(0xc4211d0280)(http://vault.query.consul:8200/v1/sys/capabilities-self),
 Proto: (string) (len=8) "HTTP/1.1",
 ProtoMajor: (int) 1,
 ProtoMinor: (int) 1,
 Header: (http.Header) (len=1) {
  (string) (len=13) "X-Vault-Token": ([]string) (len=1 cap=1) {
   (string) (len=36) "mytoken"
  }
 },
 Body: (ioutil.nopCloser) {
  Reader: (*bytes.Buffer)(0xc420414930)({"path":"sys/capabilities-self","token":"mytoken"}
)

In curl form

curl -H 'X-Vault-Token:mytoken' -d '{"path":"sys/capabilities-self","token":"mytoken"}' 'http://vault.query.consul:8200/v1/sys/capabilities-self'

Response on this request is

{"request_id":"f6610f0f-1818-0188-8273-0290172cda05","lease_id":"","renewable":false,"lease_duration":0,"data":{"capabilities":["root"],"sys/capabilities-self":["root"]},"wrap_info":null,"warnings":null,"auth":null}

So, as you can see - there is exists "capabilities" key in response - in "data" subkey. And fix in https://github.com/hashicorp/vault/pull/4553/files do

capabilitiesRaw, ok := result["capabilities"].([]interface{})

And this code not fully fixed the problem - it try to get key from wrong response path.

May be this is misunderstanding with Vault team and response format?

[adamlc]

I'm also having this issue! I'm a little worried that if my nomad servers restart for whatever reason then they aren't going to come back. So it sounds like updating the vault libs alone doesn't fix the issue?

[kak-tus]

@adamlc I fixed it in file sys_capabilities.go and use nomad version with this patch

[adamlc]

@kak-tus thanks I'll give it a try!

[adamlc]

I can confirm the above fix is working perfectly :)