Nomad: Job accessing CSI fails when ACL enabled

[vincenthuynh]

Nomad version

0.11.1

Operating system and Environment details

Debian 9

Issue

Reproduction steps

• Enable ACLs with the following Anonymous policy:

namespace "*" {
  policy       = "write"
  capabilities = ["alloc-node-exec", "csi-register-plugin", "csi-list-volume", "csi-read-volume"]
}

agent {
  policy = "write"
}

operator {
  policy = "write"
}

quota {
  policy = "write"
}

node {
  policy = "write"
}

host_volume "*" {
  policy = "write"
}

• Follow CSI guide

Job file (if appropriate)

My setup is outlined here

Nomad Client logs (if appropriate)

May  5 21:04:19 nmd-rpzq nomad[19194]:     2020-05-05T21:04:19.041Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: Permission denied" rpc=CSIVolume.Claim server=<nomad_server_ip>:4647
May  5 21:04:19 nmd-rpzq nomad[19194]:     2020-05-05T21:04:19.041Z [ERROR] client.alloc_runner: prerun failed: alloc_id=d1ece247-eb45-1a71-f4b0-424db8701926 error="pre-run hook "csi_hook" failed: claim volumes:
rpc error: Permission denied"

Nomad Server logs (if appropriate)

[tgross]

Thanks for opening this issue @vincenthuynh. We'll investigate!

[vrenjith]

+1

[tgross]

Hi @vincenthuynh and @vrenjith! I took a look at this and I think the policy is missing the ability to read plugins. If you try running nomad plugin status you should see an error as well. Add the following to your policy doc:

plugin {
  policy = "read"
}

I've tested this out and was able to register volume and claim it for a job. Let me know if that works for you, and if so I'll see what I can do to make the documentation more clear.

[vincenthuynh]

Hi @tgross,
The plugin rule worked! Thanks for the help!

[tgross]

Glad to hear! I'm going to make sure this in included in some of the ACLs reference documentation we're working on in https://github.com/hashicorp/nomad/issues/7475

[tgross]

I've also got the appropriate section added to the Learn guide on ACLs: https://learn.hashicorp.com/nomad/acls/policies#plugin-rules