Nomad: Block alloc exec

[scalp42]

Hi folks,

We'd like to be able to block alloc exec in general.

What's the best approach for that? If ACLs, we can't seem to find the policy required to block alloc exec assuming an allow policy.

Would a flag on the client side makes sense to just be able to turn alloc exec in general per agent?

Thanks in advance

[endocrimes]

Hey @scalp42,

The alloc-exec and alloc-node-exec capabilities are the ones you'd have to omit from your policy, but there's currently no way to take a default policy and then remove capabilities. You would need to start from an empty default / read and build up from there. here is where we define those in the code though.

There is an existing flag, disable_remote_exec as part of client configuration that allows usage to be configured on a per client basis.

[scalp42]

It's a little bit confusing. Do you have an example for an anonymous policy that would behave the same as no ACLs but say prevent the "alloc exec" command?

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[scalp42]

Still an issue for us, can't figure the ACL for it.

[tgross]

Hi @scalp42!

What @endocrimes is recommending is that you build up an ACL starting from one of the coarse-grained policies and then adding the specific capabilities you want (excluding alloc-exec). See https://www.nomadproject.io/guides/security/acl.html#namespace-rules
for more details, but probably something like the following:

namespace "default" {
  policy = "read"

  # these are the additional "write" policy rules, excluding alloc-exec
  capabilities = [
    "submit-job",
    "dispatch-job",
    "read-logs",
    "read-fs",     # maybe you don't want this either?
    "alloc-lifecycle",
  ]
}

But as @endocrimes mentioned, you probably want the disable_remote_exec client configuration if you want to turn this off for a client entirely.

[scalp42]

@tgross so it sounds like we can only do that with Nomad Enterprise if I'm following correctly.

The issue with disable_remote_exec is that it's on the server side so there's no way to have access or not depending on the token from the user perspective.

[tgross]

@tgross so it sounds like we can only do that with Nomad Enterprise if I'm following correctly.

ACLs are supported on Nomad OSS. You just can't scope them to specific namespaces other than the default one (because namespaces are an enterprise feature).

The issue with disable_remote_exec is that it's on the server side so there's no way to have access or not depending on the token from the user perspective.

It's configured on the Nomad agent side, but yes it's a much more blunt instrument than ACLs so you'll want to take the approach best suited for your team and architecture.

---

Here's a quick-and-dirty demo of ACLs, using Nomad OSS:

anon_reads_policy.json:

{
    "Name": "anonymous",
    "Description": "Allow read-only access for anonymous requests",
    "Rules": "
        namespace \"default\" {
            policy = \"read\"
        }
        agent {
            policy = \"read\"
        }
        node {
            policy = \"read\"
        }
    "
}

submitter_policy.json

{
    "Name": "submitter",
    "Description": "Job submitter",
    "Rules": "
        namespace \"default\" {
          policy = \"read\"

          capabilities = [
            \"submit-job\",
            \"dispatch-job\",
            \"read-logs\",
            \"read-fs\",
            \"alloc-lifecycle\",
          ]
        }
    "
}

# start up the agent
▶ nomad agent -dev -acl-enabled
...

# in another terminal
▶ nomad acl bootstrap
Accessor ID  = ...
Secret ID    = ...
Name         = Bootstrap Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2019-11-06 13:28:40.287364 +0000 UTC
Create Index = 9
Modify Index = 9

# set the bootstrap token from the Secret ID above
▶ export NOMAD_TOKEN=...

# write the anonymous policy
▶ curl --request POST \
    --data @anon_reads_policy.json \
    -H "X-Nomad-Token: $NOMAD_TOKEN" \
    http://localhost:4646/v1/acl/policy/anonymous

# write the anonymous policy
▶ curl --request POST \
    --data @submitter_policy.json \
    -H "X-Nomad-Token: $NOMAD_TOKEN" \
    http://localhost:4646/v1/acl/policy/submitter

▶ nomad acl token create -name myuser -policy submitter
Accessor ID  = ...
Secret ID    = ...
Name         = myuser
Type         = client
Global       = false
Policies     = [submitter]
Create Time  = 2019-11-06 13:42:35.166832 +0000 UTC
Create Index = 14
Modify Index = 14

# set the user token from the Secret ID above
▶ export NOMAD_TOKEN=...

▶ nomad job init 
Example job file written to example.nomad

# we have the ability to submit a job
▶ nomad job run example.nomad
==> Monitoring evaluation "4e32d45f"
    Evaluation triggered by job "example"
    Allocation "fc518d5d" created: node "75095e5a", group "cache"
    Evaluation within deployment: "8feba3da"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "4e32d45f" finished with status "complete"

# but we can't exec into it
▶ nomad alloc exec fc518d5d /bin/sh
failed to exec into task: Permission denied

# and without our token, we can't submit jobs either
▶ unset NOMAD_TOKEN
▶ nomad job run example.nomad
Error submitting job: Unexpected response code: 403 (Permission denied)

[scalp42]

thanks a lot @tgross 💯