Consul: ACL migration doesn't correctly convert '_prefix' eligible items

[mooneygr]

Overview of the Issue

consul acl translate-rules and consul acl policy create ... -from-token <legacy-token> do not correctly translate ACL rules from pre-1.4 (1.3.1 ) to post-1.4 (1.4.2.); they fail to translate node "foo" to node_prefix "foo" etc

According to the Migration Doc regarding syntax changes; items like node should convert to node_prefix.

From Rule Specification doc, you'll note the return looks like:

On success, the Policy is returned:

{
    "CreateIndex": 7,
    "Hash": "UMG6QEbV40Gs7Cgi6l/ZjYWUwRS0pIxxusFKyKOt8qI=",
    "ID": "5f423562-aca1-53c3-e121-cb0eb2ea1cd3",
    "ModifyIndex": 7,
    "Name": "my-app-policy",
    "Rules": "key \"\" { policy = \"read\" } key \"foo/\" { policy = \"write\" } key \"foo/private/\" { policy = \"deny\" } operator = \"read\""
}

Reproduction Steps

Steps to reproduce this issue, eg:

1. Have ACLs enable in legacy mode (say consul 1.3.1), upgrade to Consul >1.4,

2. Find an existing ACL with node, session, service, key, etc: that should be translated to node_prefix, etc.
   

   
   

   Example

curl -H "${creds}" localhost:8500/v1/acl/info/anonymous | jq .
[
  {
    "ID": "anonymous",
    "Name": "Anonymous Token",
    "Type": "client",
    "Rules": "{\"key\": {\"\": {\"policy\": \"read\"}, \"privatething1/\": {\"policy\": \"deny\"}, \"anapplication/private/\": {\"policy\": \"deny\"}, \"privatething2/\": {\"policy\": \"deny\"}}, \"session\": {\"\": {\"policy\": \"write\"}}, \"node\": {\"\": {\"policy\": \"read\"}}, \"agent\": {\"\": {\"policy\": \"read\"}}, \"service\": {\"\": {\"policy\": \"read\"}}, \"event\": {\"\": {\"policy\": \"read\"}}, \"query\": {\"\": {\"policy\": \"read\"}}}",
    "CreateIndex": 4,
    "ModifyIndex": 137
  }
]

1. Run consul acl translate-rules (note, same result from: consul acl policy create -name "migrated-$id" -from-token $id --description ...)

Example

consul acl translate-rules -token-accessor anonymous
"key" "" {
  "policy" = "read"
}

"key" "privatething1/" {
  "policy" = "deny"
}

"key" "anapplication/private/" {
  "policy" = "deny"
}

"key" "privatething2/" {
  "policy" = "deny"
}

"session" "" {
  "policy" = "write"
}

"node" "" {
  "policy" = "read"
}

"agent" "" {
  "policy" = "read"
}

"service" "" {
  "policy" = "read"
}

"event" "" {
  "policy" = "read"
}

"query" "" {
  "policy" = "read"
}

1. Note that the ACLs under 1.4 are WRONG. They should be node_prefix, session_prefix, etc.

Consul info for both Client and Server

Server and Client info (this is a server)

$ consul info
agent:
    check_monitors = 0
    check_ttls = 0
    checks = 0
    services = 0
build:
    prerelease =
    revision = c97c712e
    version = 1.4.2
consul:
    acl = enabled
    bootstrap = false
    known_datacenters = 1
    leader = false
    leader_addr = 10.207.70.59:8300
    server = true
raft:
    applied_index = 33604
    commit_index = 33604
    fsm_pending = 0
    last_contact = 28.86653ms
    last_log_index = 33604
    last_log_term = 322
    last_snapshot_index = 32776
    last_snapshot_term = 322
    latest_configuration = [{Suffrage:Voter ID:ff71ee77-3719-bb4d-3455-e16406a7d8c3 Address:10.207.70.59:8300} {Suffrage:Voter ID:f1190421-379d-c1a0-80a0-e6d41d9f6c02 Address:10.207.70.51:8300}]
    latest_configuration_index = 1820
    num_peers = 1
    protocol_version = 3
    protocol_version_max = 3
    protocol_version_min = 0
    snapshot_version_max = 1
    snapshot_version_min = 0
    state = Follower
    term = 322
runtime:
    arch = amd64
    cpu_count = 2
    goroutines = 87
    max_procs = 2
    os = linux
    version = go1.11.4
serf_lan:
    coordinate_resets = 0
    encrypted = false
    event_queue = 0
    event_time = 43
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 39
    members = 3
    query_queue = 0
    query_time = 1
serf_wan:
    coordinate_resets = 0
    encrypted = false
    event_queue = 0
    event_time = 1
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 34
    members = 2
    query_queue = 0
    query_time = 1

Operating system and Environment details

Ubuntu 18.04 Bionic; x86_64.

Code References

All the tests in policy_test.go presently pass. But I believe this is because those tests take legit JSON input, but NOT the input that would be created/returned by the Rules field of an ACL info call:

{
    "CreateIndex": 7,
    "Hash": "UMG6QEbV40Gs7Cgi6l/ZjYWUwRS0pIxxusFKyKOt8qI=",
    "ID": "5f423562-aca1-53c3-e121-cb0eb2ea1cd3",
    "ModifyIndex": 7,
    "Name": "my-app-policy",
    "Rules": "key \"\" { policy = \"read\" } key \"foo/\" { policy = \"write\" } key \"foo/private/\" { policy = \"deny\" } operator = \"read\""
}

When running the output of Rules through: the policy code translating, but add in some debugs to make it:

        switch n := node.(type) {
        case *ast.ObjectKey:
            fmt.Printf("Node: '%v'\n", n.Token.Text)
            switch n.Token.Text {
            case "agent":
                n.Token.Text = "agent_prefix"
            case "key":
                n.Token.Text = "key_prefix"
            case "node":
                n.Token.Text = "node_prefix"
            case "query":
                n.Token.Text = "query_prefix"
            case "service":
                n.Token.Text = "service_prefix"
            case "session":
                n.Token.Text = "session_prefix"
            case "event":
                n.Token.Text = "event_prefix"
            default:
                fmt.Printf("Didnt match anything: '%s'\n", n.Token.Text)
            }
        }

Output:

Node: '"session"'
Didnt match anything: '"session"'

So when we look at the token "session" (with quotes still there) that never matches in the case statement (looking for session (no quotes)) we essentially pass though every single case and don't translate anything.

[mkeeler]

@mooneygr What is the output of running consul acl token read -token anonymous -self. This would be useful so I could compare the current rules with what the translate-rules command turned it into.

[mooneygr]

$ consul acl token read -token anonymous -self
AccessorID:   00000000-0000-0000-0000-000000000002
SecretID:     anonymous
Description:  Anonymous Token
Local:        false
Create Time:  0001-01-01 00:00:00 +0000 UTC
Policies:
Rules:
{"key": {"": {"policy": "read"}, "privatething1/": {"policy": "deny"}, "anapplication/private/": {"policy": "deny"}, "privatething2/": {"policy": "deny"}}, "session": {"": {"policy": "write"}}, "node": {"": {"policy": "read"}}, "agent": {"": {"policy": "read"}}, "service": {"": {"policy": "read"}}, "event": {"": {"policy": "read"}}, "query": {"": {"policy": "read"}}}

[mkeeler]

Sorry for the delay here. I was able to reproduce this.

My hunch right now is that its something with regards to expecting hcl as input and not the expanded JSON as we might get from decoding HCL and then serializing as JSON.

[mkeeler]

You pointed to the exact problem. The HCL parser was treating object keys that were from identifiers differently from those as strings and was leaving the quotes in place.

With the changes in the PR I just opened your token would get translated to:

key_prefix "" {
  policy = "read"
}

key_prefix "privatething1/" {
  policy = "deny"
}

key_prefix "anapplication/private/" {
  policy = "deny"
}

key_prefix "privatething2/" {
  policy = "deny"
}

session_prefix "" {
  policy = "write"
}

node_prefix "" {
  policy = "read"
}

agent_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}

event_prefix "" {
  policy = "read"
}

query_prefix "" {
  policy = "read"
}

Note that some things that were previously strings were converted to their identifier forms. This policy should produce the desired results.