Consul: Intermittent Issue: Consul ACL Not Found

[charlesxu2000]

Symptom:

When doing the ACL list or ACL create, get "ACL not found".

curl -X GET http://localhost:8500/v1/acl/list?token=DUYH9oRaiWXu+gMMIUSGmg==

ACL not found

Reproduce procedure:

1) Add the acl_master_token to consul.json.
"acl_datacenter": "cqxabc",
"acl_master_token": "DUYH9oRaiWXu+gMMIUSGmg==",
"acl_default_policy": "deny",
"acl_down_policy": "extend-cache",
2) Restart consul service
3) Try to do Consul ACL list, but it returns 403 with ACL Not Found. (I have made sure the Consul leader is elected when I do the ACL list)

Consul version:

Consul v0.6.4
Consul Protocol: 3 (Understands back to: 1)

cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)

Consul info:

consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 26a0ef8c
version = 0.6.4
consul:
bootstrap = true
known_datacenters = 1
leader = true
server = true
raft:
applied_index = 32
commit_index = 32
fsm_pending = 0
last_contact = never
last_log_index = 32
last_log_term = 1
last_snapshot_index = 0
last_snapshot_term = 0
num_peers = 0
state = Leader
term = 1
runtime:
arch = amd64
cpu_count = 2
goroutines = 52
max_procs = 2
os = linux
version = go1.6
serf_lan:
encrypted = true
event_queue = 1
event_time = 2
failed = 0
intent_queue = 1
left = 0
member_time = 2
members = 1
query_queue = 0
query_time = 1
serf_wan:
encrypted = true
event_queue = 0
event_time = 1
failed = 0
intent_queue = 1
left = 0
member_time = 2
members = 1
query_queue = 0
query_time = 1

Consul config:

/etc/consul/consul.json
{
"server": true,
"rejoin_after_leave": true,
"node_name": "master",
"bind_addr": "172.22.117.250",
"advertise_addr_wan": "172.22.117.250",
"start_join": ["172.22.117.250"],
"bootstrap_expect": 1,
"client_addr": "0.0.0.0",
"recursors": ["171.70.168.183"],
"datacenter": "testdc",
"domain": "mycorp.com",
"data_dir": "/var/lib/consul",
"encrypt" : "DUYH9oRaiWXu+gMMIUSGmg==",
"disable_remote_exec": true,
"acl_datacenter": "testdc",
"acl_master_token": "DUYH9oRaiWXu+gMMIUSGmg==",
"acl_default_policy": "deny",
"acl_down_policy": "extend-cache",
"log_level": "TRACE",
"services": [],
"retry_join_wan": [ "172.22.117.250" ]
}

Here is the log:

Oct 07 14:05:27 master consul[2018]: 2016/10/07 14:05:27 [DEBUG] http: Request GET /v1/acl/list?token=DUYH9oRaiWXu+gMMIUSGmg== (212.687µs) from=127.0.0.1:59364
Oct 07 14:05:27 master consul[2018]: 2016/10/07 14:05:27 [ERR] http: Request GET /v1/acl/list?token=DUYH9oRaiWXu+gMMIUSGmg==, error: ACL not found from=127.0.0.1:59364
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] agent: Synced service 'consul'
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] consul: member 'master' joined, marking health alive
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [DEBUG] consul: reset tombstone GC to index 2
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [DEBUG] raft: Node 172.22.117.250:8300 updated peer set (2): [172.22.117.250:8300]
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] raft: Disabling EnableSingleNode (bootstrap)
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] consul: New leader elected: master
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] consul: cluster leadership acquired
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] raft: Node at 172.22.117.250:8300 [Leader] entering Leader state
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [INFO] raft: Election won. Tally: 1
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [DEBUG] raft: Vote granted from 172.22.117.250:8300. Tally: 1
Oct 07 14:05:10 master consul[2018]: 2016/10/07 14:05:10 [DEBUG] raft: Votes needed: 1
Oct 07 14:05:09 master consul[2018]: 2016/10/07 14:05:09 [INFO] raft: Node at 172.22.117.250:8300 [Candidate] entering Candidate state
Oct 07 14:05:09 master consul[2018]: 2016/10/07 14:05:09 [WARN] raft: Heartbeat timeout reached, starting election
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: Join -wan completed. Synced with 1 initial agents
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: (WAN) joined: 1 Err:
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [DEBUG] memberlist: TCP connection from=172.22.117.250:60596
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.22.117.250:8302
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: (WAN) joining: [172.22.117.250]
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: Joining WAN cluster...
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [ERR] agent: failed to sync remote state: No cluster leader
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: (LAN) joined: 1 Err:
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [DEBUG] memberlist: TCP connection from=172.22.117.250:55806
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.22.117.250:8301
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] agent: (LAN) joining: [172.22.117.250]
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] consul: adding WAN server master.cqxabc (Addr: 172.22.117.250:8300) (DC: testdc)
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] serf: EventMemberJoin: master.cqxabc 172.22.117.250
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] consul: adding LAN server master (Addr: 172.22.117.250:8300) (DC: cqxabc)
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] serf: EventMemberJoin: master 172.22.117.250
Oct 07 14:05:08 master consul[2018]: 2016/10/07 14:05:08 [INFO] raft: Node at 172.22.117.250:8300 [Follower] entering Follower state
Oct 07 14:05:08 master consul[2018]: ==> Log data will now stream in as it occurs:
Oct 07 14:05:08 master consul[2018]: Atlas:
Oct 07 14:05:08 master consul[2018]: Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Oct 07 14:05:08 master consul[2018]: Cluster Addr: 172.22.117.250 (LAN: 8301, WAN: 8302)
Oct 07 14:05:08 master consul[2018]: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Oct 07 14:05:08 master consul[2018]: Server: true (bootstrap: true)
Oct 07 14:05:08 master consul[2018]: Datacenter: 'testdc'
Oct 07 14:05:08 master consul[2018]: Node name: 'master'
Oct 07 14:05:08 master consul[2018]: ==> Consul agent running!
Oct 07 14:05:08 master consul[2018]: Join completed. Synced with 1 initial agents
Oct 07 14:05:08 master consul[2018]: ==> Joining cluster...
Oct 07 14:05:08 master consul[2018]: ==> Starting Consul agent RPC...
Oct 07 14:05:08 master consul[2018]: ==> Starting Consul agent...
Oct 07 14:05:08 master consul[2018]: ==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
Oct 07 14:05:08 master consul[2018]: ==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
Oct 07 14:05:08 master systemd[1]: Started Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable..
Oct 07 14:05:08 master systemd[1]: Starting Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable....

[mengyilangan]

请问这个问题怎么解决的