Consul-helm: Error with Connect-Inject when ACLs enabled.

[stvdilln]

I am able to create a Consul Cluster and Agents on an AKS cluster.
```git checkout tags/v0.10.0
helm install --name consul4 -f values-c6.yaml ./ --tls
stevedi_mac2:consul-helm stevedi$ k get pods
NAME READY STATUS RESTARTS AGE
ambassador-5db669bdd8-68c7c 1/1 Running 0 3h35m
ambassador-5db669bdd8-7ck79 1/1 Running 0 3h35m
ambassador-5db669bdd8-bv95j 1/1 Running 0 3h35m
consul4-consul-9ntf9 1/1 Running 0 6m32s
consul4-consul-connect-injector-webhook-deployment-55b88bf4znlz 1/1 Running 0 6m31s
consul4-consul-cw5v5 1/1 Running 0 6m32s
consul4-consul-kvpkl 1/1 Running 0 6m32s
consul4-consul-server-0 1/1 Running 0 6m31s
consul4-consul-server-1 1/1 Running 0 6m31s
consul4-consul-server-2 1/1 Running 0 6m31s
consul4-consul-server-acl-init-6b57v 0/1 Completed 0 3m53s
consul4-consul-server-acl-init-745m5 0/1 Error 0 5m13s
consul4-consul-server-acl-init-8526j 0/1 Error 0 6m31s
consul4-consul-server-acl-init-kw2qn 0/1 Error 0 6m24s
consul4-consul-server-acl-init-lz6g9 0/1 Error 0 5m53s
consul4-consul-server-acl-init-p8mgs 0/1 Error 0 6m13s
consul4-consul-sync-catalog-6d86d65c45-5mqzf 1/1 Running 0 6m31s
organizational-management-java-7664948589-rwkb9 0/1 CrashLoopBackOff 45 3h27m
spring-boot-demo-d4467c874-4d4cn 1/1 Running 0 3h27m

I start my test server container as:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# This name doesn't need to match anything
name: spring-boot-demo-consul
annotations:
flux.weave.works/automated: "true"
flux.weave.works/tag.gs-demo-container-name: glob:2019*
spec:
template:
metadata:
name: consul-connect-sbd
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service": "sbd-helloworld"
labels:
#MATCHSRING1
app: spring-boot-demo-consul
.....

I then launch the test service, and the init container fails:

stevedi_mac2:cluster3 stevedi$ k get pods
NAME READY STATUS RESTARTS AGE
...
spring-boot-demo-consul-84d6b846ff-kdxhg 0/2 Init:CrashLoopBackOff 4
2m59s
...
stevedi_mac2:cluster3 stevedi$ k logs spring-boot-demo-consul-84d6b846ff-kdxhg consul-connect-inject-init
Error logging in: Unexpected response code: 403 (rpc error making call: rpc error making call: Permission denied)
stevedi_mac2:cluster3 stevedi$

Inside the Init Container it looks like this line is failing:

  /bin/consul login -method="consul4-consul-k8s-auth-method" \
    -bearer-token-file="/var/run/secrets/kubernetes.io/serviceaccount/token" \
    -token-sink-file="/consul/connect-inject/acl-token" \
    -meta="pod=${POD_NAMESPACE}/${POD_NAME}"

/var/run/secrets/kubernetes.io/serviceaccount/token has a long encoded string in it.

Logs from the acl-init container are scarce, but ACLs are enabled, and tokens have been created for the k8s servers.

stevedi_mac2:consul-helm stevedi$ k logs consul4-consul-server-acl-init-6b57v
2019-10-08T17:51:54.554Z [INFO ] 10.254.0.25:8500
2019-10-08T17:51:54.554Z [INFO ] 10.254.2.28:8500
2019-10-08T17:51:54.554Z [INFO ] 10.254.1.26:8500
```
I've created new AKS clusters and have a very repeatable scenario. LMK what other logs/diagnostics you need.

[lkysow]

Hi Steve,
Are you using the latest consul-helm chart (v0.10.0) with the latest version of consul-k8s (v0.9.2)? We fixed a lot of issues with ACL bootstrapping in the latest release.

FYI, there are still some bugs with it (around upgrading) that we're hoping to fix in the coming days.

[stvdilln]

I was on consul-helm chart v0.10.0, consul-k8s v0.9.1 and consul 1.6.0. I updated to consul-k8s to v0.9.2 and consul to 1.6.1.
Unfortunately this did not clear up the issue.
I then tested with a fresh values.yaml, just updating consul to 1.6.1, enabling bootstrap acls and enabling connect-inject and am still having the issue.

[lkysow]

Hi Steve,
I think I've figured out what's happening.

I'm guessing that your app is using the default ServiceAccount?

$ kubectl get pod static-server -o yaml | grep serviceAccountName
  serviceAccountName: default

By default, when the helm chart sets up the Auth Method it uses the following binding rule selector:
https://github.com/hashicorp/consul-helm/blob/57f9b8554426142692eda29afd2dab71d27fd564/values.yaml#L427

This means that the default ServiceAccount can't log in.

So to resolve this, you'll either need to

1. Use a different ServiceAccount for your Pod
2. or set the binding rule selector to empty (see https://www.consul.io/docs/acl/auth-methods/kubernetes.html#trusted-identity-attributes)
   yaml connectInject: aclBindingRuleSelector: ""

in addition...

The service account name must match the Consul service name. In this case, you're annotating your pod with "consul.hashicorp.com/connect-service": "sbd-helloworld" and so your ServiceAccount must be sbd-helloworld. Otherwise you'll get a different error:

Error registering service "sbd-helloworld-sidecar-proxy": Unexpected response code: 403 (Permission denied)

Because the token when you successfully log in will only give you permissions as if you were a service with the same name as the service account. A different service name (e.g. default) is not allowed to register sidecar proxies for a service called sbd-helloworld.

Finally, this was all way too complicated to figure out and we need to do better, both with our documentation and our error messages. I'm going to create some issues in our backlog accordingly. Sorry for the trouble!

[stvdilln]

I'll spin up a test k8s cluster and give it a try. Now that I have external consul servers working, my need to for bootstrapACLs has gone away, but willing to investigate the issue.

[stvdilln]

Thanks, I don't know how you guessed that, but the service will now register with consul without the 403 error.

I was having an error with the client, but still looking at it.

[lkysow]

Can you take some screenshots of the consul UI? I need to see the upstream config of the client and the proxies health

[stvdilln]

It's time to open the beer!

root@static-client:/# curl http://localhost:1234/
Hello World
root@static-client:/# 

So on Friday I did as you had asked, and my demonstration containers got past init, but were just getting connection reset when the client hit the proxy sidecar. Looking at the GUI the services hadn't registered. I debugged side-car init scripts and bunch of stuff today, but eventually in frustration I nuked the k8s cluster and rebuilt.
I don't know what changed, but it's working now.
A couple of steps after what you have given me:
1) Go get the bootstrap token from the kubernetes secrets and put it in the UI.
2) After both of your client and server have initialized, go into the gui and create an intention (In the correct direction).

Thanks so much for the help, would have never gotten it all without the help!
There isn't a bug in consul-helm, it's just hard to figure out connect-inject.
Steve

[stvdilln]

I wrote this thread and what I learned up as a demo repository. Anybody that got here by google can see what all I did to make this work. Suggestions welcome. Currently just tested with Azure/AKS but will try to do AWS/EKS soon.
https://github.com/stvdilln/consul-inject-demo

[lkysow]

Great work Steve. We need to add a section to our docs about working with Connect when ACLs are enabled. We have https://github.com/hashicorp/consul-helm/issues/202 to track some documentation changes so maybe we can do it as a part of that.

I'll close this for now since I think we're good?