Aws-iam-authenticator: Bad certificate error

[hugespoon]

Kubernetes Version: 1.8.4

I'm attempting to give this a try in a testing cluster I've spun up with kops 1.8.0. I think I'm quite close, but it looks like I've got a certificate problem somewhere. When attempting to do the following, I get a few errors:

kubectl --kubeconfig=/path/to/my/kubeconfig --token="$(heptio-authenticator-aws token -i mycluster.local)" get nodes

The first is immediate on my local workstation:
error: You must be logged in to the server (Unauthorized)

The second is from the container itself:
time="2017-12-05T20:14:47Z" level=info msg="http: TLS handshake error from 127.0.0.1:41980: remote error: tls: bad certificate" http=error

I'm assuming I have the wrong certificate configured somewhere, but from the docs it is not clear where that might be. Does the authenticator need to use the cluster certificates (in this case generated by kops) or are the certs it generates on its own correct?

[mirthy]

I think it's the API server not liking the fact that the heptio service is using a self-signed cert. I can confirm it's a kops with Kubernetes 1.8 issue.

From the API server logs:
E0110 15:57:04.102999 6 authentication.go:64] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, Post https://127.0.0.1:21362/authenticate: x509: certificate signed by unknown authority]]

From the service:
time="2018-01-10T15:57:04Z" level=info msg="http: TLS handshake error from 127.0.0.1:56138: remote error: tls: bad certificate" http=error

[baniol]

@hugespoon
you're not assuming role (with -r) and trying to authenticate as IAM user, which is probably not mapped in ConfigMap (https://github.com/heptio/authenticator/blob/master/example.yaml#L68)

I was able to fix it after examining the logs of the heptio-authenticator pod, where I spotted:

time="2018-01-20T22:44:05Z" level=warning msg="access denied because ARN is not mapped" arn="arn:aws:iam::000000000000:user/my-user" client="127.0.0.1:38836" method=POST path=/authenticate

EDIT:
I should have read your description more carefully, as you refer to the TLS handshake error in heptio logs. Sorry.
Having said that, I confirm stumbling upon the same (or similar) issue.
I've used kops 1.8, with added

kubeAPIServer:
    authenticationTokenWebhookConfigFile: /srv/kubernetes/webhook-authentication.yaml

to the manifest file.
I've also included fileAssets field to avoid the API server crash, as suggested in https://github.com/heptio/authenticator/issues/31.

I've finally managed to get the authenticator running by making any change to the /etc/kubernetes/manifests/kube-apiserver.manifest file on the master node and restarting kubelet service.
I haven't figured out the reason for the situation, though; nor found a way to automate the setup with kops.

[christopherhein]

@KaustubhKhati @Raffo @baniol I'm trying to better understand the issue you are running into. Are you saying that when using kops 1.8 and adding the webhook if the backend that it's suppose to connect to isn't alive it fails?

Or how can I best reproduce this?

[christopherhein]

Also, did anyone have a chance to run through #45 to verify if that solved their problems?

[Raffo]

@christopherhein I try to explain what happen to me:

• I create an entirely new kops cluster (kops 1.9 with kubernetes 1.9.3, private+public subnets, cni mode and weave, just to add more info about the setup) with the configuration exactly as explained in https://github.com/heptio/authenticator/pull/45
• I wait for the cluster to be healthy
• I apply the daemonset + configmap for the authenticator
• I get unauthorized. I see the error like in the first message: time="2017-12-05T20:14:47Z" level=info msg="http: TLS handshake error from 127.0.0.1:41980: remote error: tls: bad certificate" http=error
• To solve it, I ssh into the master node, do a docker stop of the apiserver container and do a systemctl restart kubelet.
• From there on everything works.

I haven't looked too much myself in trying to understand the full root cause of the issue, but it is definitely more than just annoying. Strange enough, I didn't have this issue with the a heptio quickstarter cluster.

[christopherhein]

@Raffo , this sounds like the masters don't have the certs when it boots thus it's creating those when you apply the daemonset which wouldn't have been loaded into the apiserver yet,

Here are the steps that I just took and it worked without restarting… Please tell me if you are doing something different. I think step 6 and 9 are what is missing, but that's just a hunch.

1. export CLUSTER_NAME=auth.debug.weave.k8s.local
2. kops create cluster ${CLUSTER_NAME} --zones us-west-1a --networking weave
3. kops edit cluster ${CLUSTER_NAME}
4. Add the following to the .spec

# ...
kubeAPIServer:
    authenticationTokenWebhookConfigFile: /srv/kubernetes/heptio-authenticator-aws/kubeconfig.yaml
  hooks:
  - name: kops-hook-authenticator-config.service
    before:
      - kubelet.service
    roles: [Master]
    manifest: |
      [Unit]
      Description=Download Heptio AWS Authenticator configs from S3
      [Service]
      Type=oneshot
      ExecStart=/bin/mkdir -p /srv/kubernetes/heptio-authenticator-aws
      ExecStart=/usr/local/bin/aws s3 cp --recursive ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

1. Making sure to replace the ${KOPS_STATE_STORE} and ${CLUSTER_NAME}
2. Generate and upload assets:

heptio-authenticator-aws init -i $CLUSTER_NAME
aws s3 cp cert.pem ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/cert.pem
aws s3 cp key.pem ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/key.pem
aws s3 cp heptio-authenticator-aws.kubeconfig ${KOPS_STATE_STORE}/${CLUSTER_NAME}/addons/authenticator/kubeconfig.yaml

1. kops update cluster ${CLUSTER_NAME} --yes
2. watch kops validate cluster until success
3. set output and state path in example.yaml and make sure to set the cluster name and some role
4. kubectl apply -f example.yaml
5. kubectl logs -f -n kube-system [authenticator POD] in a new terminal window
6. Create new user in kubeconfig with:

user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "CLUSTER_NAME"
        - "-r"
        - "ROLE_ARN"

1. Set the context to use the new user.
2. kubectl get nodes and watch the log output.

[Raffo]

I'm doing the same equivalent steps, nothing significantly different:

My hook:

hooks:
  - name: kops-hook-authenticator-config.service
    before:
      - kubelet.service
    roles: [Master]
    manifest: |
      [Unit]
      Description=Download Heptio AWS Authenticator configs from S3
      [Service]
      Type=oneshot
      ExecStart=/bin/mkdir -p /srv/kubernetes/heptio-authenticator-aws
      ExecStart=/usr/bin/docker run --net=host --rm -v /srv/kubernetes/heptio-authenticator-aws:/srv/kubernetes/heptio-authenticator-aws quay.io/coreos/awscli@sha256:7b893bfb22ac582587798b011024f40871cd7424b9026595fd99c2b69492791d aws s3 cp --recursive s3://bucket/cluster/addons/authenticator /srv/kubernetes/heptio-authenticator-aws/

I will perform additional tests after the weekend to understand better the interactions.

[christopherhein]

@Raffo any update?

[Raffo]

Unfortunately I haven't had the chance yet to work on this issue, but I will have to re-create a cluster soon, so I will be able to work on this again.

[christopherhein]

Thanks, @Raffo.

[KaustubhKhati]

@Raffo @christopherhein In my case i am applying the heptio-authenticator-daemonset first and copying the cert files to S3 bucket after that I am doing KOPS edit cluster followed by Rolling Update. This flow works for me without giving the bad certificate issue.

[Raffo]

@KaustubhKhati I think that works because you are essentially restarting the API server when you do the rolling update, right?

[KaustubhKhati]

@Raffo Yes

[Raffo]

Ok, but this is also different from the previously provided instructions that do not have a rolling update listed.

[christopherhein]

Cool, so we should be covered, agreed? can we close this?

[Raffo]

@christopherhein Well, in the list of commands in this issue, you have kops update, AFAIK this does not restart the API server node, which is saying to me that we didn't find the mistery :-D I'm still okay to close this and re-open it if I find a more reliable way to verify it, unfortunately I have very little time to invest to test this.

[KaustubhKhati]

@Raffo I think @christopherhein missed the point to add rolling update because without that there is no way kube-api will get the updated config for the webhook.

[christopherhein]

The cohesive list to get the authenticator working for a kops cluster (or any cluster) should go as follows - https://github.com/heptio/authenticator#kops-usage which includes #3 that says "If the cluster already exists, roll the cluster with kops rolling-update cluster ${CLUSTER_NAME} in order to recreate the master nodes."

[thedevelopnik]

I've gone through all the kops steps, including the rolling update, and I'm still having this issue.

[mazzy89]

I get the following error:

aws-iam-authenticator-4v2rj aws-iam-authenticator time="2019-01-10T00:00:15Z" level=info msg="mapping IAM role" groups="[system:masters]" role="arn:aws:iam::xxxxxxxxxxxx:role/kubernetes-admin" username="kubernetes-admin:{{AccountID}}:{{SessionName}}"
aws-iam-authenticator-4v2rj aws-iam-authenticator time="2019-01-10T00:00:15Z" level=info msg="loaded existing keypair" certPath=/var/aws-iam-authenticator/cert.pem keyPath=/var/aws-iam-authenticator/key.pem
aws-iam-authenticator-4v2rj aws-iam-authenticator time="2019-01-10T00:00:15Z" level=info msg="listening on https://127.0.0.1:21362/authenticate"
aws-iam-authenticator-4v2rj aws-iam-authenticator time="2019-01-10T00:00:15Z" level=info msg="reconfigure your apiserver with `--authentication-token-webhook-config-file=/etc/kubernetes/heptio-authenticator-aws/kubeconfig.yaml` to enable (assuming default hostPath mounts)"
aws-iam-authenticator-4v2rj aws-iam-authenticator time="2019-01-10T00:26:21Z" level=info msg="http: TLS handshake error from 127.0.0.1:52350: remote error: tls: bad certificate" http=error

running aws-iam-authenticator gcr.io/heptio-images/authenticator:v0.3.0 with kops v1.11.0 and k8s 1.11.5

[priitpaasukene]

It seems like there is some race condition when master node boots up - restarting kube-apiserver on master node after aws-iam-authenticator pod is up mitigated the the problem for me.

kops v1.11.0, k8s 1.11.6 and gcr.io/heptio-images/authenticator:v0.3.0

[mazzy89]

I've solved this by configuring kops correctly. It was misconfigured because I was using in the Cluster Spec both the authentication: aws block and the hook as described in the documentation. Then I just removed the hook and just used the authentication block to enable the addon and install aws-iam-authenticator and it worked

[christopherhein]

Now that kops has this in as a feature this should be an automated config. Closing. Please open if you are still running into issues.

/close

[k8s-ci-robot]

@christopherhein: Closing this issue.

In response to this:

Now that kops has this in as a feature this should be an automated config. Closing. Please open if you are still running into issues.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.