Kubeadm: Node fails to join if master's kube-apiserver has --anonymous-auth=false and basic auth

[rhtyd]

Is this a request for help?

No

What keywords did you search in kubeadm issues before filing this one?

Yes, could not find anything related to passing username/password for kube-apiserver

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
v1.11.3

Environment:

• Kubernetes version (use kubectl version): v1.11.3
• Cloud provider or hardware configuration: Manually deployed coreos VMs in CloudStack
• OS (e.g. from /etc/os-release): coreos 1800.7.0
• Kernel (e.g. uname -a): Linux k8s-kluster-k8s-master 4.14.63-coreos #1 SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 QEMU Virtual CPU version 1.5.3 GenuineIntel GNU/Linux
• Others:

What happened?

I deployed a master k8s coreos VM, after doing kubeadm init I changed the kube-apiserver yml to support basic auth using - --basic-auth-file=/etc/kubernetes/pki/auth.csv and disable anonymous auth using --anonymous-auth=false. After kube-apiserver was restarted, on the worker/node I installed the dependencies, and did /opt/bin/kubeadm join <ip>:6443 --token a4qgg6.84hjisyqzvth7yp6 --discovery-token-unsafe-skip-ca-verification I get:

[discovery] Trying to connect to API Server "10.1.1.59:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.1.1.59:6443"
[discovery] Failed to request cluster info, will try again: [Unauthorized]

With debug -v9, I can see the error is due to the apiserver trying to get the basic auth:

I0910 20:31:56.104645    6343 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://10.1.1.59:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0910 20:31:56.149999    6343 round_trippers.go:405] GET https://10.1.1.59:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 401 Unauthorized in 45 milliseconds
I0910 20:31:56.150422    6343 round_trippers.go:411] Response Headers:
I0910 20:31:56.150439    6343 round_trippers.go:414]     Content-Type: application/json
I0910 20:31:56.150453    6343 round_trippers.go:414]     Www-Authenticate: Basic realm="kubernetes-master"
I0910 20:31:56.150521    6343 round_trippers.go:414]     Content-Length: 129
I0910 20:31:56.150580    6343 round_trippers.go:414]     Date: Mon, 10 Sep 2018 20:31:56 GMT
I0910 20:31:56.160465    6343 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[discovery] Failed to request cluster info, will try again: [Unauthorized]

What you expected to happen?

Either kubeadmin join prompts for the basic auth username/password, or allows a way to pass the username/password. I think it will be useful for people who may want to deploy nodes in future for a k8s cluster whose kube-apiserver may not allow anonymous requests.

How to reproduce it (as minimally and precisely as possible)?

See above ^^.

[neolit123]

@rhtyd
you need to mount your basic auth file in the api server container.

ref:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#static-password-file

from kubeadm config this is doable with
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha2/types.go#L79
and
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha2/types.go#L303

if you look at the pod manifest for the api server, kubeadm already has some volumes and volumeMounts

give that a try and let us know how that goes.

related to;
https://github.com/kubernetes/kubeadm/issues/441

/kind feature
/priority awaiting-more-evidence

[rhtyd]

@neolit123 the auth csv is indeed mounted, that's not the issue. You missed the issue by a mile, let me try again: Using kubectl -s https://user:pass@ip:6443 works for me. The issue is that when I try to join a node to the master using kubeadm join command, it fails because there is no (documented) way to provide/pass basic auth information to it. I see it fail with unauthorized 401 error. IMO, it's a bug that blocks kudeadm join auth even when token in provided, or a minor enhancement if kubeadm can accept basic auth credentials.

In a related situation, when --anonymous-auth= override is not set but basic auth is set in the kube-apiserver manifest file, kubectl works but the url to master's endpoint (like https://ip:6443/api/v1/..) does not prompt for basic auth in browser.

[neolit123]

@kad do you have any comments on this?

@rhtyd
here is a discussion that briefly mentioned the idea of having the basic auth as part of kubeadm join and it was not addressed as an implementation:
https://github.com/kubernetes/kubeadm/issues/23#issuecomment-262316955

also contributions to kubeadm are welcome.

/remove-priority awaiting-more-evidence
/priority backlog

[fabriziopandini]

@rhtyd afaik kubeadm join with tokens/https discovery needs anonymous authentication for getting cluster-info ConfigMap
I'm pretty sure cluster-info con be passed via file, most probably removing the need of anonymous authentication, but never tested personally

[rhtyd]

I tried all options, including a manually created cluster-info file and they all don't work for me if the kube-apiserver is configured to use authorization=AlwaysAllow. The kubeadm join always fails due to missing basic auth.

[foxyriver]

when join work node, kubeadm uses bootstrap token to access kube-apiserver, it will be transferred to anonymous user, but you set --anonymous-auth=false.

[timothysc]

closing due to age and inactivity. If the issue still exists in 1.13 please feel free to reopen.

[wernerb]

This happens in 1.13 as well

[Slach]

yes, same situation
--anonymous-auth=false broke kubeadm join in 1.13
have anyone workaround?

[wernerb]

@timothysc this is still a real problem can you reopen?. There is currently no way with kubeadm in 1.13 to join the cluster (either control plane or regular) when anonymous auth is disabled as we see in this thread.

[sudosean]

same

[linxcat]

I see this issue and I don't even have basic auth enabled. Just trying to join a node with a kubeadm config through a load balancer fails with this error.

[yastij]

/assign
/lifecycle active

will get to this after kubeCon

[rhtyd]

Thanks @yastij looking forward to a fix

[neolit123]

moved to the 1.16 milestone as per @yastij 's comment.
i'm still not convinced basic auth support is something we want to add to kubeadm.

[rhtyd]

'We' the users would want simplicity of deployment and usage. If Google and partners have decided basic auth won't be supported then why move the milestone, just close the issue as won't fix.

[neolit123]

I think it will be useful for people who may want to deploy nodes in future for a k8s cluster whose kube-apiserver may not allow anonymous requests.

--anonymous-auth=true is not exactly insecure, so if someone is setting --anonymous-auth=false they have to understand the implications.

[fabriziopandini]

AFAIK joining a cluster with --anonymous-auth=false should already work in combination with file/HTTP discovery. If not, IMO this a regression and should be fixed
edited 1: this is required in case the user wants to get rid of the cluster-info config map in the public namespace, that is read by kubeadm join with anonymous access only when using tokens;
edited 2: this does not consider implications of --anonymous-auth=false on liveness ad readiness checks for the API server;

With regards to adding a new join discovery method that should prompt for the basic auth username/password, I'm -1 too because "_currently, the basic auth credentials last indefinitely, and the password cannot be changed without restarting API server. Note that basic authentication is currently supported for convenience while we finish making the more secure modes described above easier to use._" (quote from Kubernetes docs/Authenticating)

but let's discuss this today in the kubeadm office hours during the backlog grooming

[rhtyd]

Kindly discuss and close if basic auth is no longer going to be supported. I see in GKE that it notes that with v1.12+ it will be deprecated and no longer encouraged.

[wernerb]

Is it an option to sovle the the auth with client certificates? You can even issue new ones that have a short expiry date to get a role to access the cluster info configmap.

[neolit123]

we had a talk about --anonymous-auth=false during the kubeadm office hours meeting last week
VOD/timestamp is here: https://youtu.be/XNOQS8JByxM?t=1840

and we realized that it's pretty much unsupported and even if we add basic auth your cluster will start failing liveness probe checks. it's a design problem in k8s, see:
https://github.com/kubernetes/kubeadm/issues/798#issuecomment-470579937

so while basic auth itself is not something we want to add, --anonymous-auth=false will also fail unless you patch your control-plane manifests.

closing this as a unsupported. again, --anonymous-auth=true (default) is already secure.
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

we had a talk about --anonymous-auth=false during the kubeadm office hours meeting last week
VOD/timestamp is here: https://youtu.be/XNOQS8JByxM?t=1840

and we realized that it's pretty much unsupported and even if we add basic auth your cluster will start failing liveness probe checks. it's a design problem in k8s, see:
https://github.com/kubernetes/kubeadm/issues/798#issuecomment-470579937

so while basic auth itself is not something we want to add, --anonymous-auth=false will also fail unless you patch your control-plane manifests.

closing this an unsupported. again, --anonymous-auth=true (default) is already secure.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rhtyd]

I personally don't think that the default --anonymous-auth=true is secure. People host/use kubernetes cluster on publicly hosted cloud providers, many of such k8s apiservers are accessible over the public IP by default. The default allows anyone with the knowledge of the public IP address of such a k8s cluster to find the version and other details (that could be used to exploit a vulnerability) that I may not want to expose, at the same time provide a mechanism to join the cluster/master. I think I'll explore non-token based approach to solve my problem.

[neolit123]

@rhtyd

I personally don't think that the default --anonymous-auth=true is secure.

if you wish to have a public discussion about this with the owners of this area, you can log a ticket in kubernetes/kubernetes and tag it with /sig auth. SIG Auth will get to it after they do their triage.

The default allows anyone with the knowledge of the public IP address of such a k8s cluster to find the version and other details (that could be used to exploit a vulnerability) that I may not want to expose, at the same time provide a mechanism to join the cluster/master

yes, technically they will be able to find the server version and exploit a vulnerability in it.
then again, your server can be behind a firewall that accepts traffic only from trusted locations.
or perhaps your cluster is updated regularly so that it's not vulnerable - which is actually recommended.

I think I'll explore non-token based approach to solve my problem.

bootstrap tokens are secure, as long as someone does not find out a token.
that's why they also expire.

if you wish to not use tokens you can also try the kubeadm file discovery based approach:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/#file-or-https-based-discovery

have a look at it's pros and cons.

[wernerb]

@neolit123 I'm pretty sure I tested the filebased discovery but still got the same error with anonymous=false.

[neolit123]

@neolit123 I'm pretty sure I tested the filebased discovery but still got the same error with anonymous=false.

as commented above --anonymous-auth=false is unsupported by kubeadm.

[redbaron]

as commented above --anonymous-auth=false is unsupported by kubeadm.

HTTP Probe supports arbitrary headers, what if kubeadm adds static token with system:discovery clusterrole binding and uses it in liveness probes?

[neolit123]

as commented above --anonymous-auth=false is unsupported by kubeadm.

should be supported in 1.16+ if one is using file discovery with client cert and key instead of token.

@redbaron

HTTP Probe supports arbitrary headers, what if kubeadm adds static token with system:discovery clusterrole binding and uses it in liveness probes?

are you referring to?
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#putting-a-bearer-token-in-a-request
i'm aware of at least one user of HTTP probes + tokens.

i think ideally k8s HTTPS probes need TLS support, by using certificates mounted in volumes.

[redbaron]

@neolit123 ,

are you referring to?

yes, I implemented this approach by creating a static token and changing apiserever manifests to do healtchecks with them, works like a charm, would be nice if kubeadm did it for me though.

[neolit123]

i'm sure the kubadm team wouldn't mind examining a detailed proposal e.g. in a google doc.