Kubeadm: Liveness probes for kube-apiserver pod are failing with `--anonymous-auth=false` in place

[Evalle]

What keywords did you search in kubeadm issues before filing this one?

apiserver, anonymous-auth

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• OS:

[root@evgeny-k8s-master02:~] cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel:

Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

When I activated the option --anonymous-auth=false in kube-apiserver.yaml kubelet started to kill the apiserver pod over and over again because the liveness probes were unsuccessful. I can workaround this issue with insucure-port and insecure-bind-address options and make liveness probes ask on this insecure address and port but as of Kubernetes 1.10, the insecure flags will be deprecated: kubernetes/kubernetes#59018
Currently, there is no other way to allow unauthenticated health checks (requests on kube-apiserver's /healthz endpoint) other than allowing anonymous requests (which we do not want). Related issue: kubernetes/kubernetes#43784. Is there something I'm missing?

What you expected to happen?

apiserver works fine with --anonymous-auth=falseoption.

How to reproduce it (as minimally and precisely as possible)?

Just add --anonymous-auth=false option to kube-apiserver.yaml

[yagonobre]

@timothysc can I work on that?

[Evalle]

@yagonobre I don't know about @timothysc but I would be happy if someone will take a look at this issue :). I'm happy to help you as much as I can.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

@yagonobre @Evalle have you experienced this in newer k8s/kubeadm versions?
we might have fixed this already, but i haven't checked.

[Evalle]

@neolit123 I've tested v.1.10.x a couple of minutes ago - the same issue.

[neolit123]

@Evalle we have a small team and therefore have bandwidth issues testing 1.10 near the release of 1.12.
does v1.11 or the v1.12 beta/RC work?

[neolit123]

/remove-lifecycle stale

[xmik]

I think this won't be resolved until the idea from this comment is implemented.

[timothysc]

/assign @yagonobre

[jnummelin]

This is still happening with 1.13.2. IMO kubeadm should be clever enough to use different probe if anonymous-auth is set to false.

And without setting anonymous-auth=false it's impossible to get clean papers from CIS etc. security tests. So I feel this is kinda important to fix.

[neolit123]

And without setting anonymous-auth=false it's impossible to get clean papers from CIS etc. security tests. So I feel this is kinda important to fix.

some of the k8s maintainers will probably disagree with such tests; i'm tempted to do the same.

but most importantly this is not a kubeadm problem, we just expose the apiserver flags.
see this comment and the discussion bellow for the wider problem:
https://github.com/kubernetes/kubernetes/issues/51076#issuecomment-412846482

i'm going to close this issue and kindly ask someone to open a new one in kubernetes/kubernetes and reference both this ticket and 51076. also tag with /sig api-machinery, /sig auth, /kind bug.

thank you.