Kubeadm: Kubeadm deploy kube-proxy's --kubeconfig file is wrong

[riverzhang]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT
Deploy a highly available k8s cluster. Each time kubeadm init, kube-proxy user authority kubeconfig file server value will become the last kubeadm init apiserver address.

Versions

-kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Environment:
• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}- Cloud provider or hardware configuration:
  none
• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux node1 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready master 31m v1.10.2
node2 Ready master 30m v1.10.2
node3 Ready master 30m v1.10.2
node4 Ready node 30m v1.10.2
[root@node1 kubespray]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kube-apiserver-node1 1/1 Running 0 27m
kube-apiserver-node2 1/1 Running 0 27m
kube-apiserver-node3 1/1 Running 0 27m
kube-controller-manager-node1 1/1 Running 0 27m
kube-controller-manager-node2 1/1 Running 0 27m
kube-controller-manager-node3 1/1 Running 0 27m
kube-dns-7bd4d5fbb6-dwlkg 3/3 Running 0 25m
kube-dns-7bd4d5fbb6-wr57v 3/3 Running 0 25m
kube-flannel-5bs56 2/2 Running 0 26m
kube-flannel-62zgt 2/2 Running 0 26m
kube-flannel-l5kw6 2/2 Running 0 26m
kube-flannel-lgh22 2/2 Running 0 26m
kube-proxy-4h9th 1/1 Running 0 27m
kube-proxy-5bdfm 1/1 Running 0 28m
kube-proxy-gvwpr 1/1 Running 0 28m
kube-proxy-rv2r4 1/1 Running 0 28m
kube-scheduler-node1 1/1 Running 0 27m
kube-scheduler-node2 1/1 Running 0 27m
kube-scheduler-node3 1/1 Running 0 27m
kubedns-autoscaler-679b8b455-l7db8 1/1 Running 0 25m
kubernetes-dashboard-55fdfd74b4-pdrml 1/1 Running 0 25m
nginx-proxy-node4 1/1 Running 0 27m
[root@node1 kubespray]# kubectl exec -it kube-proxy-4h9th /bin/bash -n kube-system
root@node4:/# cat /var/lib/kube-proxy/kubeconfig.conf
apiVersion: v1
kind: Config
clusters:

cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://10.10.0.6:6443
name: default
contexts:
context:
cluster: default
namespace: default
user: default
name: default
current-context: default
users:
name: default
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokenroot@node4:/# exit
exit
[root@node1 kubespray]# kubectl exec -it kube-proxy-rv2r4 /bin/bash -n kube-system
root@node2:/# cat /var/lib/kube-proxy/kubeconfig.conf
apiVersion: v1
kind: Config
clusters:
cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://10.10.0.6:6443
name: default
contexts:
context:
cluster: default
namespace: default
user: default
name: default
current-context: default
users:
name: default
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokenroot@node2:/#

The kube-proxy' kubeconfig server value of all nodes points to the same apiserver. Is there any problem with this configuration?

What you expected to happen?

The kube-proxy kubeconfig value server points to the apiserver address of its own node

[riverzhang]

/assign @timothysc

[seh]

Note that you can use the _MasterConfiguration_ file's "api.controlPlaneEndpoint" field to point all the _kube-proxy_ configurations to the _same_ API server, but that doesn't accomplish what you want: Components sitting on each master machine talk to the local API server.

That configuration comes from the "kube-proxy" _ConfigMap_ in the "kube-system" namespace, and the last master machine to pass through _kubeadm alpha phase addon kube-proxy_ writes its own address to that _ConfigMap_. That's why the current "Creating HA clusters with kubeadm" document advises editing that _ConfigMap_ manually in the "Configure workers" section. (I don't know why it's mentioned in that section, so late in the process.)

Note that as you bring each master up one by one, it's possible that the _kube-proxy_ process starting on each will observe a different value from that _ConfigMap_, and you wind up with a reasonable variation—until any of the processes fail and get replaced.

I'd like to see a cross-cutting pattern afforded by _kubeadm_ for HA clusters:

• Components on master machines talk to the local API server.
• Components on other non-master worker machines talk to a load balancer fronting the API servers.

That would require, though, that components like _kube-proxy_ that run on all the nodes would need different configuration on the masters. Loading the configuration from a local file rather than a _ConfigMap_ would make it easier to lay down that non-uniform arrangement (that is, there's a file at the same path on each machine, but with varying content).

[seh]

Back on 18 May 2018, I asked this question in the "kubeadm" channel of the "Kuberetes" Slack team:

In the HA guide’s “Configure workers” section, it advises modifying the “kube-proxy” _ConfigMap_ consumed by _kube-proxy_.

However, looking at my _kube-proxy_ pods, while it looks like they mount this _ConfigMap_, they only reference the “config.conf” key, and not the “kubeconfig.conf” key.

Is this modification still necessary?

[seh]

Looking again today, I see that the "config.conf" key has the following field:

clientConnection:
  # ...
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf

Hence, _kube-proxy_ is using that "kubeconfig.conf" key.

[seh]

Here's how I wound up working around this problem.

First, I created two different KUBECONFIG files: one for the masters that refers to the API server via the _127.0.0.1_ IP address, and another for the workers that refers to the API server load balancer via its domain name.

Here's the masters' file:

apiVersion: v1
kind: Config
clusters:
- name: default
  cluster:
    server: https://127.0.0.1:6443
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
contexts:
- name: default
  context:
    cluster: default
    namespace: default
    user: default
current-context: default
users:
- name: default
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Here's the workers' file:

apiVersion: v1
kind: Config
clusters:
- name: default
  cluster:
    server: https://<FQDN of load balancer>
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
contexts:
- name: default
  context:
    cluster: default
    namespace: default
    user: default
current-context: default
users:
- name: default
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

On each master and worker machine, I store this file at path _/etc/kubernetes/in-cluster.conf_.
Before running _kubeadm alpha phase addon all_, be sure to include the following stanza in the _MasterConfiguration_ file:

kubeProxy:
  config:
    clientConnection:
      kubeconfig: /var/lib/kube-proxy-kubeconfig.yaml

Run _kubeadm alpha phase addon all_, feeding it that _MasterConfiguration_ file, then patch _kube-proxy_'s _DaemonSet_ immediately afterward to mount this file at the path nominated in the "kube-proxy" _ConfigMap_'s "config.conf" entry.

# Coerce kube-proxy to read a KUBECONFIG file mounted from the host
# rather than the one mounted from the "kube-proxy" ConfigMap in the
# "kube-system" namespace, in order to allow the API server endpoint
# to vary between the load balancer for the workers and localhost for
# the masters (avoiding using the load balancer, which fails for
# hairpinned packets).
add_volumes_patch=$(cat <<EOF
spec:
  template:
    spec:
      containers:
      - name: kube-proxy
        volumeMounts:
        - name: kube-proxy-kubeconfig
          mountPath: /var/lib/kube-proxy-kubeconfig.yaml
      volumes:
      - name: kube-proxy-kubeconfig
        hostPath:
          path: /etc/kubernetes/in-cluster.conf
          type: File
EOF
)
sudo /opt/bin/kubectl \
     --namespace=kube-system \
     patch daemonset kube-proxy \
     --patch "${add_volumes_patch}"

With that patch, we force _kube-proxy_ to use the host-provided file, rather than the KUBECONFIG file stored in the "kube-proxy" _ConfigMap_.

[riverzhang]

@seh Thank you for your help. I think kube-proxy's kubeconfig can be configured by kubeadm in the best way.This hope can be improved in the future, and users can easily configure it.

[riverzhang]

fix in kubespray https://github.com/kubernetes-incubator/kubespray/pull/2872

[munai-das]

@seh I followed your steps in v1beta1 kubeadm for kubernetes 1.14.1
It fails with the error
F0625 19:33:36.516452 1 server.go:463] failed complete: no kind "Config" is registered for version "v1" in scheme "k8s.io/kubernetes/pkg/proxy/apis/config/scheme/scheme.go:29"

Did you encounter this issue?

[munai-das]

Found the issue. I was passing the kubeconfig instead of kube-proxy config to kube-proxy.
I will let it be here for anyone doing the same mistake.