Kubeadm: "kubeadm join" does not add worker node to the cluster

[amar123deep]

What keywords did you search in kubeadm issues before filing this one?

kubeadm join,
error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Ubuntu 16.04.3 LTS xenial
• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.3 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.3 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a):
  Linux master 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

Workers do not join the cluster

How to reproduce it (as minimally and precisely as possible)?

On Master (as root)
1) sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version stable-1.8
2) mkdir -p $HOME/.kube
3) sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
4) sudo chown $(id -u):$(id -g) $HOME/.kube/config

pod network (flannel):
5) kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
6) kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml

On worker node:
1) kubeadm join --token token_id master_ip:6443 --discovery-token-ca-cert-hash discovery_token_hash
2) kubectl get nodes

outputs only the master

What you expected to happen?

Expected worker to join the cluster

Anything else we need to know?

On the worker node, output from
1) kubeadm join --token token_id master_ip:6443 --discovery-token-ca-cert-hash
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "master_ip:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://master_ip:6443"
[discovery] Requesting info from "https://master_ip:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "master_ip:6443"
[discovery] Successfully established connection with API Server "master_ip:6443"
[bootstrap] Detected server version: v1.8.4
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)

Node join complete:

• Certificate signing request sent to master and response
  received.
• Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

2) sudo systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since mån 2017-11-27 09:28:06 CET; 6s ago
Docs: http://kubernetes.io/docs/
Process: 19419 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADV
Main PID: 19419 (code=exited, status=1/FAILURE)

nov 27 09:28:06 master systemd[1]: kubelet.service: Unit entered failed state.
nov 27 09:28:06 master systemd[1]: kubelet.service: Failed with result 'exit-code'.

3) sudo journalctl -xeu kubelet
nov 27 09:29:39 master kubelet[19616]: Flag --require-kubeconfig has been deprecated, You no longer need to use --require-kubeconfig. This will be removed in a future
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.124785 19616 feature_gate.go:156] feature gates: map[]
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.124844 19616 controller.go:114] kubelet config controller: starting controller
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.124850 19616 controller.go:118] kubelet config controller: validating combination of defaults and flags
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.137750 19616 client.go:75] Connecting to docker on unix:///var/run/docker.sock
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.137779 19616 client.go:95] Start docker client with request timeout=2m0s
nov 27 09:29:39 master kubelet[19616]: W1127 09:29:39.138728 19616 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
nov 27 09:29:39 master kubelet[19616]: I1127 09:29:39.143972 19616 feature_gate.go:156] feature gates: map[]
nov 27 09:29:39 master kubelet[19616]: W1127 09:29:39.144117 19616 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubec
nov 27 09:29:39 master kubelet[19616]: W1127 09:29:39.144134 19616 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be s
nov 27 09:29:39 master kubelet[19616]: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
nov 27 09:29:39 master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
nov 27 09:29:39 master systemd[1]: kubelet.service: Unit entered failed state.
nov 27 09:29:39 master systemd[1]: kubelet.service: Failed with result 'exit-code'.

4) Further, I checked /etc/kubernetes directory, only bootstrap-kubelet.conf exists, not kubelet.conf file.

[thecatwasnot]

I'm seeing this on CoreOS 1662.0.0 using kubeadm-installer container to setup machines. Looks like kubeadm init creates a kubelet.conf file for master, but no such file is created for nodes. Kubeadm lists version as: v1.9.2, Git commit: 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b

[timothysc]

We don't do anything special with https://github.com/heptiolabs/wardroom -> https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/ .

I'd have to rejig and compare.
cc: @chuckha @detiber

[roadrunner]

same here with kube version 1.8.7
is there anything to do for quick-fixing?

[bart0sh]

I couldn't reproduce this with kubelet and kubeadm 1.8.4. The reasons seems to be caused by the difference in kubelet command line options in my and your setups. My wild guess is that you don't use --bootstrap-kubeconfig option and kubelet fails to get certificates from /etc/kubernetes/bootstrap-kubelet.conf

Below is my kubelet systemd config. Note that I've added --require-kubeconfig=true there to check if that causes this issue.

$ sudo cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

And the output of "sudo journalctl -xeu kubelet" to show the difference. you can see that it's similar to yours except that it loads certificates at the end.

Feb 13 13:24:32 ed kubelet[21691]: Flag --require-kubeconfig has been deprecated, You no longer need to use --require-kubeconfig. This will be removed in a future version. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode unless --require-kubeconfig=true is also set. In the latter case, the legacy default kubeconfig path will be used until --require-kubeconfig is removed.
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.439442   21691 feature_gate.go:156] feature gates: map[HugePages:true DevicePlugins:true CPUManager:true]
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.439485   21691 controller.go:114] kubelet config controller: starting controller
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.439489   21691 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.594979   21691 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.594997   21691 client.go:95] Start docker client with request timeout=2m0s
Feb 13 13:24:32 ed kubelet[21691]: W0213 13:24:32.595657   21691 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.597224   21691 feature_gate.go:156] feature gates: map[DevicePlugins:true CPUManager:true HugePages:true]
Feb 13 13:24:32 ed kubelet[21691]: W0213 13:24:32.597317   21691 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
Feb 13 13:24:32 ed kubelet[21691]: W0213 13:24:32.597327   21691 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
Feb 13 13:24:32 ed kubelet[21691]: I0213 13:24:32.724296   21691 certificate_store.go:132] Loading cert/key pair from ("/var/lib/kubelet/pki/kubelet-client.crt", "/var/lib/kubelet/pki/kubelet-client.key").

It would help to investigate this further if you can show your /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[derekoco]

Had this issue, on your joining node ensure that the cgroup driver used by the kubelet is the same as the one used by Docker.

From the joining node:

# kubeadm reset

# cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# systemctl restart docker
# docker info | grep -i cgroup
Cgroup Driver: systemd

# kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

From the master node:

# kubectl get nodes
NAME    STATUS     ROLES     AGE       VERSION
node1   Ready      master    26m       v1.9.3
node2   Ready      <none>    1m        v1.9.3

Hope this helps,
DOC

[bart0sh]

@amar123deep Can you show the output of the following commands:

$ sudo ls -la  /etc/systemd/system/kubelet.service.d/
$ sudo cat /etc/systemd/system/kubelet.service.d/*.conf
$ sudo journalctl -xeu kubelet

It would help me to reproduce this issue and investigate it further. Thanks.

[mythi]

FWIW,

My wild guess is that you don't use --bootstrap-kubeconfig option and kubelet
fails to get certificates from /etc/kubernetes/bootstrap-kubelet.conf

helped for me. As --bootstrap-kubeconfig is critical for kubelet, it'd probably be justified to have a preflight check in kubeadm join for that.

[bart0sh]

@amar123deep Is this issue still reproducible on your cluster? Can you provide details I asked in my previous comment?

[mcronce]

@bart0sh Thank you for posting the note about --bootstrap-kubeconfig, that was a problem I was having and it took me most of the day to arrive here :)

A pre-flight check (or adding that to the default kubeadm drop-in for kubelet, or both) might be a good idea.

FTR - using kubeadm 1.9.1 on Fedora 27.

[bart0sh]

@mcronce --bootstrap-kubeconfig is used in the kubeadm drop-in since last July:
https://github.com/kubernetes/kubernetes/commit/a6fe42f78cc540a82822fb1db6c3e3a144d31126

You may want to ask Fedora guys to use that or install packages from https://packages.cloud.google.com

[bart0sh]

@timothysc I'd propose to close this issue as it's most probably caused by using incorrect/outdated command line options in the kubeadm systemd drop-in.

[thehufbro]

One more comment here, just in case someone is hunting down this problem, and finds this thread first.
I am using kubeadm 1.9.5, on CentOS 7.2

I found that all of the command line args were in the kubelet.service file, and that kubeadm join would work without returning an error, but never joined the cluster.

I found that kubelet would log that it could not find the kubelet config file.
Seen in: journalctl -xeu kubelet

... error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory

What I had to do was reload the service config, and manually start kubelet to get it working.
I am not sure how kubeadm starts the service, but during the join command it may not have loaded the modified service config after storing the bootstrap config and updating the service file.

systemctl daemon-reload
systemctl start kubelet

[rykelley]

I am getting Failed to start kubelet.service: Unit kubelet.service not found. after kubeadm bootstrap on worker node. No idea what is going on. can someone point me in the right direction?

[thehufbro]

Probably means that kubelet.service definition does not exist.
Check if the kubelet service is defined. Look in /etc/systemd/system for kubelet.service.

Did you install the kubelet?
which kubelet

install with yum
sudo yum install kubelet

[moqichenle]

@thehufbro Hi, I am trying to setup a k8s cluster on VMs.
OS: centos 7
kubeadm: 1.9.6

I followed the steps mentioned on the official website here, however, I could not even get master up. I wonder if you had any problems when setting it up. Thank you very much!

[moqichenle]

@thehufbro I checked that the kubelet is installed and denied in /etc/systemd/system/kubelet.servie.
I had the same issue that the node didn't join the cluster.
... error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
I checked I indeed doesn't have this file, even after run kubeadm join.
I tried stop kubelet , reload and restart.
Still fail.

[moqichenle]

Update: removing 10-kubeadm.conf file solved the problem for me.

[krzysztofantczak]

@moqichenle soooo are You running joning node without kubelet? Or you're running it without any arguments? I'm hitting the same problem and removing 10-kubeadm.conf doesn't seem to solve my issue.

I mean, in my case, kubelet starts - despite missing /etc/kubernetes/kubelet.conf file. But node doesn't seem to join master.

[timothysc]

Per @bart0sh comment, I'm closing this issue.

We test the installation instructions for every single https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/ release on the documented issue and have not seen failures.

We also have image automation here:
https://github.com/heptiolabs/wardroom

If folks have an exact step-by-step reliable reproduction scenario for 1.9+ please feel free to reopen, but I have not been able to reproduce.

[VincentFF]

I have the same problem on kubernetes v1.9.5.
kubectl get nodes not show the new node I joined.
I also found there is not kubelet.conf file on the new node. But other working nodes have this file.

[bart0sh]

@VincentFF Can you describe the issue you're having in more detailed way?

Does your /etc/systemd/system/kubelet.service.d/10-kubeadm.conf contain --bootstrap-kubeconfig option?

[VincentFF]

@bart0sh I have push a issue here, and not resolved:
https://github.com/kubernetes/kubeadm/issues/747

my /etc/systemd/system/kubelet.service.d/10-kubeadm.conf:

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

[bart0sh]

@VincentFF Thank you for the info!

Can you run journalctl -u kubelet after kubelet join and show its output here?

[VincentFF]

@bart0sh
Thanks for your help.
I have checked this logs and compared to other working node, they are same.

It is strange that after a few days I changed a machine and exec kubeadm join, it suddenly worked. Before this, I have changed two machines and all the three machines' settings are same.

Now it solved, but I really don't know what happend.

[VincentFF]

@bart0sh I meet the same problem again. The output of journalctl -u kubelet below:

5月 14 13:41:36 nodepreview01 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
5月 14 13:41:36 nodepreview01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
5月 14 13:41:57 nodepreview01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
5月 14 13:41:57 nodepreview01 systemd[1]: Starting kubelet: The Kubernetes Node Agent...
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. Se
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --allow-privileged has been deprecated, will be removed in a future version
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See h
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. S
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See h
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely i
5月 14 13:41:57 nodepreview01 kubelet[28887]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See ht
5月 14 13:41:57 nodepreview01 kubelet[28887]: I0514 13:41:57.563416   28887 feature_gate.go:226] feature gates: &{{} map[]}
5月 14 13:41:58 nodepreview01 kubelet[28887]: W0514 13:41:58.189928   28887 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
5月 14 13:41:58 nodepreview01 kubelet[28887]: W0514 13:41:58.193505   28887 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection
5月 14 13:41:58 nodepreview01 kubelet[28887]: I0514 13:41:58.193541   28887 server.go:376] Version: v1.10.2
5月 14 13:41:58 nodepreview01 kubelet[28887]: I0514 13:41:58.193588   28887 feature_gate.go:226] feature gates: &{{} map[]}
5月 14 13:41:58 nodepreview01 kubelet[28887]: I0514 13:41:58.193678   28887 plugins.go:89] No cloud provider specified.

[d11wtq]

I'm having this same issue. The kubelet.conf never gets generated on the node. I've enabled --v=4 to get debug logs and it seems it thinks all is good but just never progresses beyond the CSR check:

-- Unit kubelet.service has begun starting up.
Jul 18 04:06:01 jasmine systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Jul 18 04:06:02 jasmine kubelet[10891]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 04:06:02 jasmine kubelet[10891]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.939599   10891 flags.go:27] FLAG: --address="0.0.0.0"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.942073   10891 flags.go:27] FLAG: --allow-privileged="true"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.944172   10891 flags.go:27] FLAG: --allowed-unsafe-sysctls="[]"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.946287   10891 flags.go:27] FLAG: --alsologtostderr="false"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948349   10891 flags.go:27] FLAG: --anonymous-auth="true"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948456   10891 flags.go:27] FLAG: --application-metrics-count-limit="100"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948534   10891 flags.go:27] FLAG: --authentication-token-webhook="false"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948601   10891 flags.go:27] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948701   10891 flags.go:27] FLAG: --authorization-mode="AlwaysAllow"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948781   10891 flags.go:27] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948859   10891 flags.go:27] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948932   10891 flags.go:27] FLAG: --azure-container-registry-config=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.948996   10891 flags.go:27] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949068   10891 flags.go:27] FLAG: --bootstrap-checkpoint-path=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949132   10891 flags.go:27] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/bootstrap-kubelet.conf"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949205   10891 flags.go:27] FLAG: --cadvisor-port="0"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949282   10891 flags.go:27] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949351   10891 flags.go:27] FLAG: --cgroup-driver="cgroupfs"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949417   10891 flags.go:27] FLAG: --cgroup-root=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949482   10891 flags.go:27] FLAG: --cgroups-per-qos="true"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949546   10891 flags.go:27] FLAG: --chaos-chance="0"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949643   10891 flags.go:27] FLAG: --client-ca-file=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949710   10891 flags.go:27] FLAG: --cloud-config=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949773   10891 flags.go:27] FLAG: --cloud-provider=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949837   10891 flags.go:27] FLAG: --cluster-dns="[]"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.949966   10891 flags.go:27] FLAG: --cluster-domain=""
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.950031   10891 flags.go:27] FLAG: --cni-bin-dir="/opt/cni/bin"
Jul 18 04:06:02 jasmine kubelet[10891]: I0718 04:06:02.950097   10891 flags.go:27] FLAG: --cni-conf-dir="/etc/cni/net.d"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.950163   10891 flags.go:27] FLAG: --config="/var/lib/kubelet/config.yaml"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.950232   10891 flags.go:27] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.950303   10891 flags.go:27] FLAG: --container-log-max-files="5"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.950367   10891 flags.go:27] FLAG: --container-log-max-size="10Mi"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957248   10891 flags.go:27] FLAG: --container-runtime="docker"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957354   10891 flags.go:27] FLAG: --container-runtime-endpoint="unix:///var/run/dockershim.sock"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957429   10891 flags.go:27] FLAG: --containerd="unix:///var/run/containerd.sock"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957503   10891 flags.go:27] FLAG: --containerized="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957570   10891 flags.go:27] FLAG: --contention-profiling="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957634   10891 flags.go:27] FLAG: --cpu-cfs-quota="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957697   10891 flags.go:27] FLAG: --cpu-manager-policy="none"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957774   10891 flags.go:27] FLAG: --cpu-manager-reconcile-period="10s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957855   10891 flags.go:27] FLAG: --docker="unix:///var/run/docker.sock"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957925   10891 flags.go:27] FLAG: --docker-disable-shared-pid="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.957990   10891 flags.go:27] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958061   10891 flags.go:27] FLAG: --docker-env-metadata-whitelist=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958126   10891 flags.go:27] FLAG: --docker-only="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958192   10891 flags.go:27] FLAG: --docker-root="/var/lib/docker"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958259   10891 flags.go:27] FLAG: --docker-tls="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958324   10891 flags.go:27] FLAG: --docker-tls-ca="ca.pem"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958390   10891 flags.go:27] FLAG: --docker-tls-cert="cert.pem"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958454   10891 flags.go:27] FLAG: --docker-tls-key="key.pem"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958518   10891 flags.go:27] FLAG: --dynamic-config-dir=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958609   10891 flags.go:27] FLAG: --enable-controller-attach-detach="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958674   10891 flags.go:27] FLAG: --enable-debugging-handlers="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958738   10891 flags.go:27] FLAG: --enable-load-reader="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958801   10891 flags.go:27] FLAG: --enable-server="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958865   10891 flags.go:27] FLAG: --enforce-node-allocatable="[pods]"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.958973   10891 flags.go:27] FLAG: --event-burst="10"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959041   10891 flags.go:27] FLAG: --event-qps="5"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959105   10891 flags.go:27] FLAG: --event-storage-age-limit="default=0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959170   10891 flags.go:27] FLAG: --event-storage-event-limit="default=0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959242   10891 flags.go:27] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959415   10891 flags.go:27] FLAG: --eviction-max-pod-grace-period="0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959485   10891 flags.go:27] FLAG: --eviction-minimum-reclaim=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959567   10891 flags.go:27] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959648   10891 flags.go:27] FLAG: --eviction-soft=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959716   10891 flags.go:27] FLAG: --eviction-soft-grace-period=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959782   10891 flags.go:27] FLAG: --exit-on-lock-contention="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959848   10891 flags.go:27] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959915   10891 flags.go:27] FLAG: --experimental-bootstrap-kubeconfig="/etc/kubernetes/bootstrap-kubelet.conf"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.959991   10891 flags.go:27] FLAG: --experimental-check-node-capabilities-before-mount="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960057   10891 flags.go:27] FLAG: --experimental-dockershim="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960121   10891 flags.go:27] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960196   10891 flags.go:27] FLAG: --experimental-fail-swap-on="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960262   10891 flags.go:27] FLAG: --experimental-kernel-memcg-notification="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960327   10891 flags.go:27] FLAG: --experimental-mounter-path=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960390   10891 flags.go:27] FLAG: --fail-swap-on="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960546   10891 flags.go:27] FLAG: --feature-gates=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960634   10891 flags.go:27] FLAG: --file-check-frequency="20s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960710   10891 flags.go:27] FLAG: --global-housekeeping-interval="1m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960792   10891 flags.go:27] FLAG: --google-json-key=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960858   10891 flags.go:27] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960925   10891 flags.go:27] FLAG: --healthz-bind-address="127.0.0.1"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.960993   10891 flags.go:27] FLAG: --healthz-port="10248"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961064   10891 flags.go:27] FLAG: --help="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961129   10891 flags.go:27] FLAG: --host-ipc-sources="[*]"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961259   10891 flags.go:27] FLAG: --host-network-sources="[*]"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961343   10891 flags.go:27] FLAG: --host-pid-sources="[*]"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961448   10891 flags.go:27] FLAG: --hostname-override=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961512   10891 flags.go:27] FLAG: --housekeeping-interval="10s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961585   10891 flags.go:27] FLAG: --http-check-frequency="20s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961657   10891 flags.go:27] FLAG: --image-gc-high-threshold="85"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961723   10891 flags.go:27] FLAG: --image-gc-low-threshold="80"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961787   10891 flags.go:27] FLAG: --image-pull-progress-deadline="1m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961865   10891 flags.go:27] FLAG: --image-service-endpoint=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961928   10891 flags.go:27] FLAG: --iptables-drop-bit="15"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.961996   10891 flags.go:27] FLAG: --iptables-masquerade-bit="14"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962061   10891 flags.go:27] FLAG: --keep-terminated-pod-volumes="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962125   10891 flags.go:27] FLAG: --kube-api-burst="10"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962191   10891 flags.go:27] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962262   10891 flags.go:27] FLAG: --kube-api-qps="5"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962326   10891 flags.go:27] FLAG: --kube-reserved=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962395   10891 flags.go:27] FLAG: --kube-reserved-cgroup=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962457   10891 flags.go:27] FLAG: --kubeconfig="/etc/kubernetes/kubelet.conf"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962528   10891 flags.go:27] FLAG: --kubelet-cgroups=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962591   10891 flags.go:27] FLAG: --lock-file=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962656   10891 flags.go:27] FLAG: --log-backtrace-at=":0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962732   10891 flags.go:27] FLAG: --log-cadvisor-usage="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962797   10891 flags.go:27] FLAG: --log-dir=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962859   10891 flags.go:27] FLAG: --log-flush-frequency="5s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.962943   10891 flags.go:27] FLAG: --logtostderr="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963008   10891 flags.go:27] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963081   10891 flags.go:27] FLAG: --make-iptables-util-chains="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963145   10891 flags.go:27] FLAG: --manifest-url=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963207   10891 flags.go:27] FLAG: --manifest-url-header=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963301   10891 flags.go:27] FLAG: --master-service-namespace="default"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963366   10891 flags.go:27] FLAG: --max-open-files="1000000"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963450   10891 flags.go:27] FLAG: --max-pods="110"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963516   10891 flags.go:27] FLAG: --maximum-dead-containers="-1"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963580   10891 flags.go:27] FLAG: --maximum-dead-containers-per-container="1"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963646   10891 flags.go:27] FLAG: --minimum-container-ttl-duration="0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963711   10891 flags.go:27] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963786   10891 flags.go:27] FLAG: --network-plugin="cni"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963850   10891 flags.go:27] FLAG: --network-plugin-mtu="0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963913   10891 flags.go:27] FLAG: --node-ip=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.963975   10891 flags.go:27] FLAG: --node-labels=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964058   10891 flags.go:27] FLAG: --node-status-max-images="50"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964132   10891 flags.go:27] FLAG: --node-status-update-frequency="10s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964204   10891 flags.go:27] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964271   10891 flags.go:27] FLAG: --oom-score-adj="-999"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964338   10891 flags.go:27] FLAG: --pod-cidr=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964399   10891 flags.go:27] FLAG: --pod-infra-container-image="k8s.gcr.io/pause:3.1"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964466   10891 flags.go:27] FLAG: --pod-manifest-path=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964528   10891 flags.go:27] FLAG: --pod-max-pids="-1"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964594   10891 flags.go:27] FLAG: --pods-per-core="0"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964657   10891 flags.go:27] FLAG: --port="10250"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964722   10891 flags.go:27] FLAG: --protect-kernel-defaults="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964787   10891 flags.go:27] FLAG: --provider-id=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964853   10891 flags.go:27] FLAG: --qos-reserved=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964922   10891 flags.go:27] FLAG: --read-only-port="10255"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.964987   10891 flags.go:27] FLAG: --really-crash-for-testing="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965051   10891 flags.go:27] FLAG: --redirect-container-streaming="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965114   10891 flags.go:27] FLAG: --register-node="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965177   10891 flags.go:27] FLAG: --register-schedulable="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965241   10891 flags.go:27] FLAG: --register-with-taints=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965328   10891 flags.go:27] FLAG: --registry-burst="10"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965392   10891 flags.go:27] FLAG: --registry-qps="5"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965455   10891 flags.go:27] FLAG: --resolv-conf="/run/systemd/resolve/resolv.conf"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965524   10891 flags.go:27] FLAG: --root-dir="/var/lib/kubelet"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965590   10891 flags.go:27] FLAG: --rotate-certificates="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965653   10891 flags.go:27] FLAG: --runonce="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965717   10891 flags.go:27] FLAG: --runtime-cgroups=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965779   10891 flags.go:27] FLAG: --runtime-request-timeout="2m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965857   10891 flags.go:27] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965926   10891 flags.go:27] FLAG: --serialize-image-pulls="true"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.965989   10891 flags.go:27] FLAG: --stderrthreshold="2"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966053   10891 flags.go:27] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966129   10891 flags.go:27] FLAG: --storage-driver-db="cadvisor"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966194   10891 flags.go:27] FLAG: --storage-driver-host="localhost:8086"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966260   10891 flags.go:27] FLAG: --storage-driver-password="root"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966324   10891 flags.go:27] FLAG: --storage-driver-secure="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966396   10891 flags.go:27] FLAG: --storage-driver-table="stats"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966460   10891 flags.go:27] FLAG: --storage-driver-user="root"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966524   10891 flags.go:27] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966607   10891 flags.go:27] FLAG: --sync-frequency="1m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966683   10891 flags.go:27] FLAG: --system-cgroups=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966745   10891 flags.go:27] FLAG: --system-reserved=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966814   10891 flags.go:27] FLAG: --system-reserved-cgroup=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966876   10891 flags.go:27] FLAG: --tls-cert-file=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.966941   10891 flags.go:27] FLAG: --tls-cipher-suites="[]"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967035   10891 flags.go:27] FLAG: --tls-min-version=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967099   10891 flags.go:27] FLAG: --tls-private-key-file=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967269   10891 flags.go:27] FLAG: --v="4"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967340   10891 flags.go:27] FLAG: --version="false"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967431   10891 flags.go:27] FLAG: --vmodule=""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967499   10891 flags.go:27] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967574   10891 flags.go:27] FLAG: --volume-stats-agg-period="1m0s"
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.967963   10891 feature_gate.go:230] feature gates: &{map[]}
Jul 18 04:06:03 jasmine kubelet[10891]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 04:06:03 jasmine kubelet[10891]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.987351   10891 feature_gate.go:230] feature gates: &{map[]}
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:02.988045   10891 feature_gate.go:230] feature gates: &{map[]}
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.014327   10891 mount_linux.go:214] Detected OS with systemd
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.014611   10891 server.go:408] Version: v1.11.0
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.014978   10891 feature_gate.go:230] feature gates: &{map[]}
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.015510   10891 feature_gate.go:230] feature gates: &{map[]}
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.016188   10891 plugins.go:97] No cloud provider specified.
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.016294   10891 server.go:524] No cloud provider specified: "" from the config file: ""
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.016585   10891 bootstrap.go:56] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.047607   10891 bootstrap.go:86] No valid private key and/or certificate found, reusing existing private key or creating a new one
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.630650   10891 csr.go:105] csr for this node already exists, reusing
Jul 18 04:06:03 jasmine kubelet[10891]: I0718 04:06:03.980662   10891 csr.go:113] csr for this node is still valid

[chokhareganesh]

I had the same issue, Probably my master and worker node had the same host name's, After setting the different hostname, I resolve my issue

[madhu964]

I am trying to execute kubeadm join master-ip:6443 command on the new worker node I want to join to the existing kubernetes cluster

But there is no connectivity between the master and new worker node I am trying join.

I want to know on which source and destination ports I need to raise the fireflow connectivity.

Is it from source: master-Ip:6443 to destination: workernode-Ip:6443

Do i need to open up ssh dedicated port 22 also?