kubeadm init failed in standalone vm

[jeffwubj]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:02:01Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  NAME="VMware Photon OS"
  VERSION="2.0"
  ID=photon
  VERSION_ID=2.0
  PRETTY_NAME="VMware Photon OS/Linux"
  ANSI_COLOR="1;34"
  HOME_URL="https://vmware.github.io/photon/"
  BUG_REPORT_URL="https://github.com/vmware/photon/issues"
• Kernel (e.g. uname -a):
  Linux photon-machine 4.9.53-5.ph2-esx #1-photon SMP Thu Oct 26 02:44:24 UTC 2017 x86_64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz GenuineIntel GNU/Linux
• Others:

What happened?

int command line, type
kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.13.0 , it will fail with below error

[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [photon-machine localhost] and IPs [172.16.34.130 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [photon-machine localhost] and IPs [172.16.34.130 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [photon-machine kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.34.130]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
    timed out waiting for the condition

This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
    - 'docker ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluste

What you expected to happen?

kubeadm could success

How to reproduce it (as minimally and precisely as possible)?

yes, kubeadm init

Anything else we need to know?

journalctl -f -u kubelet

Dec 10 23:03:24 photon-machine systemd[1]: Started kubelet: The Kubernetes Node Agent.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.605580    3384 server.go:407] Version: v1.13.0
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.605861    3384 plugins.go:103] No cloud provider specified.
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.605962    3384 server.go:552] standalone mode, no API client
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.617826    3384 server.go:464] No api server defined - no events will be sent to API server.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.617840    3384 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.617997    3384 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: []
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.618008    3384 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.618072    3384 container_manager_linux.go:272] Creating device plugin manager: true
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.618142    3384 state_mem.go:36] [cpumanager] initializing new in-memory state store
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.624222    3384 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.624384    3384 client.go:104] Start docker client with request timeout=2m0s
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.625217    3384 docker_service.go:540] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.625341    3384 docker_service.go:236] Hairpin mode set to "hairpin-veth"
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.625493    3384 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.626775    3384 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.627511    3384 docker_service.go:251] Docker cri networking managed by kubernetes.io/no-op
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.632474    3384 docker_service.go:256] Docker Info: &{ID:U6ZI:XT3N:CP3E:HU2E:HSDB:ZR4C:NUG2:52DJ:ZY7D:PC2V:26UK:SQD2 Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:18 OomKillDisable:true NGoroutines:25 SystemTime:2018-12-10T23:03:25.628955239Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.9.53-5.ph2-esx OperatingSystem:VMware Photon OS/Linux OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc00084c3f0 NCPU:2 MemTotal:2101800960 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:photon-machine Labels:[] ExperimentalBuild:false ServerVersion:17.06.0-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:cfb82a876ecc11b5ca0977d1733adbe58599088a Expected:cfb82a876ecc11b5ca0977d1733adbe58599088a} RuncCommit:{ID:2d41c047c83e09a6d61d464906feb2a2f3c52aa4 Expected:2d41c047c83e09a6d61d464906feb2a2f3c52aa4} InitCommit:{ID:949e6fa Expected:949e6fa} SecurityOptions:[name=seccomp,profile=default]}
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.634036    3384 docker_service.go:269] Setting cgroupDriver to cgroupfs
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.648951    3384 kuberuntime_manager.go:198] Container runtime docker initialized, version: 17.06.0-ce, apiVersion: 1.30.0
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.654991    3384 server.go:999] Started kubelet
Dec 10 23:03:25 photon-machine kubelet[3384]: E1210 23:03:25.655303    3384 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.655417    3384 kubelet.go:1412] No api server defined - no node status update will be sent.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.655953    3384 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.656071    3384 status_manager.go:148] Kubernetes client is nil, not starting status manager.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.656192    3384 kubelet.go:1829] Starting kubelet main sync loop.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.656290    3384 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.656424    3384 server.go:137] Starting to listen on 0.0.0.0:10250
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.656918    3384 server.go:333] Adding debug handlers to kubelet server.
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.658019    3384 volume_manager.go:248] Starting Kubelet Volume Manager
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.660395    3384 desired_state_of_world_populator.go:130] Desired state populator starts to run
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.713359    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.722741    3384 cpu_manager.go:155] [cpumanager] starting with none policy
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.723166    3384 cpu_manager.go:156] [cpumanager] reconciling every 10s
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.723354    3384 policy_none.go:42] [cpumanager] none policy: Start
Dec 10 23:03:25 photon-machine kubelet[3384]: W1210 23:03:25.723967    3384 manager.go:528] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.725546    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:03:25 photon-machine kubelet[3384]: I1210 23:03:25.763264    3384 reconciler.go:154] Reconciler: start to sync state
Dec 10 23:03:35 photon-machine kubelet[3384]: I1210 23:03:35.738545    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:03:45 photon-machine kubelet[3384]: I1210 23:03:45.749813    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:03:55 photon-machine kubelet[3384]: I1210 23:03:55.763865    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:05 photon-machine kubelet[3384]: I1210 23:04:05.776111    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:15 photon-machine kubelet[3384]: I1210 23:04:15.788775    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:25 photon-machine kubelet[3384]: I1210 23:04:25.800617    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:35 photon-machine kubelet[3384]: I1210 23:04:35.813075    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:45 photon-machine kubelet[3384]: I1210 23:04:45.828025    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:04:55 photon-machine kubelet[3384]: I1210 23:04:55.840518    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:05 photon-machine kubelet[3384]: I1210 23:05:05.854403    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:15 photon-machine kubelet[3384]: I1210 23:05:15.867459    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:25 photon-machine kubelet[3384]: I1210 23:05:25.884291    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:35 photon-machine kubelet[3384]: I1210 23:05:35.900578    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:45 photon-machine kubelet[3384]: I1210 23:05:45.912557    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:05:55 photon-machine kubelet[3384]: I1210 23:05:55.924606    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:06:05 photon-machine kubelet[3384]: I1210 23:06:05.936605    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:06:15 photon-machine kubelet[3384]: I1210 23:06:15.950359    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Dec 10 23:06:25 photon-machine kubelet[3384]: I1210 23:06:25.961810    3384 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach

root@photon-machine [ ~ ]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-12-10 23:03:24 UTC; 7min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 3384 (kubelet)
    Tasks: 14 (limit: 4915)
   Memory: 29.6M
      CPU: 5.992s
   CGroup: /system.slice/kubelet.service
           └─3384 /usr/bin/kubelet

Dec 10 23:09:06 photon-machine kubelet[3384]: I1210 23:09:06.168176    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:09:16 photon-machine kubelet[3384]: I1210 23:09:16.179300    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:09:26 photon-machine kubelet[3384]: I1210 23:09:26.194037    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:09:36 photon-machine kubelet[3384]: I1210 23:09:36.205698    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:09:46 photon-machine kubelet[3384]: I1210 23:09:46.218042    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:09:56 photon-machine kubelet[3384]: I1210 23:09:56.229847    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:10:06 photon-machine kubelet[3384]: I1210 23:10:06.245000    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:10:16 photon-machine kubelet[3384]: I1210 23:10:16.260744    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:10:26 photon-machine kubelet[3384]: I1210 23:10:26.276827    3384 kubelet_node_status.go:278] Setting node annotation
 to enable volume controller attach/detach
Dec 10 23:10:36 photon-machine kubelet[3384]: I1210 23:10:36.292520    3384 kubelet_node_status.go:278] Setting node ann

root@photon-machine [ ~ ]# cat /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

[neolit123]

CGroup: /system.slice/kubelet.service

it seems that you are missing this file:
https://github.com/kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf

it should come with the kubeadm package.

/priority awaiting-more-evidence
/kind ecosystem

[jeffwubj]

Thanks, I was adding kubelet/kubeadm manually, after adding 10-kubeadm.conf, it works, thanks !

[QiuBiuBiu]

Thanks, I was adding kubelet/kubeadm manually, after adding 10-kubeadm.conf, it works, thanks !

Hi, Could you tell me how do you solve this problem detail? Thinks a lot.

[vroomvroomvroom]

this got me past that error @qiubiubiu
sudo cp /10-kubeadm.conf /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
should have to systemctl daemon-reload

[aayoubi]

Who creates this file? kubeadm?

[neolit123]

the official k8s kubelet package bundles this file.
https://github.com/kubernetes/release/blob/e29fdb0c249a44e9e1cc3f0507de65a770c8bf44/build/rpms/kubelet.spec#L28