K3s: rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs
I am trying to start k3s in rhel7. The service starts fine but no pods get deployed. Looks like some issues in containerd as shown in the log.
I suspect this warning in containerd is causing the issue:
failed to create CRI service: failed to find snapshotter \"overlayfs\"
System: rhel7.6
Trying to install k3s in air gap mode.
File contents:
./
- k3s
- k3s-airgap-images-amd64.tar
#./k3s server
...
INFO[2019-05-23T07:52:35.466217824-07:00] Run: k3s kubectl
INFO[2019-05-23T07:52:35.466227353-07:00] k3s is up and running
INFO[2019-05-23T07:52:35.513838186-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-23T07:52:35.513952421-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-23T07:52:35.514177382-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-23T07:52:36.515150846-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T07:52:37.515878061-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
E0523 07:52:37.557634 32117 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs"]
INFO[2019-05-23T07:52:38.516591198-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T07:52:39.517415316-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
...
The last line repeats indefinitely.
containerd.log:
time="2019-05-23T07:51:52.904216373-07:00" level=info msg="Connect containerd service"
time="2019-05-23T07:51:52.904375778-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""
time="2019-05-23T07:51:52.904402647-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2019-05-23T07:51:52.904679823-07:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2019-05-23T07:51:52.904699156-07:00" level=info msg="containerd successfully booted in 0.003223s"
Note:
- firewalld disabled
- host IP resolved to server name in /etc/hosts
orhel
rako1980
👍1
All 12 comments
Is the overlay kernel module available? (output for lsmod | grep overlay)
Anything useful in the logs leading up to k3s is up and running?
erikwilson
on 23 May 2019
@erikwilson
# lsmod | grep over
overlay 71964 0
Yes k3s service is running. Here is the status
[root@xxxxxx ~]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-05-23 09:52:07 PDT; 10min ago
Docs: https://k3s.io
Process: 40305 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 40303 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Main PID: 40308 (k3s-server)
Tasks: 30
Memory: 102.8M
CGroup: /system.slice/k3s.service
├─40308 /usr/local/bin/k3s server
└─40331 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var...
May 23 10:02:41 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:41.038711533-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:42 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:42.039755725-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:43 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:43.041077281-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:44 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:44.042223950-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:45 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:45.043232193-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:46 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:46.044413974-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:47 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:47.045353133-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:48 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:48.046353999-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:49 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:49.047118545-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:50 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:50.048046848-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
Hint: Some lines were ellipsized, use -l to show in full.
# k3s kubectl get all --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 16m
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 16m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 0/1 0 0 16m
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik 0/1 16m
Here is the detail of the log when I start as #./k3s server
# ./k3s server
INFO[2019-05-23T10:41:15.365566723-07:00] Starting k3s v0.5.0 (8c0116dd)
INFO[2019-05-23T10:41:15.366560494-07:00] Running kube-apiserver --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-username-headers=X-Remote-User --allow-privileged=true --advertise-address=127.0.0.1 --tls-cert-file=/var/lib/rancher/k3s/server/tls/localhost.crt --insecure-port=0 --bind-address=127.0.0.1 --requestheader-extra-headers-prefix=X-Remote-Extra- --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --advertise-port=6445 --service-account-issuer=k3s --api-audiences=unknown --kubelet-client-key=/var/lib/rancher/k3s/server/tls/token-node.key --requestheader-allowed-names=kubernetes-proxy --requestheader-group-headers=X-Remote-Group --watch-cache=false --authorization-mode=Node,RBAC --tls-private-key-file=/var/lib/rancher/k3s/server/tls/localhost.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --secure-port=6444 --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/token-node-1.crt
E0523 10:41:15.370664 47043 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.370946 47043 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.370990 47043 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371025 47043 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371056 47043 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371083 47043 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0523 10:41:15.419277 47043 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0523 10:41:15.426893 47043 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0523 10:41:15.447519 47043 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447559 47043 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447605 47043 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447680 47043 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447715 47043 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447743 47043 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
INFO[2019-05-23T10:41:15.451764805-07:00] Running kube-scheduler --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --leader-elect=false
INFO[2019-05-23T10:41:15.452138189-07:00] Running kube-controller-manager --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --cluster-cidr=10.42.0.0/16 --root-ca-file=/var/lib/rancher/k3s/server/tls/token-ca.crt --leader-elect=false --allocate-node-cidrs=true --port=10252 --bind-address=127.0.0.1 --secure-port=0
W0523 10:41:15.464384 47043 authorization.go:47] Authorization is disabled
W0523 10:41:15.464410 47043 authentication.go:55] Authentication is disabled
INFO[2019-05-23T10:41:15.531532252-07:00] Listening on :6443
E0523 10:41:15.538233 47043 controller.go:148] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
INFO[2019-05-23T10:41:15.632791038-07:00] Node token is available at /var/lib/rancher/k3s/server/node-token
INFO[2019-05-23T10:41:15.632821655-07:00] To join node to cluster: k3s agent -s https://10.103.102.48:6443 -t ${NODE_TOKEN}
INFO[2019-05-23T10:41:15.633755733-07:00] Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz
INFO[2019-05-23T10:41:15.634040227-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml
INFO[2019-05-23T10:41:15.634192136-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml
INFO[2019-05-23T10:41:15.681774803-07:00] Wrote kubeconfig /etc/rancher/k3s/k3s.yaml
INFO[2019-05-23T10:41:15.681799980-07:00] Run: k3s kubectl
INFO[2019-05-23T10:41:15.681807644-07:00] k3s is up and running
INFO[2019-05-23T10:41:15.734059148-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-23T10:41:15.734193744-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-23T10:41:15.734413260-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-23T10:41:16.736592830-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
W0523 10:41:17.678890 47043 controllermanager.go:445] Skipping "csrsigning"
INFO[2019-05-23T10:41:17.737374016-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:18.738180671-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:19.739156698-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:20.740059356-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:21.740946864-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:22.741856234-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:23.742698558-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
^CINFO[2019-05-23T10:41:23.998087128-07:00] Received SIGTERM, cancelling
FATA[2019-05-23T10:41:23.998308827-07:00] context canceled
Because of SELinux?
warmchang
on 24 May 2019
No SELinux:
# getenforce
Disabled
I was able to make some progress with docker though.
#./k3s server --docker
But, needed to start the docker native.cgroupdriver with cgroupfs instead of systemd:
# cat /etc/systemd/system/multi-user.target.wants/docker.service | grep native.cgroupdriver
--exec-opt native.cgroupdriver=cgroupfs \
However, my intention is to use the containerd instead of docker. I still assume there is some issue with the spawned contained running from k3s.
Here is the details k3s stdout and the corresponding containerd.log:
]# ./k3s server
INFO[2019-05-24T08:36:49.648912110-07:00] Starting k3s v0.5.0 (8c0116dd)
INFO[2019-05-24T08:36:49.650162620-07:00] Running kube-apiserver --advertise-port=6445 --api-audiences=unknown --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/token-node-1.crt --requestheader-group-headers=X-Remote-Group --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --service-cluster-ip-range=10.43.0.0/16 --bind-address=127.0.0.1 --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-allowed-names=kubernetes-proxy --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --authorization-mode=Node,RBAC --advertise-address=127.0.0.1 --insecure-port=0 --tls-cert-file=/var/lib/rancher/k3s/server/tls/localhost.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/localhost.key --service-account-issuer=k3s --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --requestheader-username-headers=X-Remote-User --watch-cache=false --allow-privileged=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --secure-port=6444 --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --kubelet-client-key=/var/lib/rancher/k3s/server/tls/token-node.key
E0524 08:36:49.654488 79239 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657296 79239 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657381 79239 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657421 79239 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657454 79239 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657484 79239 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0524 08:36:49.702595 79239 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0524 08:36:49.724691 79239 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0524 08:36:49.756678 79239 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756724 79239 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756779 79239 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756833 79239 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756869 79239 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756899 79239 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
INFO[2019-05-24T08:36:49.761231703-07:00] Running kube-scheduler --leader-elect=false --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml
INFO[2019-05-24T08:36:49.761597686-07:00] Running kube-controller-manager --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --cluster-cidr=10.42.0.0/16 --root-ca-file=/var/lib/rancher/k3s/server/tls/token-ca.crt --leader-elect=false --allocate-node-cidrs=true --port=10252 --bind-address=127.0.0.1 --secure-port=0
W0524 08:36:49.776390 79239 authorization.go:47] Authorization is disabled
W0524 08:36:49.776420 79239 authentication.go:55] Authentication is disabled
INFO[2019-05-24T08:36:49.837533400-07:00] Listening on :6443
E0524 08:36:49.838660 79239 controller.go:148] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
INFO[2019-05-24T08:36:49.939031633-07:00] Node token is available at /var/lib/rancher/k3s/server/node-token
INFO[2019-05-24T08:36:49.939066279-07:00] To join node to cluster: k3s agent -s https://10.103.102.48:6443 -t ${NODE_TOKEN}
INFO[2019-05-24T08:36:49.943090685-07:00] Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz
INFO[2019-05-24T08:36:49.943421808-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml
INFO[2019-05-24T08:36:49.944462948-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml
INFO[2019-05-24T08:36:50.004464688-07:00] Wrote kubeconfig /etc/rancher/k3s/k3s.yaml
INFO[2019-05-24T08:36:50.004506548-07:00] Run: k3s kubectl
INFO[2019-05-24T08:36:50.004523123-07:00] k3s is up and running
INFO[2019-05-24T08:36:50.064578703-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-24T08:36:50.064952262-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-24T08:36:50.065173135-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-24T08:36:51.066282525-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:52.067114633-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:53.068189094-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:54.069177923-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:55.070198945-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:56.071389818-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:57.072397121-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:58.075790324-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:59.077008601-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:00.077919013-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:01.079128839-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:02.080065210-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
W0524 08:37:02.280312 79239 controllermanager.go:445] Skipping "root-ca-cert-publisher"
W0524 08:37:02.282988 79239 controllermanager.go:445] Skipping "csrsigning"
W0524 08:37:02.434796 79239 shared_informer.go:312] resyncPeriod 49515231569317 is smaller than resyncCheckPeriod 75729967585759 and the informer has already started. Changing it to 75729967585759
W0524 08:37:02.435431 79239 shared_informer.go:312] resyncPeriod 55270902515792 is smaller than resyncCheckPeriod 75729967585759 and the informer has already started. Changing it to 75729967585759
E0524 08:37:02.435703 79239 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts"]
INFO[2019-05-24T08:37:03.080919525-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:04.081925505-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:05.083064795-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:06.084115511-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:07.085146046-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
^CINFO[2019-05-24T08:37:07.425870802-07:00] Received SIGTERM, cancelling
FATA[2019-05-24T08:37:07.426526098-07:00] context canceled
/var/lib/rancher/k3s/agent/containerd/containerd.log
time="2019-05-24T08:36:50.131372621-07:00" level=info msg="starting containerd" revision= version=1.2.5+unknown
time="2019-05-24T08:36:50.132169525-07:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2019-05-24T08:36:50.132221714-07:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2019-05-24T08:36:50.132267099-07:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2019-05-24T08:36:50.132487035-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
time="2019-05-24T08:36:50.132524711-07:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2019-05-24T08:36:50.132548844-07:00" level=warning msg="could not use snapshotter overlayfs in metadata plugin" error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
time="2019-05-24T08:36:50.132842127-07:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2019-05-24T08:36:50.132953494-07:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2019-05-24T08:36:50.133036536-07:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133082568-07:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133120453-07:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133153366-07:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133172447-07:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133193487-07:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133217195-07:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133238861-07:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2019-05-24T08:36:50.133300940-07:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2019-05-24T08:36:50.133361641-07:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2019-05-24T08:36:50.133891262-07:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133946642-07:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2019-05-24T08:36:50.134023974-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134082887-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134114423-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134151760-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134175442-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134197101-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134216812-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134233085-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134252815-07:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2019-05-24T08:36:50.134336176-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134363697-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134380869-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134399399-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134528786-07:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin NetworkPluginConfDir:/var/lib/rancher/k3s/agent/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:d53icpdev001t0x StreamServerPort:10010 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
time="2019-05-24T08:36:50.134599727-07:00" level=info msg="Connect containerd service"
time="2019-05-24T08:36:50.134834052-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""
time="2019-05-24T08:36:50.134863257-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.135193569-07:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2019-05-24T08:36:50.135223048-07:00" level=info msg="containerd successfully booted in 0.004611s"
AFAICT this is caused by
WARN[2019-05-27T22:53:37.502861403+02:00] could not use snapshotter overlayfs in metadata plugin error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
aucampia
on 27 May 2019
I had this issue and I solved it by using a xfs filesystem created with:
mkfs -t xfs -n ftype=1 -L LABEL -f DEVICE
Some details here: https://docs.docker.com/storage/storagedriver/overlayfs-driver/
aucampia
on 28 May 2019
Thanks a bunch @aucampia . That is the the issue. Later versions of RHEL7 mkfs the new filesystem with default fstype=1. However, I was using the old rhel7 version from template where /var was already formatted. Had to create a new xfs filesystem /var/lib/rancher where the new ftype setting is automatically formatted with fstype 1, and there is no longer overlayfs issue with containerd. My coredns and traefik pods and now running :)
rako1980
on 28 May 2019
👍3
I had this issue and I solved it by using a xfs filesystem created with:
mkfs -t xfs -n ftype=1 -L LABEL -f DEVICESome details here: https://docs.docker.com/storage/storagedriver/overlayfs-driver/
Thanks , it solved my problem also. I'm running K3S on a Oracle Linux Server release 7.5 VM and now it is functioning with containerd.
jeroavf
on 15 Aug 2019
👍2
I was also eaten by this trying to run on a much older file-system
xarses
on 25 Nov 2019
Closing due to age
brandond
on 5 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
wpwoodjr
·
3Comments
davidnuzik
·
3Comments
seanmalloy
·
3Comments
weber-software
·
3Comments
davidnuzik
·
3Comments
Most helpful comment
Thanks a bunch @aucampia . That is the the issue. Later versions of RHEL7 mkfs the new filesystem with default fstype=1. However, I was using the old rhel7 version from template where /var was already formatted. Had to create a new xfs filesystem /var/lib/rancher where the new ftype setting is automatically formatted with fstype 1, and there is no longer overlayfs issue with containerd. My coredns and traefik pods and now running :)