What did you do?
Download 1.3.0/1.3.1 and create a cluster
- How was the cluster created?
k3d create
- What did you do afterwards?
run docker logs k3d-k3s-default-server
What did you expect to happen?
k3s-cluster running
Screenshots or terminal output
docker logs k3d-k3s-default-server (ci/sys-ci-gitlab-runner)
time="2019-09-18T08:48:21.697271954Z" level=info msg="Starting k3s v0.7.0 (61bdd852)"
time="2019-09-18T08:48:21.955340430Z" level=info msg="Running kube-apiserver --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --secure-port=6444 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key --service-cluster-ip-range=10.43.0.0/16 --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --allow-privileged=true --authorization-mode=Node,RBAC --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --advertise-port=6443 --insecure-port=0 --service-account-issuer=k3s --api-audiences=unknown --requestheader-allowed-names=system:auth-proxy --requestheader-username-headers=X-Remote-User --bind-address=127.0.0.1 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key"
E0918 08:48:22.309069 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.309339 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.309383 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.309415 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.309439 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.309461 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0918 08:48:22.370905 1 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0918 08:48:22.389340 1 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0918 08:48:22.414819 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.414838 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.414923 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.415255 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.415280 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0918 08:48:22.415297 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
time="2019-09-18T08:48:22.418886151Z" level=info msg="Running kube-scheduler --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false"
time="2019-09-18T08:48:22.419346461Z" level=info msg="Running kube-controller-manager --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --allocate-node-cidrs=true --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --cluster-cidr=10.42.0.0/16 --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --bind-address=127.0.0.1 --leader-elect=false --port=10252 --use-service-account-credentials=true"
W0918 08:48:22.430838 1 authorization.go:47] Authorization is disabled
W0918 08:48:22.430851 1 authentication.go:55] Authentication is disabled
time="2019-09-18T08:48:22.441797147Z" level=info msg="Creating CRD listenerconfigs.k3s.cattle.io"
E0918 08:48:22.452869 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0918 08:48:22.452969 1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0918 08:48:22.453062 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0918 08:48:22.453129 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0918 08:48:22.453194 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
time="2019-09-18T08:48:22.459292497Z" level=info msg="Creating CRD addons.k3s.cattle.io"
E0918 08:48:22.459919 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0918 08:48:22.459998 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0918 08:48:22.460070 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0918 08:48:22.460142 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0918 08:48:22.460529 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
time="2019-09-18T08:48:22.461472784Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
time="2019-09-18T08:48:22.464093298Z" level=info msg="Waiting for CRD listenerconfigs.k3s.cattle.io to become available"
E0918 08:48:22.561369 1 controller.go:147] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIP: Invalid value: "10.43.0.1": cannot allocate resources of type serviceipallocations at this time
E0918 08:48:22.564493 1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.24.0.2, ResourceVersion: 0, AdditionalErrorMsg:
E0918 08:48:22.641945 1 autoregister_controller.go:193] v1.helm.cattle.io failed with : apiservices.apiregistration.k8s.io "v1.helm.cattle.io" already exists
time="2019-09-18T08:48:22.970772223Z" level=info msg="Done waiting for CRD listenerconfigs.k3s.cattle.io to become available"
time="2019-09-18T08:48:22.970854786Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
E0918 08:48:23.461213 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0918 08:48:23.461453 1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0918 08:48:23.461667 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0918 08:48:23.462224 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0918 08:48:23.465203 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0918 08:48:23.468453 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0918 08:48:23.471003 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0918 08:48:23.471003 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0918 08:48:23.474490 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
time="2019-09-18T08:48:23.476536318Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2019-09-18T08:48:23.476721545Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
E0918 08:48:23.479161 1 reflector.go:126] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
time="2019-09-18T08:48:23.985530573Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
time="2019-09-18T08:48:24.024081532Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz"
time="2019-09-18T08:48:24.025276607Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
time="2019-09-18T08:48:24.025982210Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2019-09-18T08:48:24.026349564Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
E0918 08:48:24.027020 1 prometheus.go:138] failed to register depth metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_depth", help: "(Deprecated) Current depth of workqueue: k3s.cattle.io/v1, Kind=Addon", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_depth" is not a valid metric name
E0918 08:48:24.027150 1 prometheus.go:150] failed to register adds metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_adds", help: "(Deprecated) Total number of adds handled by workqueue: k3s.cattle.io/v1, Kind=Addon", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_adds" is not a valid metric name
E0918 08:48:24.027405 1 prometheus.go:162] failed to register latency metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_queue_latency", help: "(Deprecated) How long an item stays in workqueuek3s.cattle.io/v1, Kind=Addon before being requested.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_queue_latency" is not a valid metric name
E0918 08:48:24.027576 1 prometheus.go:174] failed to register work_duration metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_work_duration", help: "(Deprecated) How long processing an item from workqueuek3s.cattle.io/v1, Kind=Addon takes.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_work_duration" is not a valid metric name
E0918 08:48:24.027696 1 prometheus.go:189] failed to register unfinished_work_seconds metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_unfinished_work_seconds", help: "(Deprecated) How many seconds of work k3s.cattle.io/v1, Kind=Addon has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.027804 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for k3s.cattle.io/v1, Kind=Addon been running.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.027980 1 prometheus.go:214] failed to register retries metric k3s.cattle.io/v1, Kind=Addon: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=Addon_retries", help: "(Deprecated) Total number of retries handled by workqueue: k3s.cattle.io/v1, Kind=Addon", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=Addon_retries" is not a valid metric name
time="2019-09-18T08:48:24.041342577Z" level=error msg="Update cert unable to convert string to cert: Unable to split cert into two parts"
time="2019-09-18T08:48:24.041633117Z" level=info msg="Listening on :6443"
E0918 08:48:24.068296 1 prometheus.go:138] failed to register depth metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_depth", help: "(Deprecated) Current depth of workqueue: k3s.cattle.io/v1, Kind=ListenerConfig", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_depth" is not a valid metric name
E0918 08:48:24.068409 1 prometheus.go:150] failed to register adds metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_adds", help: "(Deprecated) Total number of adds handled by workqueue: k3s.cattle.io/v1, Kind=ListenerConfig", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_adds" is not a valid metric name
E0918 08:48:24.068590 1 prometheus.go:162] failed to register latency metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_queue_latency", help: "(Deprecated) How long an item stays in workqueuek3s.cattle.io/v1, Kind=ListenerConfig before being requested.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_queue_latency" is not a valid metric name
E0918 08:48:24.068796 1 prometheus.go:174] failed to register work_duration metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_work_duration", help: "(Deprecated) How long processing an item from workqueuek3s.cattle.io/v1, Kind=ListenerConfig takes.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_work_duration" is not a valid metric name
E0918 08:48:24.069075 1 prometheus.go:189] failed to register unfinished_work_seconds metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_unfinished_work_seconds", help: "(Deprecated) How many seconds of work k3s.cattle.io/v1, Kind=ListenerConfig has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.069254 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for k3s.cattle.io/v1, Kind=ListenerConfig been running.", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.069476 1 prometheus.go:214] failed to register retries metric k3s.cattle.io/v1, Kind=ListenerConfig: descriptor Desc{fqName: "k3s.cattle.io/v1, Kind=ListenerConfig_retries", help: "(Deprecated) Total number of retries handled by workqueue: k3s.cattle.io/v1, Kind=ListenerConfig", constLabels: {}, variableLabels: []} is invalid: "k3s.cattle.io/v1, Kind=ListenerConfig_retries" is not a valid metric name
time="2019-09-18T08:48:24.572455561Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2019-09-18T08:48:24.673385989Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/node-token"
time="2019-09-18T08:48:24.673431239Z" level=info msg="To join node to cluster: k3s agent -s https://172.24.0.2:6443 -t ${NODE_TOKEN}"
time="2019-09-18T08:48:24.674062076Z" level=info msg="Starting k3s.cattle.io/v1, Kind=ListenerConfig controller"
E0918 08:48:24.685763 1 prometheus.go:138] failed to register depth metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_depth", help: "(Deprecated) Current depth of workqueue: /v1, Kind=Node", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_depth" is not a valid metric name
E0918 08:48:24.685849 1 prometheus.go:150] failed to register adds metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_adds", help: "(Deprecated) Total number of adds handled by workqueue: /v1, Kind=Node", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_adds" is not a valid metric name
E0918 08:48:24.685951 1 prometheus.go:162] failed to register latency metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_queue_latency", help: "(Deprecated) How long an item stays in workqueue/v1, Kind=Node before being requested.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_queue_latency" is not a valid metric name
E0918 08:48:24.686071 1 prometheus.go:174] failed to register work_duration metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_work_duration", help: "(Deprecated) How long processing an item from workqueue/v1, Kind=Node takes.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_work_duration" is not a valid metric name
E0918 08:48:24.686131 1 prometheus.go:189] failed to register unfinished_work_seconds metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_unfinished_work_seconds", help: "(Deprecated) How many seconds of work /v1, Kind=Node has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.686186 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for /v1, Kind=Node been running.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.686273 1 prometheus.go:214] failed to register retries metric /v1, Kind=Node: descriptor Desc{fqName: "/v1, Kind=Node_retries", help: "(Deprecated) Total number of retries handled by workqueue: /v1, Kind=Node", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Node_retries" is not a valid metric name
E0918 08:48:24.686582 1 prometheus.go:138] failed to register depth metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_depth", help: "(Deprecated) Current depth of workqueue: batch/v1, Kind=Job", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_depth" is not a valid metric name
E0918 08:48:24.686642 1 prometheus.go:150] failed to register adds metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_adds", help: "(Deprecated) Total number of adds handled by workqueue: batch/v1, Kind=Job", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_adds" is not a valid metric name
E0918 08:48:24.686740 1 prometheus.go:162] failed to register latency metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_queue_latency", help: "(Deprecated) How long an item stays in workqueuebatch/v1, Kind=Job before being requested.", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_queue_latency" is not a valid metric name
E0918 08:48:24.686851 1 prometheus.go:174] failed to register work_duration metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_work_duration", help: "(Deprecated) How long processing an item from workqueuebatch/v1, Kind=Job takes.", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_work_duration" is not a valid metric name
E0918 08:48:24.686913 1 prometheus.go:189] failed to register unfinished_work_seconds metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_unfinished_work_seconds", help: "(Deprecated) How many seconds of work batch/v1, Kind=Job has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.686966 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for batch/v1, Kind=Job been running.", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.687190 1 prometheus.go:214] failed to register retries metric batch/v1, Kind=Job: descriptor Desc{fqName: "batch/v1, Kind=Job_retries", help: "(Deprecated) Total number of retries handled by workqueue: batch/v1, Kind=Job", constLabels: {}, variableLabels: []} is invalid: "batch/v1, Kind=Job_retries" is not a valid metric name
E0918 08:48:24.687398 1 prometheus.go:138] failed to register depth metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_depth", help: "(Deprecated) Current depth of workqueue: helm.cattle.io/v1, Kind=HelmChart", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_depth" is not a valid metric name
E0918 08:48:24.687448 1 prometheus.go:150] failed to register adds metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_adds", help: "(Deprecated) Total number of adds handled by workqueue: helm.cattle.io/v1, Kind=HelmChart", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_adds" is not a valid metric name
E0918 08:48:24.687540 1 prometheus.go:162] failed to register latency metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_queue_latency", help: "(Deprecated) How long an item stays in workqueuehelm.cattle.io/v1, Kind=HelmChart before being requested.", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_queue_latency" is not a valid metric name
E0918 08:48:24.687627 1 prometheus.go:174] failed to register work_duration metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_work_duration", help: "(Deprecated) How long processing an item from workqueuehelm.cattle.io/v1, Kind=HelmChart takes.", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_work_duration" is not a valid metric name
E0918 08:48:24.687687 1 prometheus.go:189] failed to register unfinished_work_seconds metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_unfinished_work_seconds", help: "(Deprecated) How many seconds of work helm.cattle.io/v1, Kind=HelmChart has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.687741 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for helm.cattle.io/v1, Kind=HelmChart been running.", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.687830 1 prometheus.go:214] failed to register retries metric helm.cattle.io/v1, Kind=HelmChart: descriptor Desc{fqName: "helm.cattle.io/v1, Kind=HelmChart_retries", help: "(Deprecated) Total number of retries handled by workqueue: helm.cattle.io/v1, Kind=HelmChart", constLabels: {}, variableLabels: []} is invalid: "helm.cattle.io/v1, Kind=HelmChart_retries" is not a valid metric name
E0918 08:48:24.688140 1 prometheus.go:138] failed to register depth metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_depth", help: "(Deprecated) Current depth of workqueue: /v1, Kind=Service", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_depth" is not a valid metric name
E0918 08:48:24.688258 1 prometheus.go:150] failed to register adds metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_adds", help: "(Deprecated) Total number of adds handled by workqueue: /v1, Kind=Service", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_adds" is not a valid metric name
E0918 08:48:24.688424 1 prometheus.go:162] failed to register latency metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_queue_latency", help: "(Deprecated) How long an item stays in workqueue/v1, Kind=Service before being requested.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_queue_latency" is not a valid metric name
E0918 08:48:24.688665 1 prometheus.go:174] failed to register work_duration metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_work_duration", help: "(Deprecated) How long processing an item from workqueue/v1, Kind=Service takes.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_work_duration" is not a valid metric name
E0918 08:48:24.689028 1 prometheus.go:189] failed to register unfinished_work_seconds metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_unfinished_work_seconds", help: "(Deprecated) How many seconds of work /v1, Kind=Service has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.689317 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for /v1, Kind=Service been running.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.689420 1 prometheus.go:214] failed to register retries metric /v1, Kind=Service: descriptor Desc{fqName: "/v1, Kind=Service_retries", help: "(Deprecated) Total number of retries handled by workqueue: /v1, Kind=Service", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Service_retries" is not a valid metric name
E0918 08:48:24.689654 1 prometheus.go:138] failed to register depth metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_depth", help: "(Deprecated) Current depth of workqueue: /v1, Kind=Pod", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_depth" is not a valid metric name
E0918 08:48:24.689697 1 prometheus.go:150] failed to register adds metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_adds", help: "(Deprecated) Total number of adds handled by workqueue: /v1, Kind=Pod", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_adds" is not a valid metric name
E0918 08:48:24.689807 1 prometheus.go:162] failed to register latency metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_queue_latency", help: "(Deprecated) How long an item stays in workqueue/v1, Kind=Pod before being requested.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_queue_latency" is not a valid metric name
E0918 08:48:24.689927 1 prometheus.go:174] failed to register work_duration metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_work_duration", help: "(Deprecated) How long processing an item from workqueue/v1, Kind=Pod takes.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_work_duration" is not a valid metric name
E0918 08:48:24.689990 1 prometheus.go:189] failed to register unfinished_work_seconds metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_unfinished_work_seconds", help: "(Deprecated) How many seconds of work /v1, Kind=Pod has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.690038 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for /v1, Kind=Pod been running.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.690131 1 prometheus.go:214] failed to register retries metric /v1, Kind=Pod: descriptor Desc{fqName: "/v1, Kind=Pod_retries", help: "(Deprecated) Total number of retries handled by workqueue: /v1, Kind=Pod", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Pod_retries" is not a valid metric name
E0918 08:48:24.690399 1 prometheus.go:138] failed to register depth metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_depth", help: "(Deprecated) Current depth of workqueue: /v1, Kind=Endpoints", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_depth" is not a valid metric name
E0918 08:48:24.690451 1 prometheus.go:150] failed to register adds metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_adds", help: "(Deprecated) Total number of adds handled by workqueue: /v1, Kind=Endpoints", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_adds" is not a valid metric name
E0918 08:48:24.690530 1 prometheus.go:162] failed to register latency metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_queue_latency", help: "(Deprecated) How long an item stays in workqueue/v1, Kind=Endpoints before being requested.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_queue_latency" is not a valid metric name
E0918 08:48:24.690660 1 prometheus.go:174] failed to register work_duration metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_work_duration", help: "(Deprecated) How long processing an item from workqueue/v1, Kind=Endpoints takes.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_work_duration" is not a valid metric name
E0918 08:48:24.690719 1 prometheus.go:189] failed to register unfinished_work_seconds metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_unfinished_work_seconds", help: "(Deprecated) How many seconds of work /v1, Kind=Endpoints has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_unfinished_work_seconds" is not a valid metric name
E0918 08:48:24.690847 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_longest_running_processor_microseconds", help: "(Deprecated) How many microseconds has the longest running processor for /v1, Kind=Endpoints been running.", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_longest_running_processor_microseconds" is not a valid metric name
E0918 08:48:24.691051 1 prometheus.go:214] failed to register retries metric /v1, Kind=Endpoints: descriptor Desc{fqName: "/v1, Kind=Endpoints_retries", help: "(Deprecated) Total number of retries handled by workqueue: /v1, Kind=Endpoints", constLabels: {}, variableLabels: []} is invalid: "/v1, Kind=Endpoints_retries" is not a valid metric name
time="2019-09-18T08:48:24.773469483Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2019-09-18T08:48:24.773485933Z" level=info msg="Run: k3s kubectl"
time="2019-09-18T08:48:24.773491768Z" level=info msg="k3s is up and running"
time="2019-09-18T08:48:24.897788812Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2019-09-18T08:48:24.898331142Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2019-09-18T08:48:24.898800913Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
time="2019-09-18T08:48:25.694307220Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
W0918 08:48:25.849273 1 lease.go:222] Resetting endpoints for master service "kubernetes" to [172.24.0.2]
time="2019-09-18T08:48:25.900457786Z" level=info msg="module br_netfilter was already loaded"
time="2019-09-18T08:48:25.900603127Z" level=info msg="module overlay was already loaded"
time="2019-09-18T08:48:25.900621607Z" level=info msg="module nf_conntrack was already loaded"
time="2019-09-18T08:48:25.908623134Z" level=info msg="Connecting to proxy" url="wss://172.24.0.2:6443/v1-k3s/connect"
time="2019-09-18T08:48:25.909828206Z" level=info msg="Handling backend connection request [k3d-k3s-default-server]"
time="2019-09-18T08:48:25.910693024Z" level=warning msg="Disabling CPU quotas due to missing cpu.cfs_period_us"
time="2019-09-18T08:48:25.910772236Z" level=info msg="Running kubelet --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --seccomp-profile-root=/var/lib/rancher/k3s/agent/kubelet/seccomp --cni-bin-dir=/bin --container-runtime=remote --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --read-only-port=0 --cgroup-driver=cgroupfs --authentication-token-webhook=true --cert-dir=/var/lib/rancher/k3s/agent/kubelet/pki --healthz-bind-address=127.0.0.1 --cluster-domain=cluster.local --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --authorization-mode=Webhook --hostname-override=k3d-k3s-default-server --node-labels=node-role.kubernetes.io/master=true --resolv-conf=/tmp/k3s-resolv.conf --anonymous-auth=false --allow-privileged=true --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --root-dir=/var/lib/rancher/k3s/agent/kubelet --serialize-image-pulls=false --address=0.0.0.0 --fail-swap-on=false --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cluster-dns=10.43.0.10 --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key --cpu-cfs-quota=false"
Flag --allow-privileged has been deprecated, will be removed in a future version
W0918 08:48:25.910977 1 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
W0918 08:48:25.928285 1 options.go:266] in 1.16, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
W0918 08:48:25.928701 1 server.go:216] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
W0918 08:48:25.928948 1 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
W0918 08:48:25.928967 1 options.go:266] in 1.16, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
time="2019-09-18T08:48:25.930629904Z" level=info msg="waiting for node k3d-k3s-default-server: nodes \"k3d-k3s-default-server\" not found"
W0918 08:48:25.934972 1 fs.go:218] stat failed on /dev/mapper/vg-var with error: no such file or directory
W0918 08:48:25.937462 1 info.go:52] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
W0918 08:48:25.941424 1 proxier.go:485] Failed to read file /lib/modules/5.0.0-27-generic/modules.builtin with error open /lib/modules/5.0.0-27-generic/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.944317 1 proxier.go:498] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.944740 1 proxier.go:498] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.945133 1 proxier.go:498] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.945464 1 proxier.go:498] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.945854 1 proxier.go:498] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0918 08:48:25.953622 1 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
W0918 08:48:25.959187 1 node.go:113] Failed to retrieve node info: nodes "k3d-k3s-default-server" not found
W0918 08:48:25.959378 1 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0918 08:48:25.962637 1 fs.go:556] stat failed on /dev/mapper/vg-var with error: no such file or directory
E0918 08:48:25.962809 1 cri_stats_provider.go:375] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs": failed to get device for dir "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs": could not find device with major: 0, minor: 51 in cached partitions map.
E0918 08:48:25.962904 1 kubelet.go:1250] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
E0918 08:48:25.993992 1 controller.go:194] failed to get node "k3d-k3s-default-server" when trying to set owner ref to the node lease: nodes "k3d-k3s-default-server" not found
W0918 08:48:26.013972 1 fs.go:556] stat failed on /dev/mapper/vg-var with error: no such file or directory
F0918 08:48:26.014008 1 kubelet.go:1327] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/rancher/k3s/agent/kubelet": could not find device with major: 0, minor: 51 in cached partitions map
goroutine 7561 [running]:
github.com/rancher/k3s/vendor/k8s.io/klog.stacks(0xc0008ad800, 0xc00345b800, 0xf6, 0xfd)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:828 +0xb1
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).output(0x606aba0, 0xc000000003, 0xc001884af0, 0x5d64655, 0xa, 0x52f, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:779 +0x2d9
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).printf(0x606aba0, 0xc000000003, 0x3476532, 0x23, 0xc003c83d38, 0x1, 0x1)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:678 +0x14e
github.com/rancher/k3s/vendor/k8s.io/klog.Fatalf(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:1209
github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initializeRuntimeDependentModules(0xc004634400)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1327 +0x330
sync.(*Once).Do(0xc004634b28, 0xc00502de70)
/usr/local/go/src/sync/once.go:44 +0xb3
github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp(0xc004634400)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2153 +0x3eb
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc003fd3090)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003fd3090, 0x12a05f200, 0x0, 0x1, 0xc000494060)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc003fd3090, 0x12a05f200, 0xc000494060)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1376 +0x14b
E0918 08:48:26.083617 1 kubelet.go:2207] node "k3d-k3s-default-server" not found
Which OS & Architecture?
Ubuntu 18.04.3 LTS x86_64
Which version of k3d?
1.3.0 + 1.3.1
Which version of docker?
docker version (ci/sys-ci-gitlab-runner)
Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:29:11 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.2
API version: 1.40 (minimum version 1.12)
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:27:45 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
regards,
strowi
strowi
All 12 comments
Hey, thanks for filing this issue.
Apparently, I cannot replicate this issue on my machine and neither in a VM with those exact same versions of Ubuntu and Docker. Anyway... can you please check, which version of k3s is running inside the container?
Also, you may want to cleanup docker to ensure, that there's no hidden issue. (docker system prune -a and maybe docker network prune to clean up leftover mess) and then try to create the cluster again.
iwilltry42
on 19 Sep 2019
thx for getting back on this.
I just cleaned out /ec/docker/daemon.json, restarted docker and ran a docker system prune -af
k3d create (nxt/default)
2019/09/21 15:19:45 Created cluster network with ID bd46b942364d62cfb6627e0cbcf11a643b804f113832d508b944664d1599c8e2
2019/09/21 15:19:45 Created docker volume k3d-k3s-default-images
2019/09/21 15:19:45 Creating cluster [k3s-default]
2019/09/21 15:19:45 Creating server using docker.io/rancher/k3s:v0.7.0...
2019/09/21 15:19:45 Pulling image docker.io/rancher/k3s:v0.7.0...
2019/09/21 15:19:56 SUCCESS: created cluster [k3s-default]
2019/09/21 15:19:56 You can now use the cluster with:
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info
~> docker info (nxt/default)
Client:
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc.)
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.2
Storage Driver: btrfs
Build Version: Btrfs v4.15.1
Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-27-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.444GiB
Name: thor
ID: BFOR:I2UD:RKK4:UFHZ:LMUG:C7K7:HC2Q:J4H4:L5MA:FMLZ:X7ZP:LYWY
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
I tried again, and saw repeatedly errors about "stat failed on /dev/mapper/vg-var" which is on its own btrfs-partition.
/dev/mapper/vg-var /var btrfs noatime,nodiratime,relatime 0 2
Unmounting /var seems to solve the error..
time="2019-09-21T13:31:05.509556763Z" level=info msg="waiting for node k3d-k3s-default-server: nodes \"k3d-k3s-default-server\" not found"
W0921 13:31:05.522676 1 fs.go:218] stat failed on /dev/mapper/vg-var with error: no such file or directory
W0921 13:31:05.531432 1 proxier.go:485] Failed to read file /lib/modules/5.0.0-29-generic/modules.builtin with error open /lib/modules/5.0.0-29-generic/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
...
Ah... you may want to have a look here (k3s) and here: #48
iwilltry42
on 24 Sep 2019
👍1
Oh thx! Didn't consider looking over there since k3s on it's own seemed to work.;)
strowi
on 25 Sep 2019
I seem to have the exact same issue.
default 2m7s Normal NodeHasSufficientMemory node/k3d-k3s-default-worker-0 Node k3d-k3s-default-worker-0 status is
now: NodeHasSufficientMemory
default 2m7s Normal NodeHasNoDiskPressure node/k3d-k3s-default-worker-0 Node k3d-k3s-default-worker-0 status is
now: NodeHasNoDiskPressure
default 2m7s Normal NodeHasSufficientPID node/k3d-k3s-default-worker-0 Node k3d-k3s-default-worker-0 status is
now: NodeHasSufficientPID
default 2m7s Normal Starting node/k3d-k3s-default-server Starting kube-proxy.
default 2m7s Normal Starting node/k3d-k3s-default-server Starting kubelet.
default 2m7s Warning InvalidDiskCapacity node/k3d-k3s-default-server invalid capacity 0 on image filesystem
default 2m7s Normal NodeAllocatableEnforced node/k3d-k3s-default-server Updated Node Allocatable limit across pods
default 2m7s Normal NodeHasSufficientMemory node/k3d-k3s-default-server Node k3d-k3s-default-server status is now: NodeHasSufficientMemory
default 2m7s Normal NodeHasNoDiskPressure node/k3d-k3s-default-server Node k3d-k3s-default-server status is now: NodeHasNoDiskPressure
default 2m7s Normal NodeHasSufficientPID node/k3d-k3s-default-server Node k3d-k3s-default-server status is now: NodeHasSufficientPID
default 2m7s Normal Starting node/k3d-k3s-default-worker-1 Starting kubelet.
default 2m7s Normal Starting node/k3d-k3s-default-worker-1 Starting kube-proxy.
default 2m7s Warning InvalidDiskCapacity node/k3d-k3s-default-worker-1 invalid capacity 0 on image filesystem
default 2m7s Normal NodeAllocatableEnforced node/k3d-k3s-default-worker-1 Updated Node Allocatable limit across pods
default 2m7s Normal NodeHasSufficientMemory node/k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasSufficientMemory
default 2m7s Normal NodeHasNoDiskPressure node/k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasNoDiskPressure
default 2m7s Normal NodeHasSufficientPID node/k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasSufficientPID
default 117s Normal NodeHasDiskPressure node/k3d-k3s-default-worker-0 Node k3d-k3s-default-worker-0 status is now: NodeHasDiskPressure
default 117s Normal NodeHasDiskPressure node/k3d-k3s-default-server Node k3d-k3s-default-server status is now: NodeHasDiskPressure
default 117s Normal NodeHasDiskPressure node/k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasDiskPressure
kube-system 111s Normal SuccessfulCreate job/helm-install-traefik Created pod: helm-install-traefik-48fk4
default 111s Normal RegisteredNode node/k3d-k3s-default-worker-0 Node k3d-k3s-default-worker-0 event: Registered Node k3d-k3s-default-worker-0 in Controller
default 111s Normal RegisteredNode node/k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 event: Registered Node k3d-k3s-default-worker-1 in Controller
default 111s Normal RegisteredNode node/k3d-k3s-default-server Node k3d-k3s-default-server event: Registered Node k3d-k3s-default-server in Controller
kube-system 111s Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-66f496764 to 1
kube-system 111s Normal SuccessfulCreate replicaset/coredns-66f496764 Created pod: coredns-66f496764-447lc
kube-system 41s Warning FailedScheduling pod/coredns-66f496764-447lc 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
kube-system 41s Warning FailedScheduling pod/helm-install-traefik-48fk4 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
default 8s Warning EvictionThresholdMet node/k3d-k3s-default-worker-0 Attempting to reclaim ephemeral-storage
default 7s Warning EvictionThresholdMet node/k3d-k3s-default-server Attempting to reclaim ephemeral-storage
default 7s Warning EvictionThresholdMet node/k3d-k3s-default-worker-1 Attempting to reclaim ephemeral-storage
default 0s Warning EvictionThresholdMet node/k3d-k3s-default-worker-0 Attempting to reclaim ephemeral-storage
default 0s Warning EvictionThresholdMet node/k3d-k3s-default-server Attempting to reclaim ephemeral-storage
default 0s Warning EvictionThresholdMet node/k3d-k3s-default-worker-1 Attempting to reclaim ephemeral-storage
kube-system 0s Warning FailedSchedulin
And:
Name: k3d-k3s-default-worker-1
Roles: worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=k3d-k3s-default-worker-1
kubernetes.io/os=linux
node-role.kubernetes.io/worker=true
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"96:22:d4:95:54:d4"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.22.0.4
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 10 Oct 2019 16:28:18 +0100
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 10 Oct 2019 16:31:28 +0100 Thu, 10 Oct 2019 16:28:18 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure True Thu, 10 Oct 2019 16:31:28 +0100 Thu, 10 Oct 2019 16:28:28 +0100 KubeletHasDiskPressure kubelet has disk pressure
PIDPressure False Thu, 10 Oct 2019 16:31:28 +0100 Thu, 10 Oct 2019 16:28:18 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 10 Oct 2019 16:31:28 +0100 Thu, 10 Oct 2019 16:28:18 +0100 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.22.0.4
Hostname: k3d-k3s-default-worker-1
Capacity:
cpu: 6
ephemeral-storage: 61255492Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 6092540Ki
pods: 110
Allocatable:
cpu: 6
ephemeral-storage: 59589342571
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 6092540Ki
pods: 110
System Info:
Machine ID:
System UUID: D7CA460F-0000-0000-8AD3-4F690095321A
Boot ID: 3ee9fd3a-1478-46d4-bfb1-e2c749cf96fa
Kernel Version: 4.14.131-linuxkit
OS Image: Unknown
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.2.8-k3s.1
Kubelet Version: v1.15.4-k3s.1
Kube-Proxy Version: v1.15.4-k3s.1
PodCIDR: 10.42.2.0/24
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 3m25s kubelet, k3d-k3s-default-worker-1 Starting kubelet.
Normal Starting 3m25s kube-proxy, k3d-k3s-default-worker-1 Starting kube-proxy.
Warning InvalidDiskCapacity 3m25s kubelet, k3d-k3s-default-worker-1 invalid capacity 0 on image filesystem
Normal NodeAllocatableEnforced 3m25s kubelet, k3d-k3s-default-worker-1 Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 3m25s (x2 over 3m25s) kubelet, k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3m25s (x2 over 3m25s) kubelet, k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3m25s (x2 over 3m25s) kubelet, k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasSufficientPID
Normal NodeHasDiskPressure 3m15s kubelet, k3d-k3s-default-worker-1 Node k3d-k3s-default-worker-1 status is now: NodeHasDiskPressure
Warning EvictionThresholdMet 75s (x13 over 3m15s) kubelet, k3d-k3s-default-worker-1 Attempting to reclaim ephemeral-storage
Any ideas? @iwilltry42 @zeerorg
alexellis
on 10 Oct 2019
Hi @alexellis, which version of k3d and k3s are you using there?
Also, could you please as well post the docker logs of the node?
iwilltry42
on 11 Oct 2019
I ran into this myself today.. a docker system prune -a helped.. not really sure, what's the issue there, but seems to be related to docker itself rather than k3d in particular :thinking:
iwilltry42
on 18 Oct 2019
👍1
That is what fixed it for me too
alexellis
on 18 Oct 2019
What ideas do you have for improving error handling?
alexellis
on 18 Oct 2019
I guess we're having the same issue as kind: https://kind.sigs.k8s.io/docs/user/known-issues/#failure-for-cluster-to-properly-start
I'll look into the way that docker handles the space for containers and how we can keep the kubelet from evicting pods (changing diskpressure threshold).
And add some docs. All of this, when I finally have enough time :+1:
Any more insights are very welcome :)
iwilltry42
on 18 Oct 2019
So I experienced the same issue when I created a 4 node cluster on my dev laptop.
I deployed the elastic-cloud operator and created a 3 node ElasticSearch cluster bound to a single node.
Then, when I deployed Kibana, all pods (also on other nodes) were evicted because ofNodeHasDiskPressure.
df -hinside the nodes and on my laptop revealed 95% usage on/.
I deleted some stuff from my device and ended up with around 70% disk usage on my laptop.
Afterwards, I didn't have anymore problems with evicted nodes because of DiskPressure.So I guess, that it's really just caused by the device disk usage... in my case at least :thinking:
Note: Docker on Linux for me. In Docker for Mac, one might need to consider the Docker VM disk size as well.
https://github.com/rancher/k3d/issues/119#issuecomment-544280733
iwilltry42
on 20 Oct 2019
@strowi , do you still experience this issue?
iwilltry42
on 24 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
kafai97
路
3Comments
jwhb
路
3Comments
dtomcej
路
5Comments
stephenc
路
4Comments
blaggacao
路
5Comments