Minikube: Using containers[i].volumeMounts[j].subPath produces "no such file or directory" error
Description
Using containers[i].volumeMounts[j].subPath produces no such file or directory errors.
$ kubectl describe pods -n my-namespace my-job-bad-6962x
:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m4s default-scheduler Successfully assigned my-namespace/my-job-bad-6962x to minikube
Normal Pulled 3m27s (x8 over 4m48s) kubelet, minikube Successfully pulled image "docker.io/centos:latest"
Warning Failed 3m27s (x8 over 4m48s) kubelet, minikube Error: stat /opt/my-path/: no such file or directory
Oddly, when a pod that is run without subPath, not only does it work, but it also initializes something that allows a Pod with subPath to work. There's an initialization problem in 'minikube' somewhere.
Steps to reproduce the issue:
- Start MiniKube
$ minikube start --cpus 4 --memory 8192 --vm-driver kvm2
😄 minikube v1.2.0 on linux (amd64)
🔥 Creating kvm2 VM (CPUs=4, Memory=8192MB, Disk=20000MB) ...
🐳 Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
🚜 Pulling images ...
🚀 Launching Kubernetes ...
⌛ Verifying: apiserver proxy etcd scheduler controller dns
🏄 Done! kubectl is now configured to use "minikube"
- Create
my-namespace.yamlfile:
cat <<EOT > my-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: my-namespace
labels:
name: my-namespace
EOT
- Create namespace:
kubectl create -f my-namespace.yaml
- Create
my-persistent-volume.yamlfile:
cat <<EOT > my-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-persistent-volume
labels:
type: local
namespace: my-namespace
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: "/opt/my-path/"
EOT
- Create persistent volume:
kubectl create -f my-persistent-volume.yaml
- Create
my-persistent-volume-claim.yamlfile:
cat <<EOT > my-persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
finalizers:
- kubernetes.io/pvc-protection
labels:
cattle.io/creator: norman
name: my-persistent-volume-claim
namespace: my-namespace
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: "manual"
volumeName: my-persistent-volume
EOT
- Create persistent volume claim:
kubectl create -f my-persistent-volume-claim.yaml
- Create
my-job-bad.yamlfile:
cat <<EOT > my-job-bad.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: my-job-bad
namespace: my-namespace
spec:
template:
spec:
containers:
- name: subpath-test
image: docker.io/centos:latest
imagePullPolicy: Always
command: ["sleep"]
args: ["infinity"]
volumeMounts:
- name: my-volume
mountPath: /opt/my-subpath
subPath: my-subpath-1
restartPolicy: Never
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-persistent-volume-claim
EOT
- Create job with
subPaththat fails:
kubectl create -f my-job-bad.yaml
- Watch for error.
$ kubectl get pods --namespace my-namespace --watch
NAME READY STATUS RESTARTS AGE
my-job-bad-6962x 0/1 ContainerCreating 0 11s
my-job-bad-6962x 0/1 CreateContainerConfigError 0 77s
- Create
my-job-good.yamlfile:
cat <<EOT > my-job-good.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: my-job-good
namespace: my-namespace
spec:
template:
spec:
containers:
- name: subpath-test
image: docker.io/centos:latest
imagePullPolicy: Always
command: ["sleep"]
args: ["infinity"]
volumeMounts:
- name: my-volume
mountPath: /opt/my-subpath
restartPolicy: Never
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-persistent-volume-claim
EOT
- Create job without
subPaththat succeeds:
kubectl create -f my-job-good.yaml
- Now both the "good job" and the "bad job" work.
$ kubectl get pods --namespace my-namespace --watch
NAME READY STATUS RESTARTS AGE
my-job-bad-6962x 1/1 Running 0 4m25s
my-job-good-dlfxn 1/1 Running 0 12s
Describe the results you received:
- View error in
my-job-bad.
$ kubectl describe pods -n my-namespace my-job-bad-6962x
Name: my-job-bad-6962x
Namespace: my-namespace
Priority: 0
PriorityClassName: <none>
Node: minikube/192.168.122.59
Start Time: Fri, 28 Jun 2019 16:36:29 -0400
Labels: controller-uid=cdc5f87b-9c59-4f58-94d5-d286f7597d65
job-name=my-job-bad
Annotations: <none>
Status: Running
IP: 172.17.0.4
Controlled By: Job/my-job-bad
Containers:
subpath-test:
Container ID: docker://965ad24defc7d2364982d9c7c5e8a5efa9293578be3b7cb7ef80cfe6e8ab3128
Image: docker.io/centos:latest
Image ID: docker-pullable://centos@sha256:b5e66c4651870a1ad435cd75922fe2cb943c9e973a9673822d1414824a1d0475
Port: <none>
Host Port: <none>
Command:
sleep
Args:
infinity
State: Running
Started: Fri, 28 Jun 2019 16:40:48 -0400
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/opt/my-subpath from my-volume (rw,path="my-subpath-1")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wrmc5 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
my-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-persistent-volume-claim
ReadOnly: false
default-token-wrmc5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wrmc5
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m4s default-scheduler Successfully assigned my-namespace/my-job-bad-6962x to minikube
Normal Pulled 3m27s (x8 over 4m48s) kubelet, minikube Successfully pulled image "docker.io/centos:latest"
Warning Failed 3m27s (x8 over 4m48s) kubelet, minikube Error: stat /opt/my-path/: no such file or directory
Normal Pulling 3m16s (x9 over 6m3s) kubelet, minikube Pulling image "docker.io/centos:latest"
Describe the results you expected:
The Pod containing containers[i].volumeMounts[j].subPath should come up without
the necessity of a Pod without subPath initializing "something".
Additional information you deem important (e.g. issue happens only occasionally):
As seen, when running without subPath, the Pod comes up properly.
My guess is that when subPath is used, an initialization step is missing.
Version of Kubernetes:
- Output of
kubectl version:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
- Output of
minikube:
$ minikube version
minikube version: v1.2.0
Cleanup
kubectl delete -f my-job-good.yaml
kubectl delete -f my-job-bad.yaml
kubectl delete -f my-persistent-volume-claim.yaml
kubectl delete -f my-persistent-volume.yaml
kubectl delete -f my-namespace.yaml
minikube stop
minikube delete
The output of the minikube logs command:
$ minikube logs
==> coredns <==
.:53
2019-06-28T20:31:33.762Z [INFO] CoreDNS-1.3.1
2019-06-28T20:31:33.762Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-06-28T20:31:33.762Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
==> dmesg <==
[Jun28 20:29] APIC calibration not consistent with PM-Timer: 106ms instead of 100ms
[ +0.000000] core: CPUID marked event: 'bus cycles' unavailable
[ +0.001021] #2
[ +0.001080] #3
[ +0.022772] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ +0.118421] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ +21.645559] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
[ +0.025752] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ +0.025602] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[ +0.230451] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ +0.047506] systemd-fstab-generator[1109]: Ignoring "noauto" for root device
[ +0.006948] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:35 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[ +0.000004] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ +0.614842] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ +0.769835] vboxguest: loading out-of-tree module taints kernel.
[ +0.003883] vboxguest: PCI device not found, probably running on physical hardware.
[ +7.041667] systemd-fstab-generator[1986]: Ignoring "noauto" for root device
[Jun28 20:30] systemd-fstab-generator[2741]: Ignoring "noauto" for root device
[ +9.291779] systemd-fstab-generator[2990]: Ignoring "noauto" for root device
[Jun28 20:31] kauditd_printk_skb: 68 callbacks suppressed
[ +13.855634] tee (3708): /proc/3426/oom_adj is deprecated, please use /proc/3426/oom_score_adj instead.
[ +7.361978] kauditd_printk_skb: 20 callbacks suppressed
[ +6.562157] kauditd_printk_skb: 47 callbacks suppressed
[ +3.921583] NFSD: Unable to end grace period: -110
==> kernel <==
21:02:00 up 32 min, 0 users, load average: 0.33, 0.35, 0.34
Linux minikube 4.15.0 #1 SMP Sun Jun 23 23:02:01 PDT 2019 x86_64 GNU/Linux
==> kube-addon-manager <==
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:54:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T20:55:30+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:55:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T20:56:30+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:56:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T20:57:30+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:57:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T20:58:31+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:58:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T20:59:31+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
error: no objects passed to apply
error: no objects passed to apply
INFO: == Kubernetes addon reconcile completed at 2019-06-28T20:59:33+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T21:00:30+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T21:00:32+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-06-28T21:01:30+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-06-28T21:01:32+00:00 ==
==> kube-apiserver <==
I0628 20:31:16.446365 1 client.go:354] scheme "" not registered, fallback to default scheme
I0628 20:31:16.446451 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0628 20:31:16.446513 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0628 20:31:16.463319 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0628 20:31:16.464540 1 client.go:354] parsed scheme: ""
I0628 20:31:16.464624 1 client.go:354] scheme "" not registered, fallback to default scheme
I0628 20:31:16.464699 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0628 20:31:16.464799 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0628 20:31:16.479368 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0628 20:31:19.006900 1 secure_serving.go:116] Serving securely on [::]:8443
I0628 20:31:19.007039 1 available_controller.go:374] Starting AvailableConditionController
I0628 20:31:19.007127 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0628 20:31:19.007616 1 crd_finalizer.go:255] Starting CRDFinalizer
I0628 20:31:19.007724 1 autoregister_controller.go:140] Starting autoregister controller
I0628 20:31:19.007821 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0628 20:31:19.007995 1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0628 20:31:19.008035 1 controller_utils.go:1029] Waiting for caches to sync for crd-autoregister controller
I0628 20:31:19.008720 1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0628 20:31:19.008768 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0628 20:31:19.009739 1 controller.go:83] Starting OpenAPI controller
I0628 20:31:19.009854 1 customresource_discovery_controller.go:208] Starting DiscoveryController
I0628 20:31:19.009924 1 naming_controller.go:288] Starting NamingConditionController
I0628 20:31:19.010007 1 establishing_controller.go:73] Starting EstablishingController
I0628 20:31:19.010074 1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController
E0628 20:31:19.010957 1 controller.go:148] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.39.223, ResourceVersion: 0, AdditionalErrorMsg:
I0628 20:31:19.011694 1 controller.go:81] Starting OpenAPI AggregationController
I0628 20:31:19.116245 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0628 20:31:19.116318 1 cache.go:39] Caches are synced for autoregister controller
I0628 20:31:19.207447 1 cache.go:39] Caches are synced for AvailableConditionController controller
I0628 20:31:19.208523 1 controller_utils.go:1036] Caches are synced for crd-autoregister controller
I0628 20:31:20.004685 1 controller.go:107] OpenAPI AggregationController: Processing item
I0628 20:31:20.004761 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0628 20:31:20.004891 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0628 20:31:20.021255 1 storage_scheduling.go:119] created PriorityClass system-node-critical with value 2000001000
I0628 20:31:20.027177 1 storage_scheduling.go:119] created PriorityClass system-cluster-critical with value 2000000000
I0628 20:31:20.027214 1 storage_scheduling.go:128] all system priority classes are created successfully or already exist.
I0628 20:31:21.789899 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0628 20:31:22.069843 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0628 20:31:22.085456 1 controller.go:606] quota admission added evaluator for: endpoints
W0628 20:31:22.375007 1 lease.go:223] Resetting endpoints for master service "kubernetes" to [192.168.39.223]
I0628 20:31:22.428889 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0628 20:31:22.872788 1 controller.go:606] quota admission added evaluator for: namespaces
I0628 20:31:23.516372 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0628 20:31:23.843781 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0628 20:31:24.145272 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0628 20:31:30.267937 1 controller.go:606] quota admission added evaluator for: replicasets.apps
I0628 20:31:30.368017 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I0628 20:36:29.652258 1 controller.go:606] quota admission added evaluator for: jobs.batch
E0628 20:46:26.034713 1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0628 20:59:47.138291 1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
==> kube-proxy <==
W0628 20:31:31.469387 1 server_others.go:249] Flag proxy-mode="" unknown, assuming iptables proxy
I0628 20:31:31.486180 1 server_others.go:143] Using iptables Proxier.
W0628 20:31:31.486736 1 proxier.go:321] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0628 20:31:31.487461 1 server.go:534] Version: v1.15.0
I0628 20:31:31.506151 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0628 20:31:31.506240 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0628 20:31:31.506405 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0628 20:31:31.506668 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0628 20:31:31.507332 1 config.go:96] Starting endpoints config controller
I0628 20:31:31.507381 1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I0628 20:31:31.507608 1 config.go:187] Starting service config controller
I0628 20:31:31.507653 1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I0628 20:31:31.607874 1 controller_utils.go:1036] Caches are synced for service config controller
I0628 20:31:31.607905 1 controller_utils.go:1036] Caches are synced for endpoints config controller
==> kube-scheduler <==
I0628 20:31:13.084899 1 serving.go:319] Generated self-signed cert in-memory
W0628 20:31:14.157003 1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0628 20:31:14.157134 1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0628 20:31:14.157255 1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0628 20:31:14.162595 1 server.go:142] Version: v1.15.0
I0628 20:31:14.162743 1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0628 20:31:14.164519 1 authorization.go:47] Authorization is disabled
W0628 20:31:14.164559 1 authentication.go:55] Authentication is disabled
I0628 20:31:14.164685 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0628 20:31:14.170555 1 secure_serving.go:116] Serving securely on 127.0.0.1:10259
E0628 20:31:19.135466 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0628 20:31:19.195942 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0628 20:31:19.196169 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0628 20:31:19.196451 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0628 20:31:19.196823 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0628 20:31:19.203454 1 reflector.go:125] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:226: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0628 20:31:19.203884 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0628 20:31:19.206488 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0628 20:31:19.206807 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0628 20:31:19.206488 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0628 20:31:20.141756 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0628 20:31:20.198193 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0628 20:31:20.199220 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0628 20:31:20.206555 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0628 20:31:20.207831 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0628 20:31:20.213491 1 reflector.go:125] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:226: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0628 20:31:20.213838 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0628 20:31:20.213983 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0628 20:31:20.217183 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0628 20:31:20.217406 1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
I0628 20:31:22.077586 1 leaderelection.go:235] attempting to acquire leader lease kube-system/kube-scheduler...
I0628 20:31:22.088791 1 leaderelection.go:245] successfully acquired lease kube-system/kube-scheduler
E0628 20:31:30.321683 1 factory.go:702] pod is already present in the activeQ
==> kubelet <==
-- Logs begin at Fri 2019-06-28 20:29:38 UTC, end at Fri 2019-06-28 21:02:00 UTC. --
Jun 28 20:31:20 minikube kubelet[3010]: E0628 20:31:20.146241 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee402e59e", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c11d99e, ext:3375456312, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a962c502, ext:3598863731, loc:(*time.Location)(0x781d740)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:20 minikube kubelet[3010]: E0628 20:31:20.547212 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee4031da5", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c1211a5, ext:3375470655, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a9633094, ext:3598891777, loc:(*time.Location)(0x781d740)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:20 minikube kubelet[3010]: E0628 20:31:20.948190 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee402603f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c11543f, ext:3375422170, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a9c2709a, ext:3605134986, loc:(*time.Location)(0x781d740)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:21 minikube kubelet[3010]: E0628 20:31:21.347536 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee402e59e", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c11d99e, ext:3375456312, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a9c39b54, ext:3605210674, loc:(*time.Location)(0x781d740)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:21 minikube kubelet[3010]: E0628 20:31:21.746377 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee4031da5", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c1211a5, ext:3375470655, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a9c4094f, ext:3605238835, loc:(*time.Location)(0x781d740)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:22 minikube kubelet[3010]: E0628 20:31:22.147513 3010 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.15ac76dee402603f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc239c11543f, ext:3375422170, loc:(*time.Location)(0x781d740)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf3dbc23a9ea1830, ext:3607733600, loc:(*time.Location)(0x781d740)}}, Count:5, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Jun 28 20:31:30 minikube kubelet[3010]: I0628 20:31:30.495279 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/3a4da883-17ca-4a5b-9324-ca24aee64a30-lib-modules") pod "kube-proxy-b2jpw" (UID: "3a4da883-17ca-4a5b-9324-ca24aee64a30")
Jun 28 20:31:30 minikube kubelet[3010]: I0628 20:31:30.495365 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/3a4da883-17ca-4a5b-9324-ca24aee64a30-kube-proxy") pod "kube-proxy-b2jpw" (UID: "3a4da883-17ca-4a5b-9324-ca24aee64a30")
Jun 28 20:31:30 minikube kubelet[3010]: I0628 20:31:30.495415 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-vng88" (UniqueName: "kubernetes.io/secret/3a4da883-17ca-4a5b-9324-ca24aee64a30-kube-proxy-token-vng88") pod "kube-proxy-b2jpw" (UID: "3a4da883-17ca-4a5b-9324-ca24aee64a30")
Jun 28 20:31:30 minikube kubelet[3010]: I0628 20:31:30.495522 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/3a4da883-17ca-4a5b-9324-ca24aee64a30-xtables-lock") pod "kube-proxy-b2jpw" (UID: "3a4da883-17ca-4a5b-9324-ca24aee64a30")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.301860 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/3a56ab17-aad6-45f3-813c-dfb6d75ddd69-config-volume") pod "coredns-5c98db65d4-grhc2" (UID: "3a56ab17-aad6-45f3-813c-dfb6d75ddd69")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.303274 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-shjs8" (UniqueName: "kubernetes.io/secret/f849e8a2-462c-47d8-9cd8-86a9d0f2c5f8-coredns-token-shjs8") pod "coredns-5c98db65d4-z6jl7" (UID: "f849e8a2-462c-47d8-9cd8-86a9d0f2c5f8")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.303648 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-shjs8" (UniqueName: "kubernetes.io/secret/3a56ab17-aad6-45f3-813c-dfb6d75ddd69-coredns-token-shjs8") pod "coredns-5c98db65d4-grhc2" (UID: "3a56ab17-aad6-45f3-813c-dfb6d75ddd69")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.303998 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/f849e8a2-462c-47d8-9cd8-86a9d0f2c5f8-config-volume") pod "coredns-5c98db65d4-z6jl7" (UID: "f849e8a2-462c-47d8-9cd8-86a9d0f2c5f8")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.605835 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-jrqgk" (UniqueName: "kubernetes.io/secret/0ea547b2-82bd-465c-b1d8-b020c49159c4-storage-provisioner-token-jrqgk") pod "storage-provisioner" (UID: "0ea547b2-82bd-465c-b1d8-b020c49159c4")
Jun 28 20:31:32 minikube kubelet[3010]: I0628 20:31:32.606061 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/0ea547b2-82bd-465c-b1d8-b020c49159c4-tmp") pod "storage-provisioner" (UID: "0ea547b2-82bd-465c-b1d8-b020c49159c4")
Jun 28 20:31:33 minikube kubelet[3010]: W0628 20:31:33.368502 3010 pod_container_deletor.go:75] Container "b73805d4a687d75d991610ad1c2552102d9f42f00e2e5529cfdd550a947c9d20" not found in pod's containers
Jun 28 20:31:33 minikube kubelet[3010]: W0628 20:31:33.554251 3010 pod_container_deletor.go:75] Container "14ff5eb56f6192f01f6f271da07ea970bf3a775f64966addcd8808a2912fb2ed" not found in pod's containers
Jun 28 20:36:29 minikube kubelet[3010]: I0628 20:36:29.780191 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-wrmc5" (UniqueName: "kubernetes.io/secret/4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153-default-token-wrmc5") pod "my-job-bad-6962x" (UID: "4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153")
Jun 28 20:36:29 minikube kubelet[3010]: I0628 20:36:29.780413 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "my-persistent-volume" (UniqueName: "kubernetes.io/host-path/4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153-my-persistent-volume") pod "my-job-bad-6962x" (UID: "4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153")
Jun 28 20:37:45 minikube kubelet[3010]: E0628 20:37:45.180086 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:37:45 minikube kubelet[3010]: E0628 20:37:45.180313 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:37:47 minikube kubelet[3010]: E0628 20:37:47.346431 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:37:47 minikube kubelet[3010]: E0628 20:37:47.346623 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:38:02 minikube kubelet[3010]: E0628 20:38:02.425642 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:38:02 minikube kubelet[3010]: E0628 20:38:02.426377 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:38:16 minikube kubelet[3010]: E0628 20:38:16.459424 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:38:16 minikube kubelet[3010]: E0628 20:38:16.459540 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:38:28 minikube kubelet[3010]: E0628 20:38:28.447362 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:38:28 minikube kubelet[3010]: E0628 20:38:28.450044 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:38:40 minikube kubelet[3010]: E0628 20:38:40.454969 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:38:40 minikube kubelet[3010]: E0628 20:38:40.455046 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:38:55 minikube kubelet[3010]: E0628 20:38:55.027408 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:38:55 minikube kubelet[3010]: E0628 20:38:55.027544 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:39:06 minikube kubelet[3010]: E0628 20:39:06.467351 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:39:06 minikube kubelet[3010]: E0628 20:39:06.467523 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:39:18 minikube kubelet[3010]: E0628 20:39:18.451915 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:39:18 minikube kubelet[3010]: E0628 20:39:18.452059 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:39:32 minikube kubelet[3010]: E0628 20:39:32.446451 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:39:32 minikube kubelet[3010]: E0628 20:39:32.446592 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:39:48 minikube kubelet[3010]: E0628 20:39:48.462440 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:39:48 minikube kubelet[3010]: E0628 20:39:48.462514 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:40:01 minikube kubelet[3010]: E0628 20:40:01.735487 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:40:01 minikube kubelet[3010]: E0628 20:40:01.735637 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:40:17 minikube kubelet[3010]: E0628 20:40:17.444403 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:40:17 minikube kubelet[3010]: E0628 20:40:17.444538 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:40:33 minikube kubelet[3010]: E0628 20:40:33.440607 3010 kuberuntime_manager.go:775] container start failed: CreateContainerConfigError: stat /opt/my-path/: no such file or directory
Jun 28 20:40:33 minikube kubelet[3010]: E0628 20:40:33.441939 3010 pod_workers.go:190] Error syncing pod 4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153 ("my-job-bad-6962x_my-namespace(4bebaf7b-adfe-480b-aa3f-ee3ea3a2e153)"), skipping: failed to "StartContainer" for "subpath-test" with CreateContainerConfigError: "stat /opt/my-path/: no such file or directory"
Jun 28 20:40:42 minikube kubelet[3010]: I0628 20:40:42.538640 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-wrmc5" (UniqueName: "kubernetes.io/secret/ebc79fb2-80c0-4ed1-9e3e-fe74bde725c4-default-token-wrmc5") pod "my-job-good-dlfxn" (UID: "ebc79fb2-80c0-4ed1-9e3e-fe74bde725c4")
Jun 28 20:40:42 minikube kubelet[3010]: I0628 20:40:42.538805 3010 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "my-persistent-volume" (UniqueName: "kubernetes.io/host-path/ebc79fb2-80c0-4ed1-9e3e-fe74bde725c4-my-persistent-volume") pod "my-job-good-dlfxn" (UID: "ebc79fb2-80c0-4ed1-9e3e-fe74bde725c4")
==> storage-provisioner <==
The operating system version:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
docktermj
👍2
All 25 comments
I am experiencing the same behavior. Any estimate on when the issue will be reviewed?
brianmacy
on 8 Jul 2019
I'm fairly ignorant about PVC's, but by what mechanism are you expecting /opt/my-path to be created by?
tstromberg
on 16 Jul 2019
@tstromberg My understanding of Persistent Volume Claims (PVCs) is that Kubernetes will inform Docker of the volume. Much like a docker run .... --volumes /opt/my-path/:/opt/my-subpath ...
Did I come close to understanding your question?
docktermj
on 17 Jul 2019
My guess is that minikube doesn't completely implement the subPath behavior, yet. The error:
Error: stat /opt/my-path/: no such file or directory
seems to indicate that when subPath is specified, minikube somehow forgets to create the PV /opt/my-path when the Job's volumeMounts is specified.
Without the subPath specified, /opt/my-path seems to be located just fine.
docktermj
on 17 Jul 2019
@docktermj - Is it possible that hostPath is only going to work if /opt/my-path exists within the minikube VM?
minikube ssh stat /opt/my-path probably needs to work first before it is used as a hostPath, but I could be entirely mistaken here. Do other paths like /tmp work?
If it helps: Docker inside of the guest VM does not speak with Docker on the host. There is minikube mount for setting up a 9p mount between the host and the guest VM, but it doesn't rely on Docker to do so.
My apologies here for lacking in knowledeg in this topic. Just trying to help anyways =)
tstromberg
on 17 Jul 2019
@tstromberg Other paths in /tmp work if subPath is not specified (i.e. the difference between my-job-bad.yaml and my-job-good.yaml above. That's what confuses me. If they both didn't work or both did work, I'd be less confused.
I'll look into the minikube mount thought. (Probably early next week)
Appreciate you helping. You ask questions that make me think. ...and that may be the way this gets solved.
docktermj
on 17 Jul 2019
Any luck with this?
tstromberg
on 15 Aug 2019
Nothing yet. Today I "upped" my minikube version to 1.3.1 and will try again early next week.
docktermj
on 16 Aug 2019
Same issue for 1.3.1:
$ minikube start --cpus 4 --memory 8192 --vm-driver kvm2
😄 minikube v1.3.1 on Ubuntu 18.04
⚠️ Error checking driver version: exit status 1
🔥 Creating kvm2 VM (CPUs=4, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.15.2 on Docker 18.09.8 ...
🚜 Pulling images ...
🚀 Launching Kubernetes ...
⌛ Waiting for: apiserver proxy etcd scheduler controller dns
🏄 Done! kubectl is now configured to use "minikube"
So still an open issue.
docktermj
on 20 Aug 2019
👍1
As a work-around, I can do this:
minikube ssh
From the minikube prompt:
sudo mkdir /opt/my-path
Then the subPath key-values work.
I consider this a work-around because it's procedural, not declarative.
docktermj
on 20 Aug 2019
Well, here's another bad work-around. Add an initContainer YAML stanza to the my-job-bad.yaml file (described in the initial description) like this:
```yaml
cat <
kind: Job
metadata:
name: my-job-bad
namespace: my-namespace
spec:
template:
spec:
initContainers:
- name: pre-mount
image: busybox:1.28
volumeMounts:
- name: my-volume
mountPath: /opt/my-subpath
containers:
- name: subpath-test
image: docker.io/centos:latest
imagePullPolicy: Always
command: ["sleep"]
args: ["infinity"]
volumeMounts:
- name: my-volume
mountPath: /opt/my-subpath
subPath: my-subpath-1
restartPolicy: Never
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-persistent-volume-claim
EOT
docktermj
on 20 Aug 2019
@tstromberg Given the two "bad" work-arounds above, how do we request a fix in the minikube code?
docktermj
on 20 Aug 2019
👍1
@docktermj - consider this issue the request.
I'm still a little unclear on where this should be fixed. The only PVC code in minikube is this package:
https://github.com/kubernetes/minikube/blob/master/pkg/storage/storage_provisioner.go
It may be possible that simply rebuilding the storage-provisioner image since we moved off of the r2d4 storage-provisioner fork (#3628) might have an effect here, but it hasn't been tested. Anyways, help wanted!
tstromberg
on 22 Aug 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 19 Dec 2019
/remove-lifecycle stale
docktermj
on 20 Dec 2019
wasn't it fixed already in https://github.com/kubernetes/minikube/issues/2256?
avoidik
on 9 Feb 2020
@docktermj Is this still an issue? Have you tried with any newer version of minikube?
sharifelgamal
on 22 Apr 2020
@sharifelgamal
Since I put in my work-around, I haven't revisited the issue. I can certainly try a new version of minikube.
docktermj
on 26 Apr 2020
I do have a similar issue with Minikube on Arch:
$ minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393
It seems to work the first time I create the deployment, and then when I stop/start minikube again, the pod won't restart with the error Warning Failed 10m (x11 over 12m) kubelet, minikube Error: stat /tmp/hostpath-provisioner/pvc-d8a5a418-4dc5-4b87-9dd1-1340b597f215: no such file or directory. PVC also has a subPath.
If I delete the PVC and create it again, the pod is able to start successfully.
isra17
on 27 Apr 2020
@isra17 do you mind sharing your full workflow example with yaml files so I could replicate this issue?
medyagh
on 13 May 2020
Fully reproducible steps:
$ minikube start -p test
😄 [test] minikube v1.9.2 on Arch rolling
✨ Automatically selected the docker driver
👍 Starting control plane node m01 in cluster test
🚜 Pulling base image ...
🔥 Creating Kubernetes in docker container with (CPUs=2) (4 available), Memory=8096MB
🐳 Preparing Kubernetes v1.18.0 on Docker 19.03.2 ...
▪ kubeadm.pod-network-cidr=10.244.0.0/16
🌟 Enabling addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "test"
Apply Statefulset with a provisionned volume using subPath.
kubectl apply -f https://gist.githubusercontent.com/isra17/ead482c2c5230e78d3d80be44877bfe0/raw/bdc9f4b0fc9baf043393152cfc0f47e7547cdbb7/objects.yaml
At that point the pod should be running.
Restart Minikube
$ minikube stop -p test
$ minikube start -p test
Pod won't restart
$ kubectl describe pod test-statefulset-0
Name: test-statefulset-0
Namespace: default
Priority: 0
Node: test/172.17.0.3
Start Time: Wed, 13 May 2020 09:40:06 -0400
Labels: controller-revision-hash=test-statefulset-66db94bc56
name=test
statefulset.kubernetes.io/pod-name=test-statefulset-0
Annotations: <none>
Status: Running
IP: 172.18.0.2
IPs:
IP: 172.18.0.2
Controlled By: StatefulSet/test-statefulset
Containers:
ubuntu:
Container ID: docker://f378ed9cc4cd7c81cae762c59eb46ac0532820a1a95a4fee6b1fc53d713edc83
Image: k8s.gcr.io/echoserver:1.4
Image ID: docker-pullable://k8s.gcr.io/echoserver@sha256:5d99aa1120524c801bc8c1a7077e8f5ec122ba16b6dda1a5d3826057f67b9bcb
Port: <none>
Host Port: <none>
State: Waiting
Reason: CreateContainerConfigError
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 13 May 2020 09:40:19 -0400
Finished: Wed, 13 May 2020 09:40:54 -0400
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q59vp (ro)
/volume from test-volume-claim (rw,path="subpath")
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
test-volume-claim:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: test-volume-claim-test-statefulset-0
ReadOnly: false
default-token-q59vp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-q59vp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 90s (x2 over 90s) default-scheduler running "VolumeBinding" filter plugin for pod "test-statefulset-0": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 88s default-scheduler Successfully assigned default/test-statefulset-0 to test
Normal Pulling 87s kubelet, test Pulling image "k8s.gcr.io/echoserver:1.4"
Normal Pulled 75s kubelet, test Successfully pulled image "k8s.gcr.io/echoserver:1.4"
Normal Created 75s kubelet, test Created container ubuntu
Normal Started 75s kubelet, test Started container ubuntu
Normal SandboxChanged 12s kubelet, test Pod sandbox changed, it will be killed and re-created.
Normal Pulled 11s (x2 over 12s) kubelet, test Container image "k8s.gcr.io/echoserver:1.4" already present on machine
Warning Failed 11s (x2 over 12s) kubelet, test Error: stat /tmp/hostpath-provisioner/pvc-823397a6-ff98-4bbb-9c49-2f5b45195933: no such file or directory
$ minikube logs -p test
$ minikube logs -p test
==> Docker <==
-- Logs begin at Wed 2020-05-13 13:47:37 UTC, end at Wed 2020-05-13 13:48:22 UTC. --
May 13 13:47:53 test dockerd[394]: time="2020-05-13T13:47:53.474499375Z" level=info msg="shim containerd-shim started" address=/containerd-shim/7a0b7141e56004f11930bd8e9fca2ff054c2be30c2cb63261175d4d7aa9d9dd1.sock debug=false pid=1616
May 13 13:47:53 test dockerd[394]: time="2020-05-13T13:47:53.567716115Z" level=info msg="shim containerd-shim started" address=/containerd-shim/419e64139a148256d8146ac6947e9d3ce45eeaea7de020c72478ad964da1d9a9.sock debug=false pid=1633
May 13 13:47:53 test dockerd[394]: time="2020-05-13T13:47:53.575254050Z" level=info msg="shim containerd-shim started" address=/containerd-shim/d5b6c54a45fb1db4ebf6d0f1ec9f4c8a2431f2482c4ea8901da580f9d640a4c6.sock debug=false pid=1656
May 13 13:47:53 test dockerd[394]: time="2020-05-13T13:47:53.646619953Z" level=info msg="shim containerd-shim started" address=/containerd-shim/f2c6e2a440aadb6095b8e480c3f0ad44964395bd780faac5265b360de50cb04a.sock debug=false pid=1707
May 13 13:47:53 test sudo[1724]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/pgrep -xnf kube-apiserver.*minikube.*
May 13 13:47:53 test sudo[1724]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:47:53 test sudo[1724]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:47:53 test sudo[1724]: pam_unix(sudo:session): session closed for user root
May 13 13:47:53 test dockerd[394]: time="2020-05-13T13:47:53.714410622Z" level=info msg="shim containerd-shim started" address=/containerd-shim/7bf0eede9d4d237df7365bcd5d320423d371f56f9788359e1e2c34589171c34e.sock debug=false pid=1753
May 13 13:47:54 test sudo[1807]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/pgrep -xnf kube-apiserver.*minikube.*
May 13 13:47:54 test sudo[1807]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:47:54 test sudo[1807]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:47:54 test sudo[1807]: pam_unix(sudo:session): session closed for user root
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.276274518Z" level=info msg="shim containerd-shim started" address=/containerd-shim/51c89c70741faf2210a4c0070628b5a1d1931d02fc239743297982cd9fdfd4e3.sock debug=false pid=1813
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.311851506Z" level=info msg="shim containerd-shim started" address=/containerd-shim/e519201ea92818471a35cf20ac284810e66095f9921fb4b25a2e87b7bbb57f55.sock debug=false pid=1837
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.322022008Z" level=info msg="shim containerd-shim started" address=/containerd-shim/bd5ca1a25fa0262b0888b37e49b0776bbb197dbd4f8d87f57f0e5fbe1284d6b4.sock debug=false pid=1856
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.339234340Z" level=info msg="shim containerd-shim started" address=/containerd-shim/63223dbf25e4214371a4e59e81f9ef9e811692abb56f8d580ce3eddb4c7eb8e0.sock debug=false pid=1897
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.340232935Z" level=info msg="shim containerd-shim started" address=/containerd-shim/8756721925bde235d79a7057d018811228aa825c3626b6af34af8cb70b934e74.sock debug=false pid=1901
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.380865194Z" level=info msg="shim reaped" id=b05bdb2b5bfd5270274e5dbcdd7e36d1f1a5fef90c8606728ef7902b9b55af9c
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.391949615Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 13 13:47:54 test dockerd[394]: time="2020-05-13T13:47:54.392052191Z" level=warning msg="b05bdb2b5bfd5270274e5dbcdd7e36d1f1a5fef90c8606728ef7902b9b55af9c cleanup: failed to unmount IPC: umount /var/lib/docker/containers/b05bdb2b5bfd5270274e5dbcdd7e36d1f1a5fef90c8606728ef7902b9b55af9c/mounts/shm, flags: 0x2: no such file or directory"
May 13 13:47:54 test sudo[2055]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/pgrep -xnf kube-apiserver.*minikube.*
May 13 13:47:54 test sudo[2055]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:47:54 test sudo[2055]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:47:54 test sudo[2055]: pam_unix(sudo:session): session closed for user root
May 13 13:47:59 test sudo[2314]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/env PATH=/var/lib/minikube/binaries/v1.18.0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin kubeadm init phase addon all --config /var/tmp/minikube/kubeadm.yaml
May 13 13:47:59 test sudo[2314]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:47:59 test sudo[2314]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:47:59 test sudo[2314]: pam_unix(sudo:session): session closed for user root
May 13 13:48:00 test sudo[2336]: root : TTY=unknown ; PWD=/ ; USER=root ; ENV=KUBECONFIG=/var/lib/minikube/kubeconfig ; COMMAND=/var/lib/minikube/binaries/v1.18.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml
May 13 13:48:00 test sudo[2336]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:00 test sudo[2336]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:00 test sudo[2336]: pam_unix(sudo:session): session closed for user root
May 13 13:48:00 test sudo[2352]: root : TTY=unknown ; PWD=/ ; USER=root ; ENV=KUBECONFIG=/var/lib/minikube/kubeconfig ; COMMAND=/var/lib/minikube/binaries/v1.18.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml
May 13 13:48:00 test sudo[2352]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:00 test sudo[2352]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:01 test sudo[2352]: pam_unix(sudo:session): session closed for user root
May 13 13:48:09 test sudo[2555]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/journalctl -u docker -n 60
May 13 13:48:09 test sudo[2555]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:09 test sudo[2555]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:09 test sudo[2555]: pam_unix(sudo:session): session closed for user root
May 13 13:48:10 test sudo[2568]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/crictl ps -a
May 13 13:48:10 test sudo[2568]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:10 test sudo[2568]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:10 test sudo[2568]: pam_unix(sudo:session): session closed for user root
May 13 13:48:10 test sudo[2603]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/var/lib/minikube/binaries/v1.18.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig
May 13 13:48:10 test sudo[2603]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:10 test sudo[2603]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:10 test sudo[2603]: pam_unix(sudo:session): session closed for user root
May 13 13:48:10 test sudo[2624]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/dmesg -PH -L=never --level warn,err,crit,alert,emerg
May 13 13:48:10 test sudo[2624]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:10 test sudo[2624]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:10 test sudo[2624]: pam_unix(sudo:session): session closed for user root
May 13 13:48:12 test sudo[2758]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/journalctl -u kubelet -n 60
May 13 13:48:12 test sudo[2758]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:12 test sudo[2758]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 13 13:48:12 test sudo[2758]: pam_unix(sudo:session): session closed for user root
May 13 13:48:22 test sudo[2920]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/journalctl -u docker -n 60
May 13 13:48:22 test sudo[2920]: pam_env(sudo:session): Unable to open env file: /etc/default/locale: No such file or directory
May 13 13:48:22 test sudo[2920]: pam_unix(sudo:session): session opened for user root by (uid=0)
==> container status <==
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
c3929176040d4 303ce5db0e90d 30 seconds ago Running etcd 0 b1062313cff81
45171cca54014 a31f78c7c8ce1 30 seconds ago Running kube-scheduler 2 34558faa69841
b05bdb2b5bfd5 303ce5db0e90d 30 seconds ago Exited etcd 2 c62db40c4ec22
49ededb937d84 74060cea7f704 30 seconds ago Running kube-apiserver 0 adc61e4f515f3
1842b516ecb84 d3e55153f52fb 30 seconds ago Running kube-controller-manager 2 e5309da29c254
bd4abaeb5b8e7 67da37a9a360e 6 minutes ago Exited coredns 2 9b0479671e904
37c07b76e2149 67da37a9a360e 6 minutes ago Exited coredns 2 19acb6816820b
171780a64b204 4689081edb103 6 minutes ago Exited storage-provisioner 2 83371d1c3ceb8
a92cd62a6027f aa67fec7d7ef7 6 minutes ago Exited kindnet-cni 2 a8dca6e796bca
503fbb40f73e1 43940c34f24f3 7 minutes ago Exited kube-proxy 1 dd39e740b8fe0
d921b5840eedc d3e55153f52fb 7 minutes ago Exited kube-controller-manager 1 17f0c4976af7e
6b96b887f044e 74060cea7f704 7 minutes ago Exited kube-apiserver 1 c4372c266f4c8
cf9a5808a6ee5 a31f78c7c8ce1 7 minutes ago Exited kube-scheduler 1 8c8c2ef0a9039
f378ed9cc4cd7 k8s.gcr.io/echoserver@sha256:5d99aa1120524c801bc8c1a7077e8f5ec122ba16b6dda1a5d3826057f67b9bcb 8 minutes ago Exited ubuntu 0 79d0aa2f2bb87
==> coredns [37c07b76e214] <==
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s
E0513 13:47:22.073394 1 reflector.go:307] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to watch *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=1303&timeout=5m19s&timeoutSeconds=319&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0513 13:47:22.073604 1 reflector.go:307] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to watch *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=2233&timeout=8m26s&timeoutSeconds=506&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0513 13:47:22.073918 1 reflector.go:307] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to watch *v1.Service: Get https://10.96.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=1303&timeout=7m7s&timeoutSeconds=427&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
==> coredns [bd4abaeb5b8e] <==
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s
==> describe nodes <==
Name: test
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=test
kubernetes.io/os=linux
minikube.k8s.io/commit=93af9c1e43cab9618e301bc9fa720c63d5efa393
minikube.k8s.io/name=test
minikube.k8s.io/updated_at=2020_05_13T09_34_30_0700
minikube.k8s.io/version=v1.9.2
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 13 May 2020 13:34:26 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: test
AcquireTime: <unset>
RenewTime: Wed, 13 May 2020 13:47:20 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 13 May 2020 13:46:22 +0000 Wed, 13 May 2020 13:34:23 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 13 May 2020 13:46:22 +0000 Wed, 13 May 2020 13:34:23 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 13 May 2020 13:46:22 +0000 Wed, 13 May 2020 13:34:23 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 13 May 2020 13:46:22 +0000 Wed, 13 May 2020 13:34:46 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.17.0.3
Hostname: test
Capacity:
cpu: 4
ephemeral-storage: 240232960Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32777108Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 240232960Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32777108Ki
pods: 110
System Info:
Machine ID: ed47d8fb8ba44d27912a804bbc124f21
System UUID: 05c6f637-99c4-4e9a-9e40-5d1fe8b77c5e
Boot ID: 9975863c-53d7-4723-bc52-4ca165d015b8
Kernel Version: 5.6.11-arch1-1
OS Image: Ubuntu 19.10
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.2
Kubelet Version: v1.18.0
Kube-Proxy Version: v1.18.0
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default test-statefulset-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8m19s
kube-system coredns-66bff467f8-dmbqx 100m (2%) 0 (0%) 70Mi (0%) 170Mi (0%) 13m
kube-system coredns-66bff467f8-s5chg 100m (2%) 0 (0%) 70Mi (0%) 170Mi (0%) 13m
kube-system etcd-test 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13m
kube-system kindnet-5x6vl 100m (2%) 100m (2%) 50Mi (0%) 50Mi (0%) 13m
kube-system kube-apiserver-test 250m (6%) 0 (0%) 0 (0%) 0 (0%) 13m
kube-system kube-controller-manager-test 200m (5%) 0 (0%) 0 (0%) 0 (0%) 13m
kube-system kube-proxy-hwrqb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13m
kube-system kube-scheduler-test 100m (2%) 0 (0%) 0 (0%) 0 (0%) 13m
kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 850m (21%) 100m (2%)
memory 190Mi (0%) 390Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 14m (x4 over 14m) kubelet, test Node test status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 14m (x5 over 14m) kubelet, test Node test status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 14m (x4 over 14m) kubelet, test Node test status is now: NodeHasSufficientPID
Normal Starting 13m kubelet, test Starting kubelet.
Normal NodeHasSufficientMemory 13m kubelet, test Node test status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 13m kubelet, test Node test status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 13m kubelet, test Node test status is now: NodeHasSufficientPID
Normal NodeNotReady 13m kubelet, test Node test status is now: NodeNotReady
Normal NodeAllocatableEnforced 13m kubelet, test Updated Node Allocatable limit across pods
Normal NodeReady 13m kubelet, test Node test status is now: NodeReady
Normal Starting 13m kube-proxy, test Starting kube-proxy.
Normal Starting 7m7s kubelet, test Starting kubelet.
Normal NodeHasSufficientMemory 7m7s (x8 over 7m7s) kubelet, test Node test status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 7m7s (x8 over 7m7s) kubelet, test Node test status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 7m7s (x7 over 7m7s) kubelet, test Node test status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 7m7s kubelet, test Updated Node Allocatable limit across pods
Normal Starting 7m1s kube-proxy, test Starting kube-proxy.
==> dmesg <==
[ +0.000004] activate_task+0x20b/0x3c0
[ +0.000003] ? sched_clock+0x5/0x10
[ +0.000001] ttwu_do_activate+0x45/0x60
[ +0.000001] try_to_wake_up+0x24a/0x750
[ +0.000003] signal_wake_up_state+0x15/0x30
[ +0.000001] __send_signal+0x1e4/0x410
[ +0.000002] do_notify_parent+0x27d/0x2c0
[ +0.000002] release_task+0x3e7/0x450
[ +0.000001] do_exit+0x709/0xb40
[ +0.000003] ? __switch_to_asm+0x40/0x70
[ +0.000001] ? __switch_to_asm+0x40/0x70
[ +0.000001] ? __switch_to_asm+0x34/0x70
[ +0.000002] do_group_exit+0x3a/0xa0
[ +0.000001] get_signal+0x132/0x8c0
[ +0.000003] do_signal+0x43/0x680
[ +0.000001] ? do_nanosleep+0xba/0x170
[ +0.000003] exit_to_usermode_loop+0x7f/0x100
[ +0.000002] do_syscall_64+0x11f/0x150
[ +0.000001] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ +0.000002] RIP: 0033:0x45ad2d
[ +0.000003] Code: Bad RIP value.
[ +0.000000] RSP: 002b:00007f2dfff3ac68 EFLAGS: 00000202 ORIG_RAX: 0000000000000023
[ +0.000001] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000045ad2d
[ +0.000000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f2dfff3ac68
[ +0.000001] RBP: 00007f2dfff3ac78 R08: 0000000000000000 R09: 0000000000000000
[ +0.000000] R10: 0000000000000008 R11: 0000000000000202 R12: 00007f2dfff3ae80
[ +0.000001] R13: 0000000000000000 R14: 0000000002085b60 R15: 00007fff4d377f20
[ +0.000001] ---[ end trace 6b6168395cb31f9c ]---
[ +0.785500] kauditd_printk_skb: 80 callbacks suppressed
[ +6.887254] kauditd_printk_skb: 26 callbacks suppressed
[ +8.719362] kauditd_printk_skb: 15 callbacks suppressed
[May13 13:31] kauditd_printk_skb: 18 callbacks suppressed
[May13 13:32] kauditd_printk_skb: 14 callbacks suppressed
[May13 13:34] device-mapper: thin: Deletion of thin device 4569 failed.
[ +0.024301] device-mapper: ioctl: remove_all left 3 open device(s)
[ +0.178364] kauditd_printk_skb: 49 callbacks suppressed
[ +5.027559] kauditd_printk_skb: 322 callbacks suppressed
[ +9.950978] kauditd_printk_skb: 43 callbacks suppressed
[ +8.194707] kauditd_printk_skb: 5 callbacks suppressed
[ +8.440263] kauditd_printk_skb: 34 callbacks suppressed
[ +11.524546] kauditd_printk_skb: 12 callbacks suppressed
[May13 13:40] kauditd_printk_skb: 14 callbacks suppressed
[ +10.740999] kauditd_printk_skb: 6 callbacks suppressed
[May13 13:41] kauditd_printk_skb: 12 callbacks suppressed
[ +5.053340] kauditd_printk_skb: 153 callbacks suppressed
[ +5.061953] kauditd_printk_skb: 174 callbacks suppressed
[ +9.468556] kauditd_printk_skb: 71 callbacks suppressed
[ +13.716949] kauditd_printk_skb: 27 callbacks suppressed
[May13 13:46] kauditd_printk_skb: 20 callbacks suppressed
[ +24.112314] kauditd_printk_skb: 20 callbacks suppressed
[May13 13:47] kauditd_printk_skb: 5 callbacks suppressed
[ +10.240911] kauditd_printk_skb: 4 callbacks suppressed
[ +14.839035] kauditd_printk_skb: 5 callbacks suppressed
[ +8.987803] kauditd_printk_skb: 12 callbacks suppressed
[ +5.879428] kauditd_printk_skb: 149 callbacks suppressed
[ +5.426731] kauditd_printk_skb: 188 callbacks suppressed
[ +5.498690] kauditd_printk_skb: 73 callbacks suppressed
[ +6.333289] kauditd_printk_skb: 14 callbacks suppressed
[May13 13:48] kauditd_printk_skb: 8 callbacks suppressed
[ +12.907649] kauditd_printk_skb: 20 callbacks suppressed
==> etcd [b05bdb2b5bfd] <==
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-05-13 13:47:54.335440 I | etcdmain: etcd Version: 3.4.3
2020-05-13 13:47:54.335464 I | etcdmain: Git SHA: 3cf2f69b5
2020-05-13 13:47:54.335467 I | etcdmain: Go Version: go1.12.12
2020-05-13 13:47:54.335472 I | etcdmain: Go OS/Arch: linux/amd64
2020-05-13 13:47:54.335474 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2020-05-13 13:47:54.335511 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-05-13 13:47:54.335539 I | embed: peerTLS: cert = /var/lib/minikube/certs/etcd/peer.crt, key = /var/lib/minikube/certs/etcd/peer.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file =
2020-05-13 13:47:54.335659 C | etcdmain: listen tcp 172.17.0.3:2380: bind: cannot assign requested address
==> etcd [c3929176040d] <==
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-05-13 13:47:54.389664 I | etcdmain: etcd Version: 3.4.3
2020-05-13 13:47:54.389698 I | etcdmain: Git SHA: 3cf2f69b5
2020-05-13 13:47:54.389701 I | etcdmain: Go Version: go1.12.12
2020-05-13 13:47:54.389703 I | etcdmain: Go OS/Arch: linux/amd64
2020-05-13 13:47:54.389706 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2020-05-13 13:47:54.389741 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-05-13 13:47:54.389761 I | embed: peerTLS: cert = /var/lib/minikube/certs/etcd/peer.crt, key = /var/lib/minikube/certs/etcd/peer.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file =
2020-05-13 13:47:54.390195 I | embed: name = test
2020-05-13 13:47:54.390204 I | embed: data dir = /var/lib/minikube/etcd
2020-05-13 13:47:54.390207 I | embed: member dir = /var/lib/minikube/etcd/member
2020-05-13 13:47:54.390209 I | embed: heartbeat = 100ms
2020-05-13 13:47:54.390211 I | embed: election = 1000ms
2020-05-13 13:47:54.390214 I | embed: snapshot count = 10000
2020-05-13 13:47:54.390223 I | embed: advertise client URLs = https://172.17.0.2:2379
2020-05-13 13:47:54.390226 I | embed: initial advertise peer URLs = https://172.17.0.2:2380
2020-05-13 13:47:54.390229 I | embed: initial cluster =
2020-05-13 13:47:54.402897 I | etcdserver: restarting member b273bc7741bcb020 in cluster 86482fea2286a1d2 at commit index 2482
raft2020/05/13 13:47:54 INFO: b273bc7741bcb020 switched to configuration voters=()
raft2020/05/13 13:47:54 INFO: b273bc7741bcb020 became follower at term 3
raft2020/05/13 13:47:54 INFO: newRaft b273bc7741bcb020 [peers: [], term: 3, commit: 2482, applied: 0, lastindex: 2482, lastterm: 3]
2020-05-13 13:47:54.544509 W | auth: simple token is not cryptographically signed
2020-05-13 13:47:54.609399 I | etcdserver: starting server... [version: 3.4.3, cluster version: to_be_decided]
2020-05-13 13:47:54.610944 I | embed: ClientTLS: cert = /var/lib/minikube/certs/etcd/server.crt, key = /var/lib/minikube/certs/etcd/server.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file =
2020-05-13 13:47:54.611037 I | embed: listening for metrics on http://127.0.0.1:2381
2020-05-13 13:47:54.611325 I | embed: listening for peers on 172.17.0.2:2380
raft2020/05/13 13:47:54 INFO: b273bc7741bcb020 switched to configuration voters=(12858828581462913056)
2020-05-13 13:47:54.611537 I | etcdserver/membership: added member b273bc7741bcb020 [https://172.17.0.3:2380] to cluster 86482fea2286a1d2
2020-05-13 13:47:54.611588 N | etcdserver/membership: set the initial cluster version to 3.4
2020-05-13 13:47:54.611614 I | etcdserver/api: enabled capabilities for version 3.4
2020-05-13 13:47:54.793155 W | etcdserver: request "ID:12691269155536321281 Method:\"PUT\" Path:\"/0/members/b273bc7741bcb020/attributes\" Val:\"{\\\"name\\\":\\\"test\\\",\\\"clientURLs\\\":[\\\"https://172.17.0.3:2379\\\"]}\" " with result "" took too long (139.470432ms) to execute
raft2020/05/13 13:47:55 INFO: b273bc7741bcb020 is starting a new election at term 3
raft2020/05/13 13:47:55 INFO: b273bc7741bcb020 became candidate at term 4
raft2020/05/13 13:47:55 INFO: b273bc7741bcb020 received MsgVoteResp from b273bc7741bcb020 at term 4
raft2020/05/13 13:47:55 INFO: b273bc7741bcb020 became leader at term 4
raft2020/05/13 13:47:55 INFO: raft.node: b273bc7741bcb020 elected leader b273bc7741bcb020 at term 4
2020-05-13 13:47:55.738378 I | etcdserver: published {Name:test ClientURLs:[https://172.17.0.2:2379]} to cluster 86482fea2286a1d2
2020-05-13 13:47:55.738391 I | embed: ready to serve client requests
2020-05-13 13:47:55.738497 I | embed: ready to serve client requests
2020-05-13 13:47:55.739196 I | embed: serving client requests on 127.0.0.1:2379
2020-05-13 13:47:55.739254 I | embed: serving client requests on 172.17.0.2:2379
==> kernel <==
13:48:23 up 9:40, 0 users, load average: 1.52, 1.42, 1.46
Linux test 5.6.11-arch1-1 #1 SMP PREEMPT Wed, 06 May 2020 17:32:37 +0000 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 19.10"
==> kube-apiserver [49ededb937d8] <==
I0513 13:47:56.320235 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
I0513 13:47:56.326550 1 client.go:361] parsed scheme: "endpoint"
I0513 13:47:56.326652 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
W0513 13:47:56.423671 1 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
W0513 13:47:56.430316 1 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0513 13:47:56.437938 1 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0513 13:47:56.450242 1 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0513 13:47:56.452535 1 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0513 13:47:56.464252 1 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0513 13:47:56.479695 1 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W0513 13:47:56.479724 1 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
I0513 13:47:56.487430 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0513 13:47:56.487454 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0513 13:47:56.488779 1 client.go:361] parsed scheme: "endpoint"
I0513 13:47:56.488798 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
I0513 13:47:56.494742 1 client.go:361] parsed scheme: "endpoint"
I0513 13:47:56.494762 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
I0513 13:47:57.895089 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0513 13:47:57.895099 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0513 13:47:57.895340 1 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/minikube/certs/apiserver.crt::/var/lib/minikube/certs/apiserver.key
I0513 13:47:57.895508 1 secure_serving.go:178] Serving securely on [::]:8443
I0513 13:47:57.895547 1 crd_finalizer.go:266] Starting CRDFinalizer
I0513 13:47:57.895562 1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0513 13:47:57.895568 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0513 13:47:57.895607 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0513 13:47:57.898554 1 available_controller.go:387] Starting AvailableConditionController
I0513 13:47:57.898569 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0513 13:47:57.895632 1 controller.go:81] Starting OpenAPI AggregationController
I0513 13:47:57.895649 1 establishing_controller.go:76] Starting EstablishingController
I0513 13:47:57.895690 1 controller.go:86] Starting OpenAPI controller
I0513 13:47:57.895697 1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0513 13:47:57.895702 1 naming_controller.go:291] Starting NamingConditionController
I0513 13:47:57.896060 1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0513 13:47:57.896066 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0513 13:47:57.896218 1 autoregister_controller.go:141] Starting autoregister controller
I0513 13:47:57.898647 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0513 13:47:57.896233 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0513 13:47:57.898655 1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller
I0513 13:47:57.896552 1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0513 13:47:57.898663 1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
I0513 13:47:57.897128 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0513 13:47:57.897134 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt
E0513 13:47:57.899304 1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.17.0.2, ResourceVersion: 0, AdditionalErrorMsg:
I0513 13:47:57.995783 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0513 13:47:57.998764 1 shared_informer.go:230] Caches are synced for crd-autoregister
I0513 13:47:57.998847 1 cache.go:39] Caches are synced for AvailableConditionController controller
I0513 13:47:57.998855 1 cache.go:39] Caches are synced for autoregister controller
I0513 13:47:57.998869 1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller
I0513 13:47:58.894991 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0513 13:47:58.895123 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0513 13:47:58.897423 1 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
W0513 13:47:59.046566 1 lease.go:224] Resetting endpoints for master service "kubernetes" to [172.17.0.2]
I0513 13:47:59.047726 1 controller.go:606] quota admission added evaluator for: endpoints
I0513 13:47:59.072855 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0513 13:47:59.596595 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0513 13:47:59.612529 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0513 13:47:59.675032 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0513 13:47:59.740328 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0513 13:47:59.748243 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0513 13:48:14.762237 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
==> kube-apiserver [6b96b887f044] <==
I0513 13:41:20.730112 1 crd_finalizer.go:266] Starting CRDFinalizer
I0513 13:41:20.730124 1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0513 13:41:20.730131 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0513 13:41:20.730139 1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0513 13:41:20.730148 1 naming_controller.go:291] Starting NamingConditionController
I0513 13:41:20.730163 1 establishing_controller.go:76] Starting EstablishingController
I0513 13:41:20.730172 1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0513 13:41:20.730179 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0513 13:41:20.730291 1 controller.go:86] Starting OpenAPI controller
I0513 13:41:20.730966 1 available_controller.go:387] Starting AvailableConditionController
I0513 13:41:20.730975 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0513 13:41:20.730987 1 autoregister_controller.go:141] Starting autoregister controller
I0513 13:41:20.730990 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0513 13:41:20.732800 1 controller.go:81] Starting OpenAPI AggregationController
I0513 13:41:20.733463 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0513 13:41:20.733526 1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller
I0513 13:41:20.732853 1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0513 13:41:20.738199 1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
I0513 13:41:20.738283 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0513 13:41:20.738354 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0513 13:41:20.835418 1 cache.go:39] Caches are synced for autoregister controller
I0513 13:41:20.835887 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0513 13:41:20.836488 1 cache.go:39] Caches are synced for AvailableConditionController controller
E0513 13:41:20.835505 1 controller.go:152] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
I0513 13:41:20.835460 1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller
I0513 13:41:20.843269 1 shared_informer.go:230] Caches are synced for crd-autoregister
I0513 13:41:20.854319 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0513 13:41:21.729621 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0513 13:41:21.729757 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0513 13:41:21.735155 1 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I0513 13:41:22.426926 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0513 13:41:22.439437 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0513 13:41:22.466389 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0513 13:41:22.489316 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0513 13:41:22.495639 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0513 13:41:38.218094 1 controller.go:606] quota admission added evaluator for: endpoints
I0513 13:41:51.150862 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0513 13:43:39.856163 1 controller.go:606] quota admission added evaluator for: statefulsets.apps
I0513 13:47:22.071880 1 controller.go:181] Shutting down kubernetes service endpoint reconciler
I0513 13:47:22.072176 1 controller.go:123] Shutting down OpenAPI controller
I0513 13:47:22.072187 1 crdregistration_controller.go:142] Shutting down crd-autoregister controller
I0513 13:47:22.072200 1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller
I0513 13:47:22.072207 1 available_controller.go:399] Shutting down AvailableConditionController
I0513 13:47:22.072215 1 apiapproval_controller.go:198] Shutting down KubernetesAPIApprovalPolicyConformantConditionController
I0513 13:47:22.072222 1 nonstructuralschema_controller.go:198] Shutting down NonStructuralSchemaConditionController
I0513 13:47:22.072230 1 establishing_controller.go:87] Shutting down EstablishingController
I0513 13:47:22.072236 1 naming_controller.go:302] Shutting down NamingConditionController
I0513 13:47:22.072242 1 customresource_discovery_controller.go:220] Shutting down DiscoveryController
I0513 13:47:22.072248 1 apiservice_controller.go:106] Shutting down APIServiceRegistrationController
I0513 13:47:22.072275 1 crd_finalizer.go:278] Shutting down CRDFinalizer
I0513 13:47:22.072283 1 autoregister_controller.go:165] Shutting down autoregister controller
I0513 13:47:22.072528 1 dynamic_cafile_content.go:182] Shutting down request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0513 13:47:22.072535 1 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0513 13:47:22.072574 1 controller.go:87] Shutting down OpenAPI AggregationController
I0513 13:47:22.072599 1 dynamic_cafile_content.go:182] Shutting down request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0513 13:47:22.072793 1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
I0513 13:47:22.072805 1 dynamic_serving_content.go:145] Shutting down serving-cert::/var/lib/minikube/certs/apiserver.crt::/var/lib/minikube/certs/apiserver.key
I0513 13:47:22.072818 1 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0513 13:47:22.073816 1 secure_serving.go:222] Stopped listening on [::]:8443
E0513 13:47:22.075701 1 controller.go:184] Get https://localhost:8443/api/v1/namespaces/default/endpoints/kubernetes: dial tcp 127.0.0.1:8443: connect: connection refused
==> kube-controller-manager [1842b516ecb8] <==
I0513 13:48:16.746159 1 shared_informer.go:223] Waiting for caches to sync for tokens
I0513 13:48:16.846438 1 shared_informer.go:230] Caches are synced for tokens
I0513 13:48:17.005167 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0513 13:48:17.005206 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0513 13:48:17.005228 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
I0513 13:48:17.005254 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
I0513 13:48:17.005340 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
I0513 13:48:17.005464 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
I0513 13:48:17.005493 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
I0513 13:48:17.005520 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
I0513 13:48:17.005830 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I0513 13:48:17.006062 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0513 13:48:17.006267 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I0513 13:48:17.006400 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0513 13:48:17.006546 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
I0513 13:48:17.006721 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0513 13:48:17.006817 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
I0513 13:48:17.006912 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
I0513 13:48:17.007005 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
I0513 13:48:17.007104 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
I0513 13:48:17.007193 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
I0513 13:48:17.007329 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
I0513 13:48:17.007503 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
I0513 13:48:17.007586 1 controllermanager.go:533] Started "resourcequota"
I0513 13:48:17.007899 1 resource_quota_controller.go:272] Starting resource quota controller
I0513 13:48:17.007990 1 shared_informer.go:223] Waiting for caches to sync for resource quota
I0513 13:48:17.008085 1 resource_quota_monitor.go:303] QuotaMonitor running
I0513 13:48:17.025561 1 controllermanager.go:533] Started "horizontalpodautoscaling"
I0513 13:48:17.025639 1 horizontal.go:169] Starting HPA controller
I0513 13:48:17.025646 1 shared_informer.go:223] Waiting for caches to sync for HPA
I0513 13:48:17.028892 1 controllermanager.go:533] Started "cronjob"
W0513 13:48:17.028906 1 controllermanager.go:525] Skipping "root-ca-cert-publisher"
I0513 13:48:17.028969 1 cronjob_controller.go:97] Starting CronJob Manager
E0513 13:48:17.032924 1 core.go:89] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
W0513 13:48:17.032943 1 controllermanager.go:525] Skipping "service"
W0513 13:48:17.032950 1 core.go:243] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
W0513 13:48:17.032954 1 controllermanager.go:525] Skipping "route"
I0513 13:48:17.036873 1 controllermanager.go:533] Started "pvc-protection"
I0513 13:48:17.036963 1 pvc_protection_controller.go:101] Starting PVC protection controller
I0513 13:48:17.036972 1 shared_informer.go:223] Waiting for caches to sync for PVC protection
I0513 13:48:17.041054 1 controllermanager.go:533] Started "podgc"
I0513 13:48:17.041304 1 gc_controller.go:89] Starting GC controller
I0513 13:48:17.041386 1 shared_informer.go:223] Waiting for caches to sync for GC
I0513 13:48:17.046326 1 controllermanager.go:533] Started "serviceaccount"
I0513 13:48:17.046345 1 serviceaccounts_controller.go:117] Starting service account controller
I0513 13:48:17.046353 1 shared_informer.go:223] Waiting for caches to sync for service account
I0513 13:48:17.050317 1 controllermanager.go:533] Started "daemonset"
I0513 13:48:17.050353 1 daemon_controller.go:257] Starting daemon sets controller
I0513 13:48:17.050359 1 shared_informer.go:223] Waiting for caches to sync for daemon sets
I0513 13:48:17.161294 1 controllermanager.go:533] Started "bootstrapsigner"
I0513 13:48:17.161384 1 shared_informer.go:223] Waiting for caches to sync for bootstrap_signer
I0513 13:48:17.970616 1 garbagecollector.go:133] Starting garbage collector controller
I0513 13:48:17.970633 1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I0513 13:48:17.970647 1 graph_builder.go:282] GraphBuilder running
I0513 13:48:17.970701 1 controllermanager.go:533] Started "garbagecollector"
I0513 13:48:17.978505 1 controllermanager.go:533] Started "disruption"
I0513 13:48:17.978522 1 disruption.go:331] Starting disruption controller
I0513 13:48:17.978528 1 shared_informer.go:223] Waiting for caches to sync for disruption
I0513 13:48:17.984783 1 node_ipam_controller.go:94] Sending events to api server.
I0513 13:48:18.154203 1 request.go:621] Throttling request took 1.047600583s, request: GET:https://172.17.0.2:8443/apis/coordination.k8s.io/v1?timeout=32s
==> kube-controller-manager [d921b5840eed] <==
I0513 13:41:50.383825 1 shared_informer.go:223] Waiting for caches to sync for PV protection
I0513 13:41:50.533245 1 controllermanager.go:533] Started "job"
I0513 13:41:50.533296 1 job_controller.go:144] Starting job controller
I0513 13:41:50.533304 1 shared_informer.go:223] Waiting for caches to sync for job
I0513 13:41:50.684356 1 controllermanager.go:533] Started "replicaset"
I0513 13:41:50.684386 1 replica_set.go:181] Starting replicaset controller
I0513 13:41:50.684400 1 shared_informer.go:223] Waiting for caches to sync for ReplicaSet
I0513 13:41:50.833478 1 controllermanager.go:533] Started "tokencleaner"
I0513 13:41:50.833492 1 tokencleaner.go:118] Starting token cleaner controller
I0513 13:41:50.833707 1 shared_informer.go:223] Waiting for caches to sync for token_cleaner
I0513 13:41:50.833716 1 shared_informer.go:230] Caches are synced for token_cleaner
I0513 13:41:50.983102 1 node_lifecycle_controller.go:78] Sending events to api server
E0513 13:41:50.983325 1 core.go:229] failed to start cloud node lifecycle controller: no cloud provider provided
W0513 13:41:50.983397 1 controllermanager.go:525] Skipping "cloud-node-lifecycle"
I0513 13:41:51.134190 1 controllermanager.go:533] Started "pvc-protection"
I0513 13:41:51.134223 1 pvc_protection_controller.go:101] Starting PVC protection controller
I0513 13:41:51.134258 1 shared_informer.go:223] Waiting for caches to sync for PVC protection
I0513 13:41:51.138337 1 shared_informer.go:223] Waiting for caches to sync for resource quota
I0513 13:41:51.143279 1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I0513 13:41:51.148068 1 shared_informer.go:230] Caches are synced for endpoint_slice
W0513 13:41:51.148530 1 actual_state_of_world.go:506] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="test" does not exist
I0513 13:41:51.148564 1 shared_informer.go:230] Caches are synced for persistent volume
I0513 13:41:51.151279 1 shared_informer.go:230] Caches are synced for GC
I0513 13:41:51.154967 1 shared_informer.go:230] Caches are synced for service account
I0513 13:41:51.158308 1 shared_informer.go:230] Caches are synced for stateful set
I0513 13:41:51.183595 1 shared_informer.go:230] Caches are synced for HPA
I0513 13:41:51.183929 1 shared_informer.go:230] Caches are synced for PV protection
I0513 13:41:51.184000 1 shared_informer.go:230] Caches are synced for certificate-csrapproving
I0513 13:41:51.184529 1 shared_informer.go:230] Caches are synced for ReplicaSet
I0513 13:41:51.187247 1 shared_informer.go:230] Caches are synced for deployment
I0513 13:41:51.192669 1 shared_informer.go:230] Caches are synced for certificate-csrsigning
I0513 13:41:51.209102 1 shared_informer.go:230] Caches are synced for TTL
I0513 13:41:51.212449 1 shared_informer.go:230] Caches are synced for bootstrap_signer
I0513 13:41:51.219101 1 shared_informer.go:230] Caches are synced for expand
I0513 13:41:51.223030 1 shared_informer.go:230] Caches are synced for endpoint
I0513 13:41:51.229806 1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"test-service", UID:"8da9befa-4b23-48b6-a7c1-927740d87a3d", APIVersion:"v1", ResourceVersion:"1243", FieldPath:""}): type: 'Warning' reason: 'FailedToUpdateEndpoint' Failed to update endpoint default/test-service: Operation cannot be fulfilled on endpoints "test-service": the object has been modified; please apply your changes to the latest version and try again
I0513 13:41:51.233417 1 shared_informer.go:230] Caches are synced for job
I0513 13:41:51.234359 1 shared_informer.go:230] Caches are synced for PVC protection
I0513 13:41:51.236783 1 shared_informer.go:230] Caches are synced for node
I0513 13:41:51.236802 1 range_allocator.go:172] Starting range CIDR allocator
I0513 13:41:51.236804 1 shared_informer.go:223] Waiting for caches to sync for cidrallocator
I0513 13:41:51.236807 1 shared_informer.go:230] Caches are synced for cidrallocator
I0513 13:41:51.245917 1 shared_informer.go:230] Caches are synced for namespace
I0513 13:41:51.343110 1 shared_informer.go:230] Caches are synced for ClusterRoleAggregator
I0513 13:41:51.365747 1 shared_informer.go:230] Caches are synced for taint
I0513 13:41:51.365804 1 node_lifecycle_controller.go:1433] Initializing eviction metric for zone:
I0513 13:41:51.365826 1 taint_manager.go:187] Starting NoExecuteTaintManager
W0513 13:41:51.365849 1 node_lifecycle_controller.go:1048] Missing timestamp for Node test. Assuming now as a timestamp.
I0513 13:41:51.365870 1 node_lifecycle_controller.go:1249] Controller detected that zone is now in state Normal.
I0513 13:41:51.366091 1 event.go:278] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"test", UID:"aed9ac24-52d4-467b-bbba-09384311ce79", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node test event: Registered Node test in Controller
I0513 13:41:51.373392 1 shared_informer.go:230] Caches are synced for daemon sets
I0513 13:41:51.536338 1 shared_informer.go:230] Caches are synced for ReplicationController
I0513 13:41:51.603310 1 shared_informer.go:230] Caches are synced for disruption
I0513 13:41:51.603325 1 disruption.go:339] Sending events to api server.
I0513 13:41:51.604178 1 shared_informer.go:230] Caches are synced for resource quota
I0513 13:41:51.640676 1 shared_informer.go:230] Caches are synced for resource quota
I0513 13:41:51.743579 1 shared_informer.go:230] Caches are synced for garbage collector
I0513 13:41:51.769914 1 shared_informer.go:230] Caches are synced for attach detach
I0513 13:41:51.831938 1 shared_informer.go:230] Caches are synced for garbage collector
I0513 13:41:51.831970 1 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
==> kube-proxy [503fbb40f73e] <==
W0513 13:41:22.015364 1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
I0513 13:41:22.020570 1 node.go:136] Successfully retrieved node IP: 172.17.0.3
I0513 13:41:22.020592 1 server_others.go:186] Using iptables Proxier.
I0513 13:41:22.020847 1 server.go:583] Version: v1.18.0
I0513 13:41:22.021270 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0513 13:41:22.021342 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0513 13:41:22.021386 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0513 13:41:22.021728 1 config.go:133] Starting endpoints config controller
I0513 13:41:22.021813 1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0513 13:41:22.021874 1 config.go:315] Starting service config controller
I0513 13:41:22.021900 1 shared_informer.go:223] Waiting for caches to sync for service config
I0513 13:41:22.122030 1 shared_informer.go:230] Caches are synced for service config
I0513 13:41:22.122030 1 shared_informer.go:230] Caches are synced for endpoints config
==> kube-scheduler [45171cca5401] <==
I0513 13:47:54.475620 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:47:54.475678 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:47:54.946020 1 serving.go:313] Generated self-signed cert in-memory
W0513 13:47:57.920022 1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0513 13:47:57.920040 1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0513 13:47:57.920046 1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous.
W0513 13:47:57.920049 1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0513 13:47:57.935690 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:47:57.935704 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0513 13:47:57.936700 1 authorization.go:47] Authorization is disabled
W0513 13:47:57.936711 1 authentication.go:40] Authentication is disabled
I0513 13:47:57.936718 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0513 13:47:57.938037 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259
I0513 13:47:57.939983 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0513 13:47:57.940029 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:47:57.940038 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:47:58.038911 1 leaderelection.go:242] attempting to acquire leader lease kube-system/kube-scheduler...
I0513 13:47:58.040404 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:48:14.763483 1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler
==> kube-scheduler [cf9a5808a6ee] <==
I0513 13:41:17.179962 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:41:17.180006 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:41:17.591172 1 serving.go:313] Generated self-signed cert in-memory
W0513 13:41:20.779203 1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0513 13:41:20.782051 1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0513 13:41:20.782071 1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous.
W0513 13:41:20.782076 1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0513 13:41:20.844274 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0513 13:41:20.844297 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0513 13:41:20.845663 1 authorization.go:47] Authorization is disabled
W0513 13:41:20.845678 1 authentication.go:40] Authentication is disabled
I0513 13:41:20.845685 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0513 13:41:20.847070 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259
I0513 13:41:20.847162 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:41:20.847168 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:41:20.847193 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0513 13:41:20.947279 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0513 13:41:20.947323 1 leaderelection.go:242] attempting to acquire leader lease kube-system/kube-scheduler...
I0513 13:41:39.021060 1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler
E0513 13:47:22.073139 1 reflector.go:380] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: Get https://172.17.0.3:8443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=1303&timeout=7m31s&timeoutSeconds=451&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073473 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.PersistentVolumeClaim: Get https://172.17.0.3:8443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=1303&timeout=9m57s&timeoutSeconds=597&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073527 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Service: Get https://172.17.0.3:8443/api/v1/services?allowWatchBookmarks=true&resourceVersion=1303&timeout=8m10s&timeoutSeconds=490&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073564 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Node: Get https://172.17.0.3:8443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2107&timeout=7m35s&timeoutSeconds=455&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073598 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1beta1.PodDisruptionBudget: Get https://172.17.0.3:8443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=1303&timeout=9m5s&timeoutSeconds=545&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073738 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.StorageClass: Get https://172.17.0.3:8443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=1303&timeout=7m37s&timeoutSeconds=457&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073776 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.CSINode: Get https://172.17.0.3:8443/apis/storage.k8s.io/v1/csinodes?allowWatchBookmarks=true&resourceVersion=1303&timeout=9m18s&timeoutSeconds=558&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073806 1 reflector.go:380] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.PersistentVolume: Get https://172.17.0.3:8443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=1303&timeout=5m24s&timeoutSeconds=324&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
E0513 13:47:22.073829 1 reflector.go:380] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:233: Failed to watch *v1.Pod: Get https://172.17.0.3:8443/api/v1/pods?allowWatchBookmarks=true&fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=2094&timeoutSeconds=395&watch=true: dial tcp 172.17.0.3:8443: connect: connection refused
==> kubelet <==
-- Logs begin at Wed 2020-05-13 13:47:37 UTC, end at Wed 2020-05-13 13:48:24 UTC. --
May 13 13:48:19 test kubelet[706]: E0513 13:48:19.587433 706 kubelet.go:2267] node "test" not found
May 13 13:48:19 test kubelet[706]: E0513 13:48:19.687706 706 kubelet.go:2267] node "test" not found
May 13 13:48:19 test kubelet[706]: E0513 13:48:19.787851 706 kubelet.go:2267] node "test" not found
May 13 13:48:19 test kubelet[706]: E0513 13:48:19.888298 706 kubelet.go:2267] node "test" not found
May 13 13:48:19 test kubelet[706]: E0513 13:48:19.988472 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.088679 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.188852 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.289059 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.389315 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.489615 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.589803 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.690080 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.790277 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.890551 706 kubelet.go:2267] node "test" not found
May 13 13:48:20 test kubelet[706]: E0513 13:48:20.990786 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.090970 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.191181 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.291476 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.391668 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.491946 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.592119 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.690672 706 eviction_manager.go:255] eviction manager: failed to get summary stats: failed to get node info: node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.692261 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: W0513 13:48:21.743270 706 status_manager.go:556] Failed to get status for pod "etcd-test_kube-system(6faa35c64253a217ed2b083ff9c6366b)": Get https://172.17.0.3:8443/api/v1/namespaces/kube-system/pods/etcd-test: dial tcp 172.17.0.3:8443: connect: no route to host
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.743285 706 event.go:269] Unable to write event: 'Post https://172.17.0.3:8443/api/v1/namespaces/default/events: dial tcp 172.17.0.3:8443: connect: no route to host' (may retry after sleeping)
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.743528 706 kubelet_node_status.go:92] Unable to register node "test" with API server: Post https://172.17.0.3:8443/api/v1/nodes: dial tcp 172.17.0.3:8443: connect: no route to host
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.743599 706 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: Get https://172.17.0.3:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test?timeout=10s: dial tcp 172.17.0.3:8443: connect: no route to host
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.792687 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.892818 706 kubelet.go:2267] node "test" not found
May 13 13:48:21 test kubelet[706]: E0513 13:48:21.992997 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.093133 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.193290 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.293427 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.393579 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.493687 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.593743 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.693897 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.794131 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.894256 706 kubelet.go:2267] node "test" not found
May 13 13:48:22 test kubelet[706]: E0513 13:48:22.994392 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.094513 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.194656 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.294788 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.394935 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.495041 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.595125 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.695271 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.795701 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.895838 706 kubelet.go:2267] node "test" not found
May 13 13:48:23 test kubelet[706]: E0513 13:48:23.995996 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.096109 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.196215 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.296344 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.396459 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.496605 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.596716 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.696789 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: W0513 13:48:24.779861 706 status_manager.go:556] Failed to get status for pod "kube-apiserver-test_kube-system(112c60df9e36eeaf13a6dd3074765810)": Get https://172.17.0.3:8443/api/v1/namespaces/kube-system/pods/kube-apiserver-test: dial tcp 172.17.0.3:8443: connect: no route to host
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.796888 706 kubelet.go:2267] node "test" not found
May 13 13:48:24 test kubelet[706]: E0513 13:48:24.897031 706 kubelet.go:2267] node "test" not found
==> storage-provisioner [171780a64b20] <==
E0513 13:47:22.074042 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=13, ErrCode=NO_ERROR, debug=""
E0513 13:47:22.074055 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=13, ErrCode=NO_ERROR, debug=""
E0513 13:47:22.074075 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=13, ErrCode=NO_ERROR, debug=""
E0513 13:47:22.074461 1 reflector.go:315] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:412: Failed to watch *v1.PersistentVolume: Get https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=1303&timeoutSeconds=516&watch=true: dial tcp 10.96.0.1:443: getsockopt: connection refused
E0513 13:47:22.075451 1 reflector.go:315] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to watch *v1.PersistentVolumeClaim: Get https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=1303&timeoutSeconds=401&watch=true: dial tcp 10.96.0.1:443: getsockopt: connection refused
E0513 13:47:22.075488 1 reflector.go:315] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:379: Failed to watch *v1.StorageClass: Get https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=1303&timeoutSeconds=598&watch=true: dial tcp 10.96.0.1:443: getsockopt: connection refused
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 17 Aug 2020
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
on 16 Sep 2020
Is anyone still seeing this bug with minikube version 1.13.1?
priyawadhwa
on 23 Sep 2020
I still see this issue on v1.15.1
andrewshilliday
on 27 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
alber70g
·
3Comments
tnine
·
3Comments
Starefossen
·
3Comments
mdkess
·
3Comments
tstromberg
·
3Comments