Failed to execute microk8s enable storage
Output is as follows
Enabling default storage class.
sudo: unable to dlopen /usr/libexec/sudoers.so: (null).
sudo: fatal error, unable to load plugins.
Failed to enable storage
I have made sure that the sudoer.so library can be loaded using dlopen in c++ and the sudo command works fine.
Wish for your help!!!
github1youlc
All 13 comments
Thanks for using MicroK8s. I find that There are other issues in the log. Most especially kube-proxy.
For example:
Sep 12 08:42:09 localhost microk8s.daemon-proxy[15883]: W0912 08:42:09.562833 15883 iptables.go:550] Could not set up iptables canary nat/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 3: iptables v1.6.1: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Do you want to tell us more about the OS?
And can you reboot the system?
balchua
on 14 Sep 2020
Thanks for using MicroK8s. I find that There are other issues in the log. Most especially
kube-proxy.For example:
Sep 12 08:42:09 localhost microk8s.daemon-proxy[15883]: W0912 08:42:09.562833 15883 iptables.go:550] Could not set up iptables canary nat/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 3: iptables v1.6.1: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)Do you want to tell us more about the OS?
And can you reboot the system?
That error occured when executing microk8s enable dns, I have solved the problem by installing the nat library.
OS is based on centos, but there are some customization by my company.
cat /etc/centos-release
CentOS Linux release 7.2 (Final)
After rebooting the system, I still can not enable storage.
By the way, dashboard, dns, ingress etc works well.
github1youlc
on 14 Sep 2020
Just wanted to try this. Can you do microk8s enable helm3, just to see if you also get the same issue.
Thanks
balchua
on 14 Sep 2020
Just wanted to try this. Can you do
microk8s enable helm3, just to see if you also get the same issue.
Thanks
It works. And I can use it to install helm charts.
github1youlc
on 14 Sep 2020
Ok thanks. I think the script enable-storage.sh isn't taking the system LD_LIBRARY_PATH like it is done for other enable script.
The enable-storage.sh should be using this [1], instead of just calling sudo.
Thanks for reporting this.
For the meantime, to enable the storage,
Create the directory where the hostpath will keep the images.
$ sudo mkdir -p /var/snap/microk8s/common/default-storage
Then apply the manifest below.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hostpath-provisioner
labels:
k8s-app: hostpath-provisioner
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
k8s-app: hostpath-provisioner
template:
metadata:
labels:
k8s-app: hostpath-provisioner
spec:
serviceAccountName: microk8s-hostpath
containers:
- name: hostpath-provisioner
image: cdkbot/hostpath-provisioner-amd64:1.0.0
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: PV_DIR
value: /var/snap/microk8s/common/default-storage
volumeMounts:
- name: pv-volume
mountPath: /var/snap/microk8s/common/default-storage
volumes:
- name: pv-volume
hostPath:
path: /var/snap/microk8s/common/default-storage
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: microk8s-hostpath
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: microk8s.io/hostpath
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: microk8s-hostpath
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: microk8s-hostpath
rules:
- apiGroups: [""]
resources:
- persistentvolumeclaims
verbs:
- list
- get
- watch
- update
- apiGroups: [""]
resources:
- persistentvolumes
verbs:
- list
- get
- update
- watch
- create
- delete
- apiGroups: [""]
resources:
- events
verbs:
- create
- list
- patch
- apiGroups: ["storage.k8s.io"]
resources:
- storageclasses
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: microk8s-hostpath
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: microk8s-hostpath
subjects:
- kind: ServiceAccount
name: microk8s-hostpath
namespace: kube-system
Ok thanks. I think the script
enable-storage.shisn't taking the systemLD_LIBRARY_PATHlike it is done for other enable script.The
enable-storage.shshould be using this [1], instead of just callingsudo.
Thanks for reporting this.For the meantime, to enable the storage,
Create the directory where the hostpath will keep the images.$ sudo mkdir -p /var/snap/microk8s/common/default-storageThen apply the manifest below.
apiVersion: apps/v1 kind: Deployment metadata: name: hostpath-provisioner labels: k8s-app: hostpath-provisioner namespace: kube-system spec: replicas: 1 revisionHistoryLimit: 0 selector: matchLabels: k8s-app: hostpath-provisioner template: metadata: labels: k8s-app: hostpath-provisioner spec: serviceAccountName: microk8s-hostpath containers: - name: hostpath-provisioner image: cdkbot/hostpath-provisioner-amd64:1.0.0 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: PV_DIR value: /var/snap/microk8s/common/default-storage volumeMounts: - name: pv-volume mountPath: /var/snap/microk8s/common/default-storage volumes: - name: pv-volume hostPath: path: /var/snap/microk8s/common/default-storage --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: microk8s-hostpath annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: microk8s.io/hostpath --- apiVersion: v1 kind: ServiceAccount metadata: name: microk8s-hostpath namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: microk8s-hostpath rules: - apiGroups: [""] resources: - persistentvolumeclaims verbs: - list - get - watch - update - apiGroups: [""] resources: - persistentvolumes verbs: - list - get - update - watch - create - delete - apiGroups: [""] resources: - events verbs: - create - list - patch - apiGroups: ["storage.k8s.io"] resources: - storageclasses verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: microk8s-hostpath roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: microk8s-hostpath subjects: - kind: ServiceAccount name: microk8s-hostpath namespace: kube-system[1]
It works. If set LD_LIBRARY_PATH, will the problem be avoided?
github1youlc
on 14 Sep 2020
Sorry to say, I found another problem, the node is in NotReady status , which may caused by rebooting.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
localhost NotReady <none> 2d7h v1.18.8
The detail is as follows
$ kubectl describe node localhost
Name: localhost
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=localhost
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 12 Sep 2020 08:21:36 +0800
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: localhost
AcquireTime: <unset>
RenewTime: Mon, 14 Sep 2020 09:40:25 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Mon, 14 Sep 2020 09:39:30 +0800 Mon, 14 Sep 2020 09:43:08 +0800 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Mon, 14 Sep 2020 09:39:30 +0800 Mon, 14 Sep 2020 09:43:08 +0800 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Mon, 14 Sep 2020 09:39:30 +0800 Mon, 14 Sep 2020 09:43:08 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Mon, 14 Sep 2020 09:39:30 +0800 Mon, 14 Sep 2020 09:43:08 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 9.134.67.201
Hostname: localhost
Capacity:
cpu: 8
ephemeral-storage: 103079868Ki
hugepages-2Mi: 0
memory: 16166056Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 102031292Ki
hugepages-2Mi: 0
memory: 16063656Ki
pods: 110
System Info:
Machine ID: 4211ff3f594041f3966d836585a11a05
System UUID: 1E8282AA-7144-F54A-9A06-B0C8773AFBF4
Boot ID: a68b9a34-109c-47a6-88a4-14e38e66d432
Kernel Version: 3.10.107-1-tlinux2_kvm_guest-0048
OS Image: Tencent tlinux 2.2 (Final)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.18.8
Kube-Proxy Version: v1.18.8
Non-terminated Pods: (13 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default hbase-hbase-master-0 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hbase-regionserver-0 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hdfs-datanode-0 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hdfs-datanode-1 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hdfs-datanode-2 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hdfs-httpfs-56497f4bd5-m88cf 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
default hbase-hdfs-namenode-0 10m (0%) 1 (12%) 256Mi (1%) 2Gi (13%) 16h
ingress nginx-ingress-microk8s-controller-c9xxj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 25h
kube-system coredns-588fd544bf-dcvjw 100m (1%) 0 (0%) 70Mi (0%) 170Mi (1%) 2d7h
kube-system dashboard-metrics-scraper-db65b9c6f-zsxtd 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d6h
kube-system heapster-v1.5.2-58fdbb6f4d-7r8ft 100m (1%) 100m (1%) 184720Ki (1%) 184720Ki (1%) 2d6h
kube-system kubernetes-dashboard-67765b55f5-kd7rt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d6h
kube-system monitoring-influxdb-grafana-v4-6dc675bf8c-x9r7t 200m (2%) 200m (2%) 600Mi (3%) 600Mi (3%) 2d6h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 470m (5%) 7300m (91%)
memory 2705808Ki (16%) 15653264Ki (97%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
@github1youlc @balchua thank you for finding the storage bug. There are PRs up to fix this bug on the 1.19, 1.18 and 1.17 releases.
@github1youlc can you attach the inspection report tarball (microk8s inspect) so we see why kubelet will not post the node's state?
ktsakalozos
on 14 Sep 2020
@github1youlc @balchua thank you for finding the storage bug. There are PRs up to fix this bug on the 1.19, 1.18 and 1.17 releases.
@github1youlc can you attach the inspection report tarball (
microk8s inspect) so we see why kubelet will not post the node's state?
inspection-report-20200914_183520.tar.gz
Thank you.
github1youlc
on 14 Sep 2020
@github1youlc
- The nodename has changed from localhost to
VM_67_201_centos. - Can you set the hostname to all small letters
vm_67_201_centos
Stop and start microk8s.
balchua
on 14 Sep 2020
The underscores are also not good to have them in the hostname, right @balchua?
ktsakalozos
on 14 Sep 2020
You're right. Usually names follow this pattern
An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters, with the '-' character allowed anywhere except the first or last character, suitable for use as a hostname or segment in a domain name.
Taken from here
balchua
on 14 Sep 2020
All problem solved, thank you very much. @balchua @ktsakalozos
github1youlc
on 14 Sep 2020
Related issues
toxsick
路
4Comments
alphawolf1988
路
3Comments
linxuyalun
路
3Comments
tvansteenburgh
路
6Comments
arno01
路
4Comments
Most helpful comment
Ok thanks. I think the script
enable-storage.shisn't taking the systemLD_LIBRARY_PATHlike it is done for other enable script.The
enable-storage.shshould be using this [1], instead of just callingsudo.Thanks for reporting this.
For the meantime, to enable the storage,
Create the directory where the hostpath will keep the images.
Then apply the manifest below.
[1] https://github.com/ubuntu/microk8s/blob/22a970b03a7b24ff8f733d5a91475a1babcb323a/microk8s-resources/actions/common/utils.sh#L70