Longhorn: [BUG] [ARM] Longhorn refuses to create a PVC bigger than 1.9G
Describe the bug
I cannot provision a PVC using longhorn storage class larger than 1.9G
To Reproduce
Apply the below manifests (from examples):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: longhorn-volv-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: volume-test
namespace: default
spec:
restartPolicy: Always
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- ls
- /data/lost+found
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: volv
mountPath: /data
ports:
- containerPort: 80
volumes:
- name: volv
persistentVolumeClaim:
claimName: longhorn-volv-pvc
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: storage
operator: In
values:
- enabled
Expected behavior
Exec into the pod and run df -h I should see the /data mountpoint with ~10G filesystem. Instead, as shown below, I see 1.9G
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 58.3G 8.4G 47.5G 15% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/longhorn/pvc-7487fc6a-1357-4741-9855-c79f9707e826
1.9G 6.0M 1.9G 0% /data
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/hosts
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /dev/termination-log
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/hostname
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/resolv.conf
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 3.8G 12.0K 3.8G 0% /run/secrets/kubernetes.io/serviceaccount
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/latency_stats
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 3.8G 0 3.8G 0% /proc/scsi
tmpfs 3.8G 0 3.8G 0% /sys/firmware
/ #
Log
N/A
Environment:
- Longhorn version: v.1.0.1
- Kubernetes version:
Client Version: v1.19.0-beta.0
Server Version: v1.18.6+k3s1 - Node OS type and version:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
worker4 Ready <none> 21d v1.18.6+k3s1 10.40.0.14 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
worker7 Ready <none> 19d v1.18.6+k3s1 10.40.0.17 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker3 Ready <none> 21d v1.18.6+k3s1 10.40.0.13 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
worker5 Ready <none> 21d v1.18.6+k3s1 10.40.0.15 <none> Ubuntu 20.04.1 LTS 5.4.0-1016-raspi containerd://1.3.3-k3s2
worker6 Ready <none> 19d v1.18.6+k3s1 10.40.0.16 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
ctrl1 Ready master 21d v1.18.6+k3s1 10.40.0.101 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker1 Ready <none> 21d v1.18.6+k3s1 10.40.0.11 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker2 Ready <none> 21d v1.18.6+k3s1 10.40.0.12 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
Additional context
Longhorn UI shows no errors, all replicas healthy. Running on as rasberry pi 4 cluster.
PVC yaml dump
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"longhorn-volv-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"longhorn"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
creationTimestamp: "2020-10-02T21:10:10Z"
finalizers:
- kubernetes.io/pvc-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
manager: kubectl-client-side-apply
operation: Update
time: "2020-10-02T21:10:10Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:pv.kubernetes.io/bind-completed: {}
f:pv.kubernetes.io/bound-by-controller: {}
f:volume.beta.kubernetes.io/storage-provisioner: {}
f:spec:
f:volumeName: {}
f:status:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:phase: {}
manager: k3s
operation: Update
time: "2020-10-02T21:10:12Z"
name: longhorn-volv-pvc
namespace: default
resourceVersion: "5009265"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/longhorn-volv-pvc
uid: 7487fc6a-1357-4741-9855-c79f9707e826
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
volumeMode: Filesystem
volumeName: pvc-7487fc6a-1357-4741-9855-c79f9707e826
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
PV yaml dump
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: driver.longhorn.io
creationTimestamp: "2020-10-02T21:10:12Z"
finalizers:
- kubernetes.io/pv-protection
- external-attacher/driver-longhorn-io
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/provisioned-by: {}
f:spec:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:claimRef:
.: {}
f:apiVersion: {}
f:kind: {}
f:name: {}
f:namespace: {}
f:resourceVersion: {}
f:uid: {}
f:csi:
.: {}
f:driver: {}
f:fsType: {}
f:volumeAttributes:
.: {}
f:fromBackup: {}
f:numberOfReplicas: {}
f:staleReplicaTimeout: {}
f:storage.kubernetes.io/csiProvisionerIdentity: {}
f:volumeHandle: {}
f:persistentVolumeReclaimPolicy: {}
f:storageClassName: {}
f:volumeMode: {}
manager: csi-provisioner
operation: Update
time: "2020-10-02T21:10:12Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2020-10-02T21:10:12Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
v:"external-attacher/driver-longhorn-io": {}
manager: csi-attacher
operation: Update
time: "2020-10-02T21:10:13Z"
name: pvc-7487fc6a-1357-4741-9855-c79f9707e826
resourceVersion: "5009273"
selfLink: /api/v1/persistentvolumes/pvc-7487fc6a-1357-4741-9855-c79f9707e826
uid: 6e99957a-b217-4eda-b14a-d9d4d18d16d0
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: longhorn-volv-pvc
namespace: default
resourceVersion: "5009223"
uid: 7487fc6a-1357-4741-9855-c79f9707e826
csi:
driver: driver.longhorn.io
fsType: ext4
volumeAttributes:
fromBackup: ""
numberOfReplicas: "3"
staleReplicaTimeout: "2880"
storage.kubernetes.io/csiProvisionerIdentity: 1599863377275-8081-driver.longhorn.io
volumeHandle: pvc-7487fc6a-1357-4741-9855-c79f9707e826
persistentVolumeReclaimPolicy: Delete
storageClassName: longhorn
volumeMode: Filesystem
status:
phase: Bound
Longhorn deployment manifests
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn
parameters:
fromBackup: ""
numberOfReplicas: "3"
staleReplicaTimeout: "2880"
provisioner: driver.longhorn.io
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: EngineImage
name: engineimages.longhorn.io
spec:
group: longhorn.io
names:
kind: EngineImage
listKind: EngineImageList
plural: engineimages
shortNames:
- lhei
singular: engineimage
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Engine
name: engines.longhorn.io
spec:
group: longhorn.io
names:
kind: Engine
listKind: EngineList
plural: engines
shortNames:
- lhe
singular: engine
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: InstanceManager
name: instancemanagers.longhorn.io
spec:
group: longhorn.io
names:
kind: InstanceManager
listKind: InstanceManagerList
plural: instancemanagers
shortNames:
- lhim
singular: instancemanager
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Node
name: nodes.longhorn.io
spec:
group: longhorn.io
names:
kind: Node
listKind: NodeList
plural: nodes
shortNames:
- lhn
singular: node
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Replica
name: replicas.longhorn.io
spec:
group: longhorn.io
names:
kind: Replica
listKind: ReplicaList
plural: replicas
shortNames:
- lhr
singular: replica
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Setting
name: settings.longhorn.io
spec:
group: longhorn.io
names:
kind: Setting
listKind: SettingList
plural: settings
shortNames:
- lhs
singular: setting
scope: Namespaced
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Volume
name: volumes.longhorn.io
spec:
group: longhorn.io
names:
kind: Volume
listKind: VolumeList
plural: volumes
shortNames:
- lhv
singular: volume
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: longhorn-psp
spec:
allowPrivilegeEscalation: true
allowedCapabilities:
- SYS_ADMIN
fsGroup:
rule: RunAsAny
hostIPC: false
hostNetwork: false
hostPID: true
privileged: true
requiredDropCapabilities:
- NET_RAW
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- secret
- projected
- hostPath
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: longhorn-psp-role
namespace: longhorn-system
rules:
- apiGroups:
- policy
resourceNames:
- longhorn-psp
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: longhorn-role
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
- events
- persistentvolumes
- persistentvolumeclaims
- persistentvolumeclaims/status
- nodes
- proxy/nodes
- pods/log
- secrets
- services
- endpoints
- configmaps
verbs:
- '*'
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- apiGroups:
- apps
resources:
- daemonsets
- statefulsets
- deployments
verbs:
- '*'
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- '*'
- apiGroups:
- scheduling.k8s.io
resources:
- priorityclasses
verbs:
- watch
- list
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
- volumeattachments
- csinodes
- csidrivers
verbs:
- '*'
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- watch
- list
- delete
- update
- create
- apiGroups:
- longhorn.io
resources:
- volumes
- volumes/status
- engines
- engines/status
- replicas
- replicas/status
- settings
- engineimages
- engineimages/status
- nodes
- nodes/status
- instancemanagers
- instancemanagers/status
verbs:
- '*'
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: longhorn-psp-binding
namespace: longhorn-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: longhorn-psp-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
- kind: ServiceAccount
name: default
namespace: longhorn-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: longhorn-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: longhorn-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: v1
data:
default-setting.yaml: |-
backup-target:
backup-target-credential-secret:
create-default-disk-labeled-nodes: true
default-data-path: /data/disk0
replica-soft-anti-affinity:
storage-over-provisioning-percentage:
storage-minimal-available-percentage:
upgrade-checker:
default-replica-count: 3
guaranteed-engine-cpu: 0.125
default-longhorn-static-storage-class:
backupstore-poll-interval:
taint-toleration:
priority-class:
registry-secret:
auto-salvage:
disable-scheduling-on-cordoned-node:
replica-zone-soft-anti-affinity:
volume-attachment-recovery-policy:
mkfs-ext4-parameters:
kind: ConfigMap
metadata:
name: longhorn-default-setting
namespace: longhorn-system
---
apiVersion: v1
kind: Service
metadata:
labels:
app: longhorn-manager
name: longhorn-backend
namespace: longhorn-system
spec:
ports:
- name: manager
port: 9500
targetPort: manager
selector:
app: longhorn-manager
sessionAffinity: ClientIP
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
labels:
app: longhorn-ui
name: longhorn-frontend
namespace: longhorn-system
spec:
ports:
- name: http
nodePort: null
port: 80
targetPort: http
selector:
app: longhorn-ui
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: longhorn-driver-deployer
namespace: longhorn-system
spec:
replicas: 1
selector:
matchLabels:
app: longhorn-driver-deployer
template:
metadata:
labels:
app: longhorn-driver-deployer
spec:
containers:
- command:
- longhorn-manager
- -d
- deploy-driver
- --manager-image
- vsellier/longhorn-manager:v1.0.1
- --manager-url
- http://longhorn-backend:9500/v1
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: CSI_ATTACHER_IMAGE
value: vsellier/csi-attacher:v2.0.0
- name: CSI_PROVISIONER_IMAGE
value: vsellier/csi-provisioner:v1.4.0
- name: CSI_NODE_DRIVER_REGISTRAR_IMAGE
value: vsellier/csi-node-driver-registrar:v1.2.0
- name: CSI_RESIZER_IMAGE
value: vsellier/csi-resizer:v0.3.0
image: vsellier/longhorn-manager:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-driver-deployer
initContainers:
- command:
- sh
- -c
- while [ $(curl -m 1 -s -o /dev/null -w "%{http_code}" http://longhorn-backend:9500/v1) != "200" ]; do echo waiting; sleep 2; done
image: vsellier/longhorn-manager:v1.0.1
name: wait-longhorn-manager
securityContext:
runAsUser: 0
serviceAccountName: longhorn-service-account
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: longhorn-ui
name: longhorn-ui
namespace: longhorn-system
spec:
replicas: 1
selector:
matchLabels:
app: longhorn-ui
template:
metadata:
labels:
app: longhorn-ui
spec:
containers:
- env:
- name: LONGHORN_MANAGER_IP
value: http://longhorn-backend:9500
image: vsellier/longhorn-ui:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-ui
ports:
- containerPort: 8000
name: http
securityContext:
runAsUser: 0
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: longhorn-manager
name: longhorn-manager
namespace: longhorn-system
spec:
selector:
matchLabels:
app: longhorn-manager
template:
metadata:
labels:
app: longhorn-manager
spec:
containers:
- command:
- longhorn-manager
- -d
- daemon
- --engine-image
- vsellier/longhorn-engine:v1.0.1
- --instance-manager-image
- vsellier/longhorn-instance-manager:v1_20200514_1
- --manager-image
- vsellier/longhorn-manager:v1.0.1
- --service-account
- longhorn-service-account
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: DEFAULT_SETTING_PATH
value: /var/lib/longhorn-setting/default-setting.yaml
image: vsellier/longhorn-manager:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-manager
ports:
- containerPort: 9500
name: manager
readinessProbe:
tcpSocket:
port: 9500
securityContext:
privileged: true
volumeMounts:
- mountPath: /host/dev/
name: dev
- mountPath: /host/proc/
name: proc
- mountPath: /var/run/
mountPropagation: Bidirectional
name: varrun
- mountPath: /var/lib/longhorn/
mountPropagation: Bidirectional
name: longhorn
- mountPath: /var/lib/longhorn-setting/
name: longhorn-default-setting
serviceAccountName: longhorn-service-account
volumes:
- hostPath:
path: /dev/
name: dev
- hostPath:
path: /proc/
name: proc
- hostPath:
path: /var/run/
name: varrun
- hostPath:
path: /var/lib/longhorn/
name: longhorn
- configMap:
name: longhorn-default-setting
name: longhorn-default-setting
updateStrategy:
rollingUpdate:
maxUnavailable: 100%
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod-cluster-issuer
kubernetes.io/ingress.class: nginx
name: longhorn-ingress
namespace: longhorn-system
spec:
rules:
- host: longhorn.k8s.b5y.gs
http:
paths:
- backend:
serviceName: longhorn-frontend
servicePort: 80
path: /
tls:
- hosts:
- longhorn.k8s.b5y.gs
secretName: tls-secret-le-longhorn
bradwood
All 26 comments
This doesn't make sense...
Can you check the dmesg of the attached node to see how big is the volume's raw size? Also lsblk should help too.
yasker
on 3 Oct 2020
I think i see the problem!
root@worker3:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 48.8M 1 loop /snap/core18/1888
loop1 7:1 0 48.5M 1 loop /snap/core18/1883
loop2 7:2 0 63M 1 loop /snap/lxd/16945
loop3 7:3 0 26.3M 1 loop /snap/snapd/9287
loop4 7:4 0 26M 1 loop /snap/snapd/8791
loop6 7:6 0 63M 1 loop /snap/lxd/16908
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.7G 0 part /data/disk0
sdb 8:16 0 2G 0 disk /var/lib/kubelet/pods/0ed679e2-604a-4ced-a9d5-ab75ed96be62/volumes/kubernetes.io~csi/pvc-7487fc6a-1357-4741-9855-c79f9707e
mmcblk0 179:0 0 59.5G 0 disk
├─mmcblk0p1
│ 179:1 0 256M 0 part /boot/firmware
└─mmcblk0p2
179:2 0 59.2G 0 part /
root@worker3:~#
it should be on /dev/sda1 but it looks like it's /dev/sdb... hmmmm
bradwood
on 3 Oct 2020
Does Longhorn UI also say it's 2G?
yasker
on 3 Oct 2020
10G
bradwood
on 3 Oct 2020
It's weird, perhaps, as the boot disk is a raspi SD card at /dev/mmd* and I have an NVMe at /dev/sda with 500G partition on /dev/sda1 -- not sure why /dev/sdb is being used at all!
bradwood
on 3 Oct 2020
Longhorn uses the iscsi initiator on the node so it's expected to create a new block device for it.
But how can it only get 2G?... Likely due to something in the ARM.
@boknowswiki is working on the ARM PR. @boknowswiki did you see anything like this in your setup?
yasker
on 3 Oct 2020
root@worker3:~# mount | grep sd
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
/dev/sda1 on /data/disk0 type ext4 (rw,relatime,stripe=8191)
sdb not mounted.
root@worker3:~# cat /etc/fstab
LABEL=writable / ext4 defaults 0 0
LABEL=system-boot /boot/firmware vfat defaults 0 1
/dev/sda1 /data/disk0 ext4 defaults 0 0
root@worker3:~#
Longhorn uses the iscsi initiator on the node so it's expected to create a new block device for it.
But how can it only get 2G?... Likely due to something in the ARM.
@boknowswiki is working on the ARM PR. @boknowswiki did you see anything like this in your setup?
I am using these arm images:
longhorn ❯ kustomize build beast | grep vsel 13m56s [master] ~/Code/b-k8s/longhorn
- vsellier/longhorn-manager:v1.0.1
value: vsellier/csi-attacher:v2.0.0
value: vsellier/csi-provisioner:v1.4.0
value: vsellier/csi-node-driver-registrar:v1.2.0
value: vsellier/csi-resizer:v0.3.0
image: vsellier/longhorn-manager:v1.0.1
image: vsellier/longhorn-manager:v1.0.1
image: vsellier/longhorn-ui:v1.0.1
- vsellier/longhorn-engine:v1.0.1
- vsellier/longhorn-instance-manager:v1_20200514_1
- vsellier/longhorn-manager:v1.0.1
image: vsellier/longhorn-manager:v1.0.1
Note, the are complied for arm and not arm64, I believe, if that has any bearing.
bradwood
on 3 Oct 2020
For mount, you can search for /dev/longhorn/pvc-7487fc6a-1357-4741-9855-c79f9707e826 instead of /dev/sdb(which are the same block device).
fstab won't be updated since the mountpoint of the device is managed by Kubernetes, not the node OS.
We will look into this. Also, it seems we need to prepare images for both arm and arm64.
yasker
on 3 Oct 2020
ok, thanks @yasker -- arm(64) images would be gratefully received!
bradwood
on 3 Oct 2020
@bradwood , do you mind to share the following docker image vsellier/csi-attacher:v2.0.0 information with us ? As you can see, the one I got it doesn't have arm architect binary in it. Thanks.
root@longhorn-arm-01:~# docker images --digests vsellier/csi-attacher:v2.0.0
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
vsellier/csi-attacher v2.0.0 <none> e2cbb7a8427d 2 months ago 37.9MB
root@longhorn-arm-01:~#
root@longhorn-arm-01:~#
root@longhorn-arm-01:~# docker manifest inspect --verbose vsellier/csi-attacher:v2.0.0
{
"Ref": "docker.io/vsellier/csi-attacher:v2.0.0",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:6f4076e0e0ee6c2130a9ba1dab697ec9cfb415c2f12a3e0d5f6cad9f49f35d58",
"size": 739,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
"SchemaV2Manifest": {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 2202,
"digest": "sha256:e2cbb7a8427d68ae8cc98277e82137c1f5fdbc3d12c28c1876015298fdf957cb"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 655317,
"digest": "sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 14994036,
"digest": "sha256:8ac74a7aa51e199dd90db8aff56c69020813bda8645c39c33531d51c90764808"
}
]
}
}
root@longhorn-arm-01:~#
boknowswiki
on 3 Oct 2020
mine is amd64 too!
root@mgmt1:~# DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect --verbose vsellier/csi-attacher:v2.0.0
{
"Ref": "docker.io/vsellier/csi-attacher:v2.0.0",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:6f4076e0e0ee6c2130a9ba1dab697ec9cfb415c2f12a3e0d5f6cad9f49f35d58",
"size": 739,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
"SchemaV2Manifest": {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 2202,
"digest": "sha256:e2cbb7a8427d68ae8cc98277e82137c1f5fdbc3d12c28c1876015298fdf957cb"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 655317,
"digest": "sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 14994036,
"digest": "sha256:8ac74a7aa51e199dd90db8aff56c69020813bda8645c39c33531d51c90764808"
}
]
}
}
root@mgmt1:~#
Hrm... I've just checked here too:
It's clearly an AMD64 image, which must mean it's not been working at all on my cluster the entire time. There were no obvious errors so I guess I never noticed, but clearly something must not be working in my env...
bradwood
on 3 Oct 2020
I've tried https://hub.docker.com/r/calfox/external-attacher/tags but not convinced this one is any good.
root@mgmt1:~# DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect --verbose calfox/external-attacher:v2.1.0
[
{
"Ref": "docker.io/calfox/external-attacher:v2.1.0@sha256:7407043052984d0e1a8b3c1275592be959c4026001b6e03d84acf7239f371404",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:7407043052984d0e1a8b3c1275592be959c4026001b6e03d84acf7239f371404",
"size": 739,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
"SchemaV2Manifest": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:141118e5868aeab67a1916df4ca52d381d1d4dee056b4168a59eea6ddaae617a",
"size": 1386
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823",
"size": 639412
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:e5a3df23ed44c4df150af0c20927edf34e86ba3f5a171cd55b2474f2cb9dc454",
"size": 17785271
}
]
}
},
{
"Ref": "docker.io/calfox/external-attacher:v2.1.0@sha256:b7f2bb6fc27753508fdb22e25c3134953ad8d09ef9d4edff68f6faaeace04b87",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:b7f2bb6fc27753508fdb22e25c3134953ad8d09ef9d4edff68f6faaeace04b87",
"size": 739,
"platform": {
"architecture": "arm64",
"os": "linux"
}
},
"SchemaV2Manifest": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:f25aacf3d7974a55a08714f38e715132d651ddbd4ee7f5c27ea87879492cbf62",
"size": 1386
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823",
"size": 639412
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:e99921856fc370c68ba0c3140317741031aafcd31798f3060bdaf8bad80c33e1",
"size": 16564175
}
]
}
},
{
"Ref": "docker.io/calfox/external-attacher:v2.1.0@sha256:8720daaea40d706349b7306d8ad3ad376d08877deb3555bc623400f7e63bbfc2",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:8720daaea40d706349b7306d8ad3ad376d08877deb3555bc623400f7e63bbfc2",
"size": 739,
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v7"
}
},
"SchemaV2Manifest": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:484fdc3444a5fc6f0f18e66dcc1fd06b29d4095ce6ea021547b37b9a52f2c609",
"size": 1400
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823",
"size": 639412
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:ba23695025b5aae8eaccfd8957c4b1e5abc1688ef383d29f15da975311af1d0a",
"size": 16622913
}
]
}
},
{
"Ref": "docker.io/calfox/external-attacher:v2.1.0@sha256:0b4031e598f0fca1f89143f38cdd3e2898ba50d0ca5d7bdc633a4cae780600f3",
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:0b4031e598f0fca1f89143f38cdd3e2898ba50d0ca5d7bdc633a4cae780600f3",
"size": 739,
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v6"
}
},
"SchemaV2Manifest": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:ddfb4c3a664204a079b658b27107758d6ce08b8b47011451bb55f35e69592113",
"size": 1400
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823",
"size": 639412
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:31ae49b492bb985b3e1f804ed49465e83d0f8ad59f903964f6adbb3d96272d41",
"size": 16622913
}
]
}
}
]
root@mgmt1:~#
@boknowswiki - this morning I tried ahuffman/csi-attacher:v2.2.0 which does seem to start okay but I still see the 2G limit.
volume presents at size 10G in Longhorn UI and the .img file is 10G, but from inside the container it's still limited to 2G
bradwood
on 3 Oct 2020
@bradwood weird. Can you try if you format a sparse file like below in your system, how much space does it have?
truncate -s 10G test.img
mkfs.ext4 test.img
mkdir /mnt/tmp
mount test.img /mnt/tmp
df -h
I suspect it will somehow show as 2G.
yasker
on 4 Oct 2020
@yasker - works correctly
root@worker1:/data/disk0# ls -la
total 69900
drwxr-xr-x 4 root root 4096 Oct 4 16:36 .
drwxr-xr-x 3 root root 4096 Sep 3 19:32 ..
-rw-r--r-- 1 root root 51 Sep 3 21:59 longhorn-disk.cfg
drwx------ 2 root root 16384 Sep 3 19:25 lost+found
drwxr-xr-x 5 root root 4096 Oct 3 12:02 replicas
-rw-r--r-- 1 root root 10737418240 Oct 4 16:37 test.img
root@worker1:/data/disk0# df -h |grep mnt
/dev/loop2 9.8G 37M 9.3G 1% /mnt/tmp
root@worker1:/data/disk0#
root@worker1:/data/disk0# mount | grep ^/data
/data/disk0/test.img on /mnt/tmp type ext4 (rw,relatime)
root@worker1:/data/disk0#
ping?
bradwood
on 5 Oct 2020
@bradwood , thank you very much for your response. It was a puzzle to me that seems everyone using vsellier/csi-attacher:v2.0.0 on ARM based cluster hasn't complained about it.
boknowswiki
on 5 Oct 2020
It's a big puzzle to me too! :) Is there a way to make it log? I don't know much about the CSI stuff, but it might just be silently failing... I also searched for the source on github for the Dockerfile for this image but couldn't find it anywhere...
https://github.com/vsellier @vsellier might be able to shed some light...
bradwood
on 5 Oct 2020
Hi,
I'm really surprised to see these images used by someone else. I have created these images during this summer to try to deploy longhorn on k3s with a cluster of rpi3/rpi4 (32bits). It was only in a learning purpose.
I finally gave up because guess what.... I was not able to create a volume greater than 2gb :).
I suppose there are some limitations somewhere with the tools compiled in 32bits.
For example, I made a dirty patch on bufbuild/buf to be able to compile it in 32bit.
diff --git internal/pkg/storage/cmd/storage-go-binary-data/main.go internal/pkg/storage/cmd/storage-go-binary-data/main.go
index 09eae45..e8a3548 100644
--- internal/pkg/storage/cmd/storage-go-binary-data/main.go
+++ internal/pkg/storage/cmd/storage-go-binary-data/main.go
@@ -35,7 +35,7 @@ import (
const (
programName = "storage-go-binary-data"
version = "0.0.1-dev"
- sliceLength = math.MaxInt64
+ sliceLength = math.MaxInt32
)
func main() {
As it's used in the longhorn-engine image, it can be relative to this change but I haven't been able to prove it.
I could gather all my notes on the work I have done to build the images if you think it can help, but honestly, it was only a poc with a lot of dirty work and it was not intended to be reused.
vsellier
on 5 Oct 2020
👍1
If it's MaxInt32, then it makes sense. We're currently focusing on arm64 so that's probably the reason we haven't seen it. After that's done we will check arm as well.
yasker
on 6 Oct 2020
That would be awesome. Thank you for your great work!
vsellier
on 6 Oct 2020
I'm really surprised to see these images used by someone else. I have created these images during this summer to try to deploy longhorn on k3s with a cluster of rpi3/rpi4 (32bits). It was only in a learning purpose.
Heh, @vsellier I think you _will_ be surprised, these images seem to be pretty popular for arm longhorn experimenters, myself included. BTW, a question that has been confusing us, which you might be able to shed some light on, is vsellier/csi-attacher:v2.0.0 in Docker Hub. This image seems to only be built for AMD64, not arm(!) I've been blindly using this on my arm environment (I didn't check until later) and what is weird is, apart from the 2Gb limit, things seemed to work okay, despite this obvious issue. What was your experience with this? Did you ever get a build for that image working on arm(64)? And if you didn't (maybe you forgot too?) how was your experience with Longhorn on ARM using this image?
Thanks for shedding some light and helping out on this issue!
bradwood
on 6 Oct 2020
Hum it should not be amd64, probably a mistake during the last tests I have done. I have rebuilt and pushed the image for arm this time. I hope this could help you.
Regards.
vsellier
on 7 Oct 2020
👍1
I think the limitation only applies to the ARM32, not ARM64. @meldafrawi When you have time, can you give it a quick try?
yasker
on 17 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
rsoika
·
16Comments
mhemrg
·
19Comments
aandrushchenko
·
23Comments
khushboo-rancher
·
17Comments
shubb30
·
22Comments