Rook: k8s pvc and ceph-rbd image How to correspond?
- Feature Request
I use ceph Storageclass. through storageclass created pvc How to correspond ceph rbd image?
in k8s
# kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER AGE
ceph-rbd (default) ceph.com/rbd 168d
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-claim Bound pvc-92a8d5ff-6f08-11e9-97c4-005056bd7916 2Gi RWO ceph-rbd 66d
datadir-luminous-mongoose-mongodb-secondary-0 Bound pvc-68625f44-9335-11e9-acde-005056bd5fee 1Gi RWO ceph-rbd 20d
etcd-pvc Bound etcd-pv 8Gi RWO nfs 133d
redis-data-redis-test-master-0 Bound pvc-a092cf81-87fc-11e9-acde-005056bd5fee 8Gi RWO ceph-rbd 34d
redis-data-redis-test-slave-0 Bound pvc-a09832e3-87fc-11e9-acde-005056bd5fee 8Gi RWO ceph-rbd 34d
redis-data-redis-test-slave-1 Bound pvc-fd6364a5-87fd-11e9-acde-005056bd5fee 8Gi RWO ceph-rbd 34d
redis-data-redis-test-slave-2 Bound pvc-0a4ed4af-87fe-11e9-acde-005056bd5fee 8Gi RWO ceph-rbd 34d
in ceph
$ rbd ls kubernetes
kubernetes-dynamic-pvc-08569d54-8381-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-119edfce-87fe-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-162a432f-8381-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-28650d25-97f1-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-2865aff0-97f1-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-2be8d91a-9882-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-38bc3375-7dfe-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-4698842e-a144-11e9-b9f6-86ee373712c2
kubernetes-dynamic-pvc-68655af1-9335-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-92ad3a79-6f08-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-a09a0488-87fc-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a09dff79-87fc-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a4868419-987f-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a64bbbb8-72cf-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a7198c4f-72cf-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-aaca3ec1-708e-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-b8b7f367-5833-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-bcbf58fa-4585-11e9-9a4a-f67902143069
kubernetes-dynamic-pvc-c67785fe-7d3e-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-ed84003d-8380-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-ffbc9695-87fd-11e9-b847-86ee373712c2
[cephd@yztestopceph61v8yz test]$ rbd info kubernetes/kubernetes-dynamic-pvc-ffbc9695-87fd-11e9-b847-86ee373712c2
rbd image 'kubernetes-dynamic-pvc-ffbc9695-87fd-11e9-b847-86ee373712c2':
size 8 GiB in 2048 objects
order 22 (4 MiB objects)
id: 2c5526b8b4567
block_name_prefix: rbd_data.2c5526b8b4567
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Jun 6 09:54:19 2019
I can't find the correspondence between k8s pvc and ceph-rbd image. Looking for help
question
llussy
All 4 comments
@llussy The mapping in Kubernetes is the following:
PVC <-- PV <-- Storage (Ceph)
kubectl get pv will give you the list of PersistentVolumes in your Kubernetes cluster.
galexrt
on 10 Jul 2019
@galexrt thanks,Maybe I didn't describe it clearly.
I can get PV list,But I don't know pv name it matches which ceph RBD？
The following PV name is pvc-0a4ed4af-87fe-11e9-acde-005056bd5fee ,I can't find it in ceph.
# default namespace pv
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
etcd-pv 8Gi RWO Recycle Bound default/etcd-pvc nfs 133d
pvc-052952bb-8381-11e9-acde-005056bd5fee 10Gi RWO Delete Bound redis/data-redis-ha-server-1 ceph-rbd 40d
pvc-0a4ed4af-87fe-11e9-acde-005056bd5fee 8Gi RWO Delete Bound default/redis-data-redis-test-slave-2 ceph-rbd 34d
pvc-1624df5c-8381-11e9-acde-005056bd5fee 10Gi RWO Delete Bound redis/data-redis-ha-server-2
# all rbd in ceph
$ rbd ls kubernetes
kubernetes-dynamic-pvc-08569d54-8381-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-119edfce-87fe-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-162a432f-8381-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-28650d25-97f1-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-2865aff0-97f1-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-2be8d91a-9882-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-38bc3375-7dfe-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-4698842e-a144-11e9-b9f6-86ee373712c2
kubernetes-dynamic-pvc-68655af1-9335-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-92ad3a79-6f08-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-a09a0488-87fc-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a09dff79-87fc-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a4868419-987f-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a64bbbb8-72cf-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-a7198c4f-72cf-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-aaca3ec1-708e-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-b8b7f367-5833-11e9-986e-86ee373712c2
kubernetes-dynamic-pvc-bcbf58fa-4585-11e9-9a4a-f67902143069
kubernetes-dynamic-pvc-c67785fe-7d3e-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-ed84003d-8380-11e9-b847-86ee373712c2
kubernetes-dynamic-pvc-ffbc9695-87fd-11e9-b847-86ee373712c2
@llussy Have you already looked at the PV object in its entirety?
For Flexvolume, example:
kubectl get pv pvc-c409bdcb-6cea-11e9-8936-9600001d3fab -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: ceph.rook.io/block
creationTimestamp: "2019-05-02T14:58:38Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-c409bdcb-6cea-11e9-8936-9600001d3fab
resourceVersion: "10504117"
selfLink: /api/v1/persistentvolumes/pvc-c409bdcb-6cea-11e9-8936-9600001d3fab
uid: c453d6f5-6cea-11e9-a067-9600001d3fa9
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 150Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: REDACTED
namespace: REDACTED
resourceVersion: "10504100"
uid: c409bdcb-6cea-11e9-8936-9600001d3fab
flexVolume:
driver: ceph.rook.io/rook-ceph-system
fsType: xfs
options:
clusterNamespace: rook-ceph
dataBlockPool: ""
image: pvc-c409bdcb-6cea-11e9-8936-9600001d3fab
pool: block-repl-3-1
storageClass: rook-ceph-block-repl-3-1
persistentVolumeReclaimPolicy: Delete
storageClassName: rook-ceph-block-repl-3-1
volumeMode: Filesystem
status:
phase: Bound
The .spec.flexVolume.options.image is the PV name which normally should also be the name of the RBD volume.
In your case the RBD volumes are having kubernetes-image- as a prefix instead of just pvc- for some reason. Are you using Ceph CSI instead of Flexvolume to dynamicly provision the RBD volumes or so?
galexrt
on 10 Jul 2019
👍2
@galexrt thank you very much. original it can be seen through pv yaml (rbd.image). This has bothered me for a long time. thanks.
```bash
kubectl get pv pvc-a4f1f0a4-708e-11e9-b831-005056bd5fee -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: ceph.com/rbd
rbdProvisionerIdentity: ceph.com/rbd
creationTimestamp: 2019-05-07T06:09:26Z
finalizers:
- kubernetes.io/pv-protection
name: pvc-a4f1f0a4-708e-11e9-b831-005056bd5fee
resourceVersion: "74606920"
selfLink: /api/v1/persistentvolumes/pvc-a4f1f0a4-708e-11e9-b831-005056bd5fee
uid: aadcc6a8-708e-11e9-b831-005056bd5fee
spec:
accessModes: - ReadWriteOnce
capacity:
storage: 20Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: jenkins
namespace: jenkins
resourceVersion: "74606908"
uid: a4f1f0a4-708e-11e9-b831-005056bd5fee
persistentVolumeReclaimPolicy: Delete
rbd:
fsType: ext4
image: kubernetes-dynamic-pvc-aaca3ec1-708e-11e9-986e-86ee373712c2
keyring: /etc/ceph/keyring
monitors:
- 172.16.8.57:6789
- 172.16.8.59:6789
- 172.16.8.60:6789
pool: kubernetes
secretRef:
name: ceph-secret
user: kubernetes
storageClassName: ceph-rbd
status:
phase: Bound
```
llussy
on 11 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
zerkms
·
4Comments
lilHermit
·
4Comments
DanKerns
·
4Comments
tarjintor
·
3Comments
billimek
·
3Comments
Most helpful comment
@llussy Have you already looked at the PV object in its entirety?
For Flexvolume, example:
The
.spec.flexVolume.options.imageis the PV name which normally should also be the name of the RBD volume.In your case the RBD volumes are having
kubernetes-image-as a prefix instead of justpvc-for some reason. Are you using Ceph CSI instead of Flexvolume to dynamicly provision the RBD volumes or so?