External-storage: Ceph RBD resize volume

[unteem]

Hi
I'm trying to resize a volume provisioned with ceph rbd provisioner.

Here is the error I get:
Error expanding volume "indie-host/minio-indie-host" of plugin kubernetes.io/rbd : rbd info failed, error: executable file not found in $PATH

Is it possible to resize a volume when using ceph rbd provisioner ?
kubernetes version: 1.10.3

[wongma7]

yes, once the volume is created by the provisioner it is like any rbd volume. That error sounds like the rbd executable is not on the node
/area ceph/rbd

[gnufied]

Yes. You need rbd utils installed both on node and controller-manager for rbd resizing to work.

[JrCs]

I have this error:
Warning VolumeResizeFailed 13s (x4 over 2m) volume_expand Error expanding volume "kube-system/claim1" of plugin kubernetes.io/rbd : failed to get admin secret from ["kube-system"/"ceph-admin-secret"]: failed to get secret from ["kube-system"/"ceph-admin-secret"]
The kube-system/ceph-admin-secret secret exist and it is used to create the volume.

What rules to ClusterRole I need to change to allow the expansion of volume ?

This is the current ClusterRole i use:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "delete"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

[msau42]

I do not believe the external provisioner handles resizing. It is done by in-tree volume plugin.

[ainiml]

@JrCs for the failed to get admin secret error, is the Secret formatted wrong? Does the Key have to be a specific name?

[arnaudmz]

I do not believe the external provisioner handles resizing. It is done by in-tree volume plugin.

I aggree, I reproduced the same situation, and the following error raised into the kube-controller-manager logs:
E0130 20:24:56.889133 1 nestedpendingoperations.go:267] Operation for "\"94d60212-24b6-11e9-84f1-064f99e5ef76\"" failed. No retries permitted until 2019-01-30 20:26:58.88910867 +0000 UTC m=+22113.425060928 (durationBeforeRetry 2m2s). Error: "error expanding volume \"default/data-mydb-mariadb-master-0\" of plugin \"kubernetes.io/rbd\": rbd info failed, error: executable file not found in $PATH"

Is it possible to make the standalone rbd provisioner handle resizing?

[JrCs]

@ainiml now i use the csi-rbd plugin. All is working but not resizing for the moment.

[shuraa]

@arnaudmz I found the only workaround is to repack and replace "kube-controller-manager" docker image. Kubernetes 1.12.5.

[arnaudmz]

@arnaudmz I found the only workaround is to repack and replace "kube-controller-manager" docker image. Kubernetes 1.12.5.

I bet it works, but I would expect the job done on standard official images, without the (boring) need to follow the upstream images and rebuild them.

[Paxa]

@JrCs I had same problem, I tried manually change log lever for kube-controller-manager, and it can get secret successfully. But original message is replaced (probably should report to k8s)

Error from here https://github.com/kubernetes/kubernetes/blob/v1.12.5/pkg/volume/util/util.go#L183
Replaced here https://github.com/kubernetes/kubernetes/blob/51dd616cdd25d6ee22c83a858773b607328a18ec/pkg/volume/rbd/rbd.go#L1106

secret's type should be kubernetes.io/rbd like in example https://github.com/kubernetes/examples/blob/87f1ce167f34c7b6927a36f7e21a18f282fe5c4b/staging/volumes/rbd/secret/ceph-secret.yaml#L5

With 1.12.5 I already have rbd binary in container, but it gave error about config (auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring...)
So I changed /etc/kubernetes/manifests/kube-controller-manager.manifest to add /etc/ceph host path

[icefed]

https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-volume-resizing.md#external-resize-controller
Does this useful?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[devopstales]

any update on this?
/remove-lifecycle stale

[cclloyd]

You can use the image gcr.io/google_containers/hyperkube:v1.14.6 as the controller manager. It has the rbd binary but I still have issues (authentication I believe) resizing images when using Ceph Nautilus as the cluster.

[qct]

any updates?

[MatthiasHertel]

any updates ?

[MatthiasHertel]

@cclloyd

You can use the image gcr.io/google_containers/hyperkube:v1.14.6 as the controller manager. It has the rbd binary but I still have issues (authentication I believe) resizing images when using Ceph Nautilus as the cluster.

• yes replace the base image of the kube-controller-manager works.
• yes their is a authentication issue, i fixed it with copy the pool keyring and put an empty ceph.conf into the kube-controller-manager (see below)

kubectl cp $cephclientkeyring kube-controller-manager:/etc/ceph/
kubectl exec kube-controller-manager -- /bin/sh -c 'touch /etc/ceph/ceph.conf

now the resizing works , but it feels like a hack ?

i don't understand why the kube-controller-manager (in-tree) must handle the resizing and the rbd-provisioner (out-tree) (see: https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy/rbac ) handles the creation? (in-tree vs out-tree)

the sitituation (in-tree vs out-tree) makes it more complex and so i can simply replace the base-image (feels like a hack) and handle all actions (creation and resizing) in-tree.

or is it a good solution to replace the base-image ?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[devopstales]

/remove-lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.