External-storage: rbd status check fails -

[wyllys66]

Using the ceph.com provisioner to create RBD images for persistent volumes exactly as described in the examples.

The RBD image is created successfully, however, the pod never completely starts and generates this error:

MountVolume.WaitForAttach failed for volume "pvc-77bb8120-c0df-11e8-bdb4-001e0b4843c4" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()

root is not an option right now, we would like to just use the rbd provisioner if possible without installing rook and all of its dependencies.

[wongma7]

is 'rbd' installed on the nodes?

[wongma7]

we should document that ceph rbd utilities are required on nodes as a prerequisite

[wyllys66]

rbd utilities are part of ceph-common package.

Just installing ceph-common is not enough, the node would also then need a minimal /etc/ceph/ceph.conf file in order for the rbd client to execute correctly.

[Paxa]

+1 for adding it to documentation. I had same issue

[scm-ns]

I don't think ceph.conf needs to be present. rbd is called with all the parameters passed in arguments.
apt install ceph-common should fix yours issues.

Ofcourse there might be versioning issues that you will have to worry about. The rbd-provisioner and the rbd version on the node might have to match up.

[exodussystem]

For OS like CoreOS, I cannot install ceph-common.
However, I can manage to get rbd/rados binary accessible by running Docker container
with ceph/container

Source:
https://stackoverflow.com/questions/37996983/invoking-rbd-docker-from-kubernetes-on-coreos-returns-fork-exec-invalid-argume

In this case, I need to setup /etc/ceph/ceph.conf.
I got it work up to this step. When I brought up a PVC for MariaDB, I got this error

_MountVolume.WaitForAttach failed for volume "pvc-d82e2dd3-41e8-11e9-aa1a-000c297f8172" : fail to check rbd image status with: (exit status 22),
rbd output: (server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster!
2019-03-08 21:27:01.924 7f3981d65900 -1 monclient: get_monmap_and_config cannot identify monitors to contact )
error_

_The PVC was created successfully but it could _NOT_ be bound because RBD failed to check status._

Any idea?

[captainwenzi]

We have set up a Docker Enterprise 2.1 cluster with 3 worker nodes (docker engine v18.09, UCP v3.1.0), and we are using an external Ceph cluster to provide the persistent storage.

We have attempted to set up Ceph Rados Block Device (RBD) as a Storage Class for Kubernetes, according to an online tutorial (http://tutorial.kubernetes.noverit.com/content/ceph.html), and we are able to create a PVC with bounded PV using the Ceph RDB Storage Class we created. We have checked that the RBD image was in fact dynamically provisioned on the Ceph cluster.

However, during the installation of application using Helm, we are getting a warning event during Pod creation:

MountVolume WaitForAttach failed for volume : fail to check rbd image status with (executable file not found in $PATH), RBD OUTPUT: ()

The installation does not proceed and the containers in the pod never get successfully created. The pod remains stuck with a “ContainerCreating” status.

All worker and manager nodes in the cluster have been installed with the Ceph client libraries, and we are able to communicate with the Ceph cluster from the host OS of all nodes using the “rbd” command.

Can you advice on the error message? Especially the part about “executable file not found in $PATH”. This error appears in the ucp-kubelet logs, but we are unsure of its origin.

Is this a limitation because Docker EE is using ucp-kubelet instead of OS level kubelet?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.