External-storage: pod mount volume failed

Created on 27 Jul 2018  路  18Comments  路  Source: kubernetes-retired/external-storage 
~]# kubectl describe pod ceph-pod1
...
Events:
  Type     Reason                  Age   From                     Message
  ----     ------                  ----  ----                     -------
  Normal   Scheduled               2m    default-scheduler        Successfully assigned default/ceph-pod1 to k8s-node1
  Normal   SuccessfulAttachVolume  2m    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-3b8b2ced-913c-11e8-a54c-525400c9c704"
 Warning  FailedMount             1m    kubelet, k8s-node1       MountVolume.WaitForAttach failed for volume "pvc-3b8b2ced-913c-11e8-a54c-525400c9c704" : rbd: map failed exit status 110, rbd output: 2018-07-27 09:29:09.443021 7fab59fa4d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: sysfs write failed
2018-07-27 09:30:09.464043 7fab59fa4d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (110) Connection timed out
  Warning  FailedMount  32s  kubelet, k8s-node1  Unable to mount volumes for pod "ceph-pod1_default(70eb1283-913c-11e8-b060-525400c9c704)": timeout expired waiting for volumes to attach or mount for pod "default"/"ceph-pod1". list of unmounted volumes=[volume-test]. list of unattached volumes=[volume-test default-token-jmdwd]
  Warning  FailedMount  26s  kubelet, k8s-node1  MountVolume.WaitForAttach failed for volume "pvc-3b8b2ced-913c-11e8-a54c-525400c9c704" : rbd: map failed exit status 110, rbd output: 2018-07-27 09:30:10.533928 7f996fa35d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: sysfs write failed
2018-07-27 09:31:10.546814 7f996fa35d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                        STORAGECLASS   REASON    AGE
pvc-3b8b2ced-913c-11e8-a54c-525400c9c704   2Gi        RWO            Delete           Bound     default/ceph-claim-dynamic   fast-rbd                 16m
[root@k8s-master1 ~]# kubectl get pvc
NAME                 STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-claim-dynamic   Bound     pvc-3b8b2ced-913c-11e8-a54c-525400c9c704   2Gi        RWO            fast-rbd       16m

~]# kubectl get secrets ceph-user-secret -o yaml
apiVersion: v1
data:
  key: QVFDSmMxcGJWZUZ1TnhBQU1VRStlSmlpK2ZwRHFwMzdQY2tlUkE9PQ==
kind: Secret
metadata:
  creationTimestamp: 2018-07-27T01:21:38Z
  name: ceph-user-secret
  namespace: default
  resourceVersion: "65237"
  selfLink: /api/v1/namespaces/default/secrets/ceph-user-secret
  uid: 68e90ce2-913b-11e8-b060-525400c9c704
type: kubernetes.io/rbd

 ~]# kubectl get storageclasses fast-rbd -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: 2018-07-27T01:26:09Z
  name: fast-rbd
  resourceVersion: "65694"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/fast-rbd
  uid: 0a533110-913c-11e8-a54c-525400c9c704
parameters:
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  imageFeatures: layering
  imageFormat: "2"
  monitors: 10.0.7.102:6789, 10.0.7.103:6789, 10.0.7.104:6789
  pool: kube
  userId: kube
  userSecretName: ceph-user-secret
  userSecretNamespace: default
provisioner: ceph.com/rbd
reclaimPolicy: Delete
volumeBindingMode: Immediate

[root@k8s-master1 my-cluster]# kubectl describe pv
...
Source:
    Type:          RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
    CephMonitors:  [10.0.7.102:6789  10.0.7.103:6789  10.0.7.104:6789]
    RBDImage:      kubernetes-dynamic-pvc-85856e5d-9140-11e8-91b4-0a580af40338
    FSType:        
    RBDPool:       kube
    RadosUser:     kube
    Keyring:       /etc/ceph/keyring
    SecretRef:     &{ceph-user-secret default}
    ReadOnly:      false
Events:            <none>
areceprbd
Source
picture kuops

Most helpful comment

I'm not sure whether it helps or not but I used to get the very same error message when I mess up the content of the ceph secrets. The error message is rather misleading.
@kuops , please note, that the README.md for ceph/rbac was wrong before #899. If you followed the wrong instructions, you got a wrong client.kube secret. Please check the updated docs and pay attention to this line when you create the client secret:
ceph auth get-key client.kube > /tmp/secret

picture szamuboy on 2 Aug 2018
❤2 👍2

All 18 comments

It sounds like the "keyring" needs to be present on the host? So that the kubelet can read it.
/area ceph/rbd

wongma7 picture wongma7 on 27 Jul 2018

thanks锛寃hen i add the /etc/ceph/keyring file to node锛宲od is runing锛宐ut how to change /etc/ceph/keyring path to /etc/ceph/ceph.client.admin.keyring

kuops picture kuops on 28 Jul 2018

why node need /etc/ceph/keyring , secret is exists key

kuops picture kuops on 28 Jul 2018

It seems like a bug in the provisioner, it should set one of secret or keyring, not both.
upstream: https://github.com/kubernetes/kubernetes/blob/69176615e54468f0f43d08ffe0956c67f0706d78/pkg/volume/rbd/rbd.go#L682
@cofyc

wongma7 picture wongma7 on 30 Jul 2018
👍1

/assign
I'll take a look at later.

cofyc picture cofyc on 31 Jul 2018

hi, keyring file does not need to be present on the host.

thanks锛寃hen i add the /etc/ceph/keyring file to node锛宲od is runing锛宐ut how to change /etc/ceph/keyring path to /etc/ceph/ceph.client.admin.keyring

rbd-provisioner only set secretRef filed. /etc/ceph/keyring is default value of RBDPersistentVolumeSource. If secretRef field is set, keyring is ignored.

Could you paste output of ceph auth get client.kube on ceph node? I guess your user secret auth or permission probably is wrong.

2018-07-27 09:30:09.464043 7fab59fa4d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

This log can be ignored. rbd always search keyring file if --keyring is not set even if --key argument is provided.

cofyc picture cofyc on 31 Jul 2018 
[root@k8s-node1 ~]# ceph auth get client.kube
exported keyring for client.kube
[client.kube]
    key = AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA==
    caps mon = "allow r"
    caps osd = "allow rwx pool=kube"


[root@k8s-node1 ~]# cat /etc/ceph/keyring 
[client.kube]
    key = AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA==

~]# kubectl get secrets  ceph-user-secret  -o jsonpath={.data.key}|base64 -d
AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA==
kuops picture kuops on 31 Jul 2018

Secret auth is right, my guess is wrong.

Could you clean all files under /etc/ceph and then run this command to test your rbd client on that node can connect to ceph cluster?

rbd ls -m 10.0.7.102:6789,10.0.7.103:6789 ,10.0.7.104:6789 -p kube --id kube --key AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA==

If this does not work, run following command to view kernel logs:

dmesg -T

and paste ceph/rbd/rados related messages here.

And run following commands both on your k8s worker node and ceph server:

uname -a
rbd --version
ceph --version
cofyc picture cofyc on 1 Aug 2018

is cleaning,the result

[root@k8s-node1 etc]# ls /etc/ceph/
[root@k8s-node1 etc]# rbd ls -m 10.0.7.102:6789,10.0.7.103:6789 ,10.0.7.104:6789 -p kube --id kube --key AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA==
2018-08-01 19:04:42.896739 7fbfb62bbd40 -1 did not load config file, using default settings.
2018-08-01 19:04:42.903724 7fbfb62bbd40 -1 Errors while parsing config file!
2018-08-01 19:04:42.903754 7fbfb62bbd40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.903755 7fbfb62bbd40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.903755 7fbfb62bbd40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.906003 7fbfb62bbd40 -1 Errors while parsing config file!
2018-08-01 19:04:42.906017 7fbfb62bbd40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.906019 7fbfb62bbd40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.906019 7fbfb62bbd40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-08-01 19:04:42.949172 7fbfb62bbd40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
kubernetes-dynamic-pvc-595acdf8-9143-11e8-91b4-0a580af40338
[root@k8s-node1 etc]# uname -a
Linux k8s-node1 4.4.143-1.el7.elrepo.x86_64 #1 SMP Sun Jul 22 10:06:33 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@k8s-node1 etc]# rbd --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
[root@k8s-node1 etc]# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
kuops picture kuops on 1 Aug 2018

Your rbd ls command is run successfully. Please try rbd map, probably because "feature set mismatch" issue. 

rbd map -m 10.0.7.102:6789,10.0.7.103:6789,10.0.7.104:6789 -p kube --id kube --key AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA== kubernetes-dynamic-pvc-595acdf8-9143-11e8-91b4-0a580af40338

If it failed, run dmesg -T to view kernel logs.

BTW, your ceph client version is 12.2.7, what's your ceph server version? Could you run ceph --version on your ceph server node?

cofyc picture cofyc on 1 Aug 2018

ceph version the same as client 

[root@k8s-master2 ~]# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
[root@k8s-node1 ~]# rbd map -m 10.0.7.102:6789,10.0.7.103:6789,10.0.7.104:6789 -p kube --id kube --key AQCJc1pbVeFuNxAAMUE+eJii+fpDqp37PckeRA== kubernetes-dynamic-pvc-595acdf8-9143-11e8-91b4-0a580af40338
2018-08-01 19:31:29.046624 7f86b86bbd40 -1 did not load config file, using default settings.
2018-08-01 19:31:29.052877 7f86b86bbd40 -1 Errors while parsing config file!
2018-08-01 19:31:29.052913 7f86b86bbd40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.052915 7f86b86bbd40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.052915 7f86b86bbd40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.055222 7f86b86bbd40 -1 Errors while parsing config file!
2018-08-01 19:31:29.055243 7f86b86bbd40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.055244 7f86b86bbd40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.055245 7f86b86bbd40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-08-01 19:31:29.090974 7f86b86bbd40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kube.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
/dev/rbd0
kuops picture kuops on 1 Aug 2018

rbd map is run successfully too, that means kubelet on your node k8s-node-1 should be able to map rbd image on the host (kubelet rbd plugin executes rbd map with monitors and secrets from PV to map rbd image).
Could you create a pod on your k8s-node-1 to test it again? (you can set pod.Spec.nodeName to k8s-node-1, then it will be started on k8s-node-1).
If it still failed to mount volume, we need check kubelet logs.

cofyc picture cofyc on 1 Aug 2018

I'm not sure whether it helps or not but I used to get the very same error message when I mess up the content of the ceph secrets. The error message is rather misleading.
@kuops , please note, that the README.md for ceph/rbac was wrong before #899. If you followed the wrong instructions, you got a wrong client.kube secret. Please check the updated docs and pay attention to this line when you create the client secret:
ceph auth get-key client.kube > /tmp/secret

szamuboy picture szamuboy on 2 Aug 2018
❤2 👍2

@cofyc now, pod is running

Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Warning  FailedScheduling        4m (x602 over 29m)  default-scheduler        persistentvolumeclaim "ceph-claim-dynamic" not found
  Normal   Pulling                 1m                  kubelet, k8s-node1       pulling image "nginx"
  Normal   SuccessfulAttachVolume  1m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-67fbbd68-95a7-11e8-a6e7-525400c9c704"
  Normal   Pulled                  1m                  kubelet, k8s-node1       Successfully pulled image "nginx"
  Normal   Created                 1m                  kubelet, k8s-node1       Created container
  Normal   Started                 1m                  kubelet, k8s-node1       Started container
kuops picture kuops on 2 Aug 2018

/close

cofyc picture cofyc on 31 Oct 2018

@cofyc: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 31 Oct 2018

I created a one node ceph cluster that I can connect to with telnet on port 6789 and deployed the provisionner with the admin key and the kube key i created as secrets, encoded as base64, with create -f deploy/rbac and then kubectl create -f examples/
(slight bug btw : the provisioner didn't have the rights to access the secrets resources, i saw this listed as an issue and fixed it)

and i get the same errors. ("did not load config file, using default settings." etc)

I tripled check the secrets, didn't solve it.

SkinyMonkey picture SkinyMonkey on 21 Mar 2019

sometimes you need to update kernel .

vsxen picture vsxen on 16 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ianseyer picture ianseyer  路  11Comments
gnitzan picture gnitzan  路  9Comments
gtie picture gtie  路  12Comments
gaurang1988 picture gaurang1988  路  6Comments
killfill picture killfill  路  10Comments