External-storage: rbd-provisioner: missing Ceph monitors

[knfoo]

Kubernetes version:
Client Version: v1.10.2
Server Version: v1.10.2

OS version
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"

Ceph version
Luminous 12.2.5

rbd-provisioner version
quay.io/external_storage/rbd-provisioner latest dcc584102b93

storage class:
```apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
creationTimestamp: 2018-05-23T06:39:27Z
name: dynamic
resourceVersion: "7297999"
selfLink: /apis/storage.k8s.io/v1/storageclasses/dynamic
uid: 0a494817-5e54-11e8-ab89-94c69119d0e9
parameters:
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: kube-system
fsType: ext4
imageFeatures: layering
imageFormat: "2"
monitors: 192.168.200.15:6789,192.168.200.17:6789,192.168.200.19:6789
pool: ssd
userId: kube
userSecretName: ceph-secret-user
provisioner: kubernetes.io/rbd
reclaimPolicy: Delete
volumeBindingMode: Immediate

pvc:
```apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"9d2b5b91-5e54-11e8-901d-7a1ea547e356","leaseDurationSeconds":15,"acquireTime":"2018-05-25T06:48:52Z","renewTime":"2018-05-25T06:51:26Z","leaderTransitions":0}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"ceph-claim","namespace":"nextcloud"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"2Gi"}}}}
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd
  creationTimestamp: 2018-05-25T06:48:52Z
  finalizers:
  - kubernetes.io/pvc-protection
  name: ceph-claim
  namespace: nextcloud
  resourceVersion: "7676411"
  selfLink: /api/v1/namespaces/nextcloud/persistentvolumeclaims/ceph-claim
  uid: af7d08ca-5fe7-11e8-ab89-94c69119d0e9
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: dynamic
status:
  phase: Pending

But the pvc is stuck in pending with the error message from the rbd-provisioner pod:
Failed to provision volume for claim "nextcloud/ceph-claim" with StorageClass "dynamic": missing Ceph monitors

Anybody have experienced similar problems ?

[knfoo]

seems related to issue https://github.com/kubernetes-incubator/external-storage/issues/772 as I have upgraded to coredns recently

[jjo]

For those wanting to use ceph-mons by IPs, a nasty workaround (that works): create externalName resources pointing to each monitor, using e.g. xip.io service to get hostnames (as naked IPs are not supported by externalName services):

kind: Service
apiVersion: v1
metadata:
  name: ceph-mon-1
  namespace: ceph
spec:
  type: ExternalName
  externalName: 192.168.0.1.xip.io
---
kind: Service
apiVersion: v1
metadata:
  name: ceph-mon-2
  namespace: ceph
spec:
  type: ExternalName
  externalName: 192.168.0.2.xip.io
---
kind: Service
apiVersion: v1
metadata:
  name: ceph-mon-3
  namespace: ceph
spec:
  type: ExternalName
  externalName: 192.168.0.3.xip.io

Then pass

   monitors: ceph-mon-1.ceph.svc.cluster.local.:6789,ceph-mon-2.ceph.svc.cluster.local.:6789,ceph-mon-3.ceph.svc.cluster.local.:6789

[knfoo]

That seems to be working for me @jjo thanks for the tip :+1:

[albarki]

Not working for CEPH IPV6 nodes

[ygqygq2]

@jjo Why that? Is brd-provisioner must be to lookup the monitor name?

[jjo]

@jjo Why that? Is brd-provisioner must be to lookup the monitor name?

I recall needing the monitors to be resolvable (i.e. not naked IPs), you
could just add e.g. 192.168.0.1.xip.io and so -- I put an ExternalName in
the middle to be able to more ~easily update it while keeping an stable
monitors list (tho not strictly needed really)

[jjo]

Not working for CEPH IPV6 nodes

Wonder if it would work via ip6.name ~equivalent hack:

$ dig +short 2001.db8.8000.0.0.0.0.1.ip6.name aaaa
2001:db8:8000::1

, maybe putting the above fqdns directly into monitors list.

[albarki]

@jjo i added them like you said, but still complaining about missing monitors IPs

[wufenglinux]

我也是这个问题，但是我尝试过无数种方法。
我发现了一个让人很生气的问题
我修改了storageclass中的配置参数，这个参数
parameters:
monitors: 这个参数即使乱写 storage依然可以创建出来，并且provisioner也是报 StorageClass "rbd": missing Ceph monitors

这个错误真的很误导人，其实并不是没有ceph monitor
我到现在还没有解决这个问题。
这个问题误导了我快1个月了

[wufenglinux]

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.