External-storage: missing Ceph monitors

[ygqygq2]

I deploy with the latest documentation, and use the example file. The error log like this:

kubectl logs -f rbd-provisioner-857866b5b7-hwx79

E0820 09:41:25.984983       1 controller.go:722] error syncing claim "default/claim2": failed to provision volume with StorageClass "rbd": failed to get admin secret from ["kube-system"/"ceph-admin-secret"]: secrets "ceph-admin-secret" is forbidden: User "system:serviceaccount:default:rbd-provisioner" cannot get secrets in the namespace "kube-system"
I0820 09:41:25.984999       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"claim2", UID:"4a136790-a45a-11e8-ba76-000c29ea3e30", APIVersion:"v1", ResourceVersion:"4557080", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rbd": failed to get admin secret from ["kube-system"/"ceph-admin-secret"]: secrets "ceph-admin-secret" is forbidden: User "system:serviceaccount:default:rbd-provisioner" cannot get secrets in the namespace "kube-system"

[wongma7]

Please try these steps https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/rbd/deploy/README.md#install-with-rbac-roles

[ygqygq2]

@wongma7
I used the RBAC, also the error.

Then I add the content in the clusterrole.yaml to resolve the secret problem:

  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create","get","list","watch"]

[ygqygq2]

My env:
CEPH: "luminous"
Kubeadmin: 1.11
System: CentOS7

[ygqygq2]

I rebuild the docker image, but have a new problem:

I0822 03:53:23.799799       1 controller.go:948] provision "default/rbd-pv-claim2" class "rbd": started
I0822 03:53:23.806644       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pv-claim2", UID:"75e6ed6b-a5be-11e8-8a0d-000c29ea3e30", APIVersion:"v1", ResourceVersion:"5044205", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pv-claim2"
W0822 03:53:23.810027       1 controller.go:707] Retrying syncing claim "default/rbd-pv-claim2" because failures 4 < threshold 15
E0822 03:53:23.810086       1 controller.go:722] error syncing claim "default/rbd-pv-claim2": failed to provision volume with StorageClass "rbd": missing Ceph monitors
I0822 03:53:23.810121       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pv-claim2", UID:"75e6ed6b-a5be-11e8-8a0d-000c29ea3e30", APIVersion:"v1", ResourceVersion:"5044205", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rbd": missing Ceph monitors

[ygqygq2]

I modified the pkg/provision/provision.go. Is that the solution?

                case "monitors":
                        // Try to find DNS info in local cluster DNS so that the kubernetes
                        // host DNS config doesn't have to know about cluster DNS
                        dnsip := findDNSIP(p)
                        glog.V(4).Infof("dnsip: %q\n", dnsip)
                        fmt.Printf("dnsip: %q\n", dnsip)
                        arr := strings.Split(v, ",")
                        fmt.Printf("arr string: %s  OK\n", v)
                        for _, m := range arr {
                                mhost, mport := splitHostPort(m)
                                if dnsip != "" {
                                        var lookup []string
                                        if lookup, err = lookuphost(mhost, dnsip); err == nil {
                                                fmt.Printf("A monitors list: %v\n", opts.monitors)
                                                for _, a := range lookup {
                                                        glog.V(1).Infof("adding %+v from mon lookup\n", a)
                                                        opts.monitors = append(opts.monitors, joinHostPort(a, mport))
                                                        fmt.Printf("B monitors list: %v\n", opts.monitors)
                                                }
                                        } else {
                                                opts.monitors = append(opts.monitors, joinHostPort(mhost, mport))
                                                fmt.Printf("C monitors list: %v\n", opts.monitors)
                                        }
                                } else {
                                        opts.monitors = append(opts.monitors, joinHostPort(mhost, mport))
                                        fmt.Printf("D monitors list: %v\n", opts.monitors)
                                }
                        }
                        glog.V(4).Infof("final monitors list: %v\n", opts.monitors)
                        opts.monitors = append(opts.monitors, "192.168.105.94:6789")
                        fmt.Printf("final monitors list: %v\n", opts.monitors)
                        if len(opts.monitors) < 1 {
                                return nil, fmt.Errorf("missing Ceph monitors")
                        }

[root@lab1 rbd]# sh local-start.sh 
I0822 18:12:31.616033 1231887 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: rbd-provisioner-1
I0822 18:12:31.617319 1231887 leaderelection.go:185] attempting to acquire leader lease  kube-system/ceph.com-rbd...
I0822 18:12:49.027920 1231887 leaderelection.go:194] successfully acquired lease kube-system/ceph.com-rbd
I0822 18:12:49.028452 1231887 controller.go:597] Starting provisioner controller ceph.com/rbd_lab1_e1a5af7e-a5f3-11e8-8a18-000c29ea3e30!
I0822 18:12:49.028535 1231887 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"ceph.com-rbd", UID:"1327745f-a456-11e8-b55d-000c2931d938", APIVersion:"v1", ResourceVersion:"5119957", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' lab1_e1a5af7e-a5f3-11e8-8a18-000c29ea3e30 became leader
I0822 18:12:49.128788 1231887 controller.go:646] Started provisioner controller ceph.com/rbd_lab1_e1a5af7e-a5f3-11e8-8a18-000c29ea3e30!
I0822 18:12:49.129236 1231887 controller.go:948] provision "default/rbd-pv-claim2" class "ceph-rbd": started
I0822 18:12:49.137701 1231887 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pv-claim2", UID:"808b1ede-a5e4-11e8-a720-000c293d66a5", APIVersion:"v1", ResourceVersion:"5099319", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pv-claim2"
W0822 18:12:49.144725 1231887 provision.go:203] error getting coredns service: services "coredns" not found. Falling back to kube-dns
dnsip: "10.96.0.10"
arr string: 192.168.105.92:6789,192.168.105.93:6789,192.168.105.94:6789  OK
A monitors list: []
A monitors list: []
A monitors list: []
final monitors list: [192.168.105.94:6789]
I0822 18:12:49.569388 1231887 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-ec353042-a5f3-11e8-8a18-000c29ea3e30"
I0822 18:12:49.569449 1231887 controller.go:1048] provision "default/rbd-pv-claim2" class "ceph-rbd": volume "pvc-808b1ede-a5e4-11e8-a720-000c293d66a5" provisioned
I0822 18:12:49.569477 1231887 controller.go:1062] provision "default/rbd-pv-claim2" class "ceph-rbd": trying to save persistentvvolume "pvc-808b1ede-a5e4-11e8-a720-000c293d66a5"
I0822 18:12:49.584924 1231887 controller.go:1069] provision "default/rbd-pv-claim2" class "ceph-rbd": persistentvolume "pvc-808b1ede-a5e4-11e8-a720-000c293d66a5" saved
I0822 18:12:49.584972 1231887 controller.go:1110] provision "default/rbd-pv-claim2" class "ceph-rbd": succeeded
I0822 18:12:49.585737 1231887 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pv-claim2", UID:"808b1ede-a5e4-11e8-a720-000c293d66a5", APIVersion:"v1", ResourceVersion:"5099319", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-808b1ede-a5e4-11e8-a720-000c293d66a5

[wufenglinux]

我也是这个问题，但是我尝试过无数种方法。
我发现了一个让人很生气的问题
我修改了storageclass中的配置参数，这个参数
parameters:
monitors: 这个参数即使乱写 storage依然可以创建出来，并且provisioner也是报 StorageClass "rbd": missing Ceph monitors

这个错误真的很误导人，其实并不是没有ceph monitor
我到现在还没有解决这个问题。
这个问题误导了我快1个月了

[ygqygq2]

@wufenglinux 其实是他源代码中不管你写的monitor是什么，它都会去解析，当你使用外部ceph的时候，k8s dns肯定是解析不到的，所以变通方法就是给他增加解析。https://github.com/kubernetes-incubator/external-storage/issues/778

[wufenglinux]

我即使 增加了解析也还是不行
有人说是secret的问题，怀疑不是base64的，我尝试过重新加密。
但是发现依然报错，还是原来的错，所以排除了secret的问题。
到现在我这个问题依然没有解决，实在不行，我也只能重装k8s集群了。不用kubeadm安装了，
用2进制安装来解决这个问题。

[ainiml]

@ygqygq2 for the failed to get admin secret error, did you solve that? What did your Secret format look like?

[flyhigao]

after 3 hour debug ,i make it work .
1 - apiGroups: [""] resources: ["secrets"] verbs: ["create","get","list","watch"]
append this to deploy/rbac/clusterrole.yaml

2 in rbd/example/class.yaml， use mon's ip ，don't use hostname：