Rook: Secrets "rook-ceph-csi" not found (fresh install of Ceph 1.1.4)

[stephan2012]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
Secret rook-ceph-csi not created by Rook Ceph operator causing CephFS volume provisioning to fail. Maybe a regression due to #3874?

Expected behavior:
Operator should create secret rook-ceph-csi

How to reproduce it (minimal and precise):

1. Setup Rook Ceph operator
2. Deploy Ceph cluster and CephFS
3. Create PVC

File(s) to submit:

I1029 08:15:03.832515       1 controller.go:1196] provision "default/data-my-app-storage-b-fs" class "rook-cephfs": started
I1029 08:15:03.842239       1 controller.go:471] CreateVolumeRequest {Name:pvc-cb279805-9d03-44db-8547-5026c3d4ea3f CapacityRange:required_bytes:53687091200  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"_netdev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Parameters:map[clusterID:rook-ceph csi.storage.k8s.io/node-stage-secret-name:rook-ceph-csi csi.storage.k8s.io/node-stage-secret-namespace:rook-ceph csi.storage.k8s.io/provisioner-secret-name:rook-ceph-csi csi.storage.k8s.io/provisioner-secret-namespace:rook-ceph fsName:cephfs mounter:kernel pool:cephfs-data0] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1029 08:15:03.842464       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-my-app-storage-b-fs", UID:"cb279805-9d03-44db-8547-5026c3d4ea3f", APIVersion:"v1", ResourceVersion:"381636", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/data-my-app-storage-b-fs"
I1029 08:15:03.865472       1 controller.go:979] Final error received, removing PVC cb279805-9d03-44db-8547-5026c3d4ea3f from claims in progress
W1029 08:15:03.865516       1 controller.go:886] Retrying syncing claim "cb279805-9d03-44db-8547-5026c3d4ea3f", failure 9
E1029 08:15:03.865548       1 controller.go:908] error syncing claim "cb279805-9d03-44db-8547-5026c3d4ea3f": failed to provision volume with StorageClass "rook-cephfs": error getting secret rook-ceph-csi in namespace rook-ceph: secrets "rook-ceph-csi" not found
I1029 08:15:03.865590       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-my-app-storage-b-fs", UID:"cb279805-9d03-44db-8547-5026c3d4ea3f", APIVersion:"v1", ResourceVersion:"381636", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": error getting secret rook-ceph-csi in namespace rook-ceph: secrets "rook-ceph-csi" not found
I1029 08:15:04.202037       1 leaderelection.go:258] successfully renewed lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
I1029 08:15:07.579738       1 reflector.go:370] sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:800: Watch close - *v1.PersistentVolumeClaim total 8 items received

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
• Kernel (e.g. uname -a): Linux n0204 5.0.0-31-generic #33~18.04.1-Ubuntu SMP Tue Oct 1 10:20:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Cloud provider or hardware configuration: Kubernetes 1.15.4, setup by kubeadm on VMs (no cloud provider)
• Rook version (use rook version inside of a Rook Pod): 1.1.4
• Storage backend version (e.g. for ceph do ceph -v): ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
• Kubernetes version (use kubectl version): 1.15.4
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_WARN too few PGs per OSD (8 < min 30)

[stephan2012]

Hm… The operator reports

2019-10-29 08:07:48.726385 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph"

but there is no secret.

[leseb]

Did you use the storageclass example provided with 1.1.4?
Things changed with https://github.com/rook/rook/pull/4144

[stephan2012]

Yes, I did. Adjusted secret name in StorageClass (in fact, it has to be deleted and recreated) and now it works.

parameters:
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

Thanks for the pointer!