Project: Apiservices fail to initialize on Minikube

[borremosch]

I am trying to get KubeDB running on Minikube without a VM, on WSL 2. During the installation, I get the following output:

checking kubeconfig context
minikube

checking whether extended apiserver feature is enabled

KUBEDB_DOCKER_REGISTRY=kubedb
KUBEDB_ENABLE_ANALYTICS=true
KUBEDB_ENABLE_APISERVER=true
KUBEDB_ENABLE_MUTATING_WEBHOOK=true
KUBEDB_ENABLE_RBAC=true
KUBEDB_ENABLE_VALIDATING_WEBHOOK=true
KUBEDB_IMAGE_PULL_POLICY=IfNotPresent
KUBEDB_IMAGE_PULL_SECRET=
KUBEDB_NAMESPACE=kube-system
KUBEDB_OPERATOR_NAME=operator
KUBEDB_OPERATOR_TAG=0.8.0
KUBEDB_PURGE=0
KUBEDB_RUN_ON_MASTER=0
KUBEDB_SERVICE_ACCOUNT=kubedb-operator
KUBEDB_UNINSTALL=0

Wrote ca certificates in  /home/borre/cubonacci
Wrote server certificates in  /home/borre/cubonacci
deployment.apps/kubedb-operator configured
secret/kubedb-server-cert configured
service/kubedb-operator unchanged
apiservice.apiregistration.k8s.io/v1alpha1.validators.kubedb.com configured
apiservice.apiregistration.k8s.io/v1alpha1.mutators.kubedb.com configured
serviceaccount/kubedb-operator unchanged
clusterrole.rbac.authorization.k8s.io/kubedb-operator reconciled
clusterrolebinding.rbac.authorization.k8s.io/kubedb-operator reconciled
rolebinding.rbac.authorization.k8s.io/kubedb-server-extension-server-authentication-reader reconciled
clusterrolebinding.rbac.authorization.k8s.io/kubedb-server-auth-delegator reconciled
clusterrole.rbac.authorization.k8s.io/kubedb:core:admin reconciled
clusterrole.rbac.authorization.k8s.io/kubedb:core:edit reconciled
clusterrole.rbac.authorization.k8s.io/kubedb:core:view reconciled
validatingwebhookconfiguration.admissionregistration.k8s.io/validators.kubedb.com unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutators.kubedb.com unchanged

waiting until kubedb operator deployment is ready
waiting until kubedb apiservice is available
timed out waiting for the condition
KubeDB apiservice v1alpha1.validators failed to be ready

When I check the status of the apiservices using kubectl get apiservices, I see:

NAME                                   SERVICE                       AVAILABLE                      AGE
[...]
v1alpha1.mutators.kubedb.com           kube-system/kubedb-operator   False (FailedDiscoveryCheck)   1m
v1alpha1.validators.kubedb.com         kube-system/kubedb-operator   False (FailedDiscoveryCheck)   1m
[...]

The output of kubectl describe apiservice v1alpha1.validators.kubedb.com is:

Name:         v1alpha1.validators.kubedb.com
Namespace:
Labels:       app=kubedb
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"app":"kubedb"},"name":"v1alpha1...
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2019-06-26T14:46:44Z
  Resource Version:    5988
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.validators.kubedb.com
  UID:                 37ac1de2-9821-11e9-83c7-6226b2628906
Spec:
  Ca Bundle:               LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN1RENDQWFDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFOTVFzd0NRWURWUVFERXdKallUQWUKRncweE9UQTJNall4TlRNNU5ERmFGdzB5T1RBMk1qTXhOVE01TkRGYU1BMHhDekFKQmdOVkJBTVRBbU5oTUlJQgpJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd0NBRzROb1pkalRiTUxPczBOeUV0L3RkCm4ydUViVkNncXUzRkI3SGdmRTM2NGl2Vk5vK3cweDU3aEIzWDk0dlYvWkRSVmVEUVovc1hIT09mUmc3NE8weloKT3JIU0ZtYXREZzg1cmlHMlYxd2xPdi9GdXNVNHVKSTRRMU03emRmQ0ZzNmdoVWV3eDM3SThyRFo5L1NERFM0SgppWHduZlc3M2Q5aXhURFJ3eHdPUkdGNjBDTTdRd1ljdG1IaUdFZU5HczdPSUNOTHdzY3YyYklGQVJjMk9GNzh6CnNHMmRrOUEwUlVTM3dPZTNlVkpFSDhWbGl5WmdoeVJkdy9GZHNvT2hWT1gvbzBsK0pRcWc1T1ZCbjdpUEhoY0UKdUtMc0E0eGdrcGp5YVkzdG5iRFVUeG0yZGY2RUxpS3ZYcjlZMzhyZmhkRHNBeWYvSVp4ZWJzd1NsZ0tUbXdJRApBUUFCb3lNd0lUQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHCjl3MEJBUXNGQUFPQ0FRRUFCb2pSQW9FMVFFdHlkdUtJZWtOZ2dsRkM1S1E5VlpuYnJlWCtvbkRWZVpaNlFpYzkKQW9rMTd2NlVxQVdsYzR0aDZpOElZM3REZitJeXFyY1o5WVBkOC9nQ1lBc0FDTzBidEpjVnZwU3JSazBoT0o5egpETVIyTm1wRHBNNFZiRlNhdWtpR2lxaTY1cFlBTUpjZXJYK0duQ29rU0dxc0YxQmxnTjQ4bkZBbk9SWlJsVFB6CjdkMVQ4Tmt6QXYyODNramNNVHlYbDlETUp1OWV0WldhakN2ZVpLdEVJMy83WEMvYnVxekIvQnRWZlpNZTJ3MlgKTG95YlRxQnBudUZKYkdHTlNNcEZRMWQvYjZ2OVhDM09vNWZ6b3hpMnVJT1NYNWRPWjdUalduRmFTWlEwK1Qzbwp4b2RZMGx3WmN4VHJqT3J1UXhhZ3FDMkd1OUJlY3ZqMC9FUXFHZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  Group:                   validators.kubedb.com
  Group Priority Minimum:  1000
  Service:
    Name:            kubedb-operator
    Namespace:       kube-system
  Version:           v1alpha1
  Version Priority:  15
Status:
  Conditions:
    Last Transition Time:  2019-06-26T14:46:44Z
    Message:               no response from https://10.106.246.92:443: Get https://10.106.246.92:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

Any help would be greatly appreciated

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.80. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[donaldguy]

To pull the salient detail over from ^

I can't speak for OP, but for my part this seems to be downstream of #504 #655

This is to say it seems to be a TLS issue in the operator pod probes and adding --set apiserver.healthcheck.enabled=false to helm chart install seems to at least unblock the install.

You know... if I had to guess this might be as simple as a race in TLS generation vs the rest of the initialization rolling

Doing:

$ kubectl port-forward -n kube-system kubedb-operator-d8756579b-xmk9q 8443
# then
$ openssl s_client -connect 127.0.0.1:8443 | openssl x509 -noout -text

I see a Not Before: Nov 7 21:51:54 2019 GMT where my install was done LAST DEPLOYED: Thu Nov 7 14:55:55 2019 – so like that implies the cert is backdated a few minutes ... but is minikube coming up with a correct clock immediately?

idk just a theory

[donaldguy]

(It appears I started the minikube cluster in question, rapidly iterating to test this, about 14:49 (21:49 UTC) )

[yujunz]

I have met similar issue on a cluster deployed with Kubespray

[janvdvegt]

I'm running into this issue with a cluster deployed with kops, kubernetes v1.15.10 and kubeDB 0.12.

[virtuman]

Same here on GKE clusters 1.15.11+ and latest kubedb version 0.12 and 0.13.0-rc-0 installed with helm or installer script.

[vikas-shaw]

I am also running with same issue GKE version 1.14.10-gke.36 and kubedb latest version 0.13.0-rc-0 tried installing with both helm and installer script.

[vikas-shaw]

I added port 8443 in firewall rule of kube master node then mutator and validator apiservices worked.

[dhurley94]

Same issue here, GKE 1.14.10-gke.36 and kubedb version v0.13.0-rc.0. The issue occurs if GKE is configured as a private cluster

[bitfactory-henno-schooljan]

Same issue here, GKE 1.14.10-gke.36 and kubedb version v0.13.0-rc.0. The issue occurs if GKE is configured as a private cluster

Did you find any workaround other than not using a private cluster (which is not an option for our security requirements)? Why would a private cluster prevent communication to pods within the cluster?

[bitfactory-henno-schooljan]

This appears to be a VPC firewall issue because the port is not 443: https://github.com/kubernetes/kubernetes/issues/79739