Project: kubedb-operator hangs after deploying with helm
After installing kubedb using the following commands, the installation never completes:
Versions:
Kubernetes CLI: /usr/local/bin/kubectl: Client Version: v1.14.1
Minikube: /usr/local/bin/minikube: minikube version: v1.0.1
Helm: /usr/local/bin/helm: Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
helm upgrade kubedb-operator appscode/kubedb --install --wait --version 0.12.0 --namespace kube-system
helm upgrade kubedb-catalog appscode/kubedb-catalog --install --wait --version 0.12.0 --namespace kube-system
I0508 20:03:11.342017 1 log.go:172] FLAG: --alsologtostderr="false"
I0508 20:03:11.342686 1 log.go:172] FLAG: --audit-dynamic-configuration="false"
I0508 20:03:11.342704 1 log.go:172] FLAG: --audit-log-batch-buffer-size="10000"
I0508 20:03:11.342707 1 log.go:172] FLAG: --audit-log-batch-max-size="1"
I0508 20:03:11.342711 1 log.go:172] FLAG: --audit-log-batch-max-wait="0s"
I0508 20:03:11.342714 1 log.go:172] FLAG: --audit-log-batch-throttle-burst="0"
I0508 20:03:11.342717 1 log.go:172] FLAG: --audit-log-batch-throttle-enable="false"
I0508 20:03:11.342721 1 log.go:172] FLAG: --audit-log-batch-throttle-qps="0"
I0508 20:03:11.342725 1 log.go:172] FLAG: --audit-log-format="json"
I0508 20:03:11.342728 1 log.go:172] FLAG: --audit-log-maxage="0"
I0508 20:03:11.342731 1 log.go:172] FLAG: --audit-log-maxbackup="0"
I0508 20:03:11.342733 1 log.go:172] FLAG: --audit-log-maxsize="0"
I0508 20:03:11.342737 1 log.go:172] FLAG: --audit-log-mode="blocking"
I0508 20:03:11.342937 1 log.go:172] FLAG: --audit-log-path="-"
I0508 20:03:11.342955 1 log.go:172] FLAG: --audit-log-truncate-enabled="false"
I0508 20:03:11.342959 1 log.go:172] FLAG: --audit-log-truncate-max-batch-size="10485760"
I0508 20:03:11.342963 1 log.go:172] FLAG: --audit-log-truncate-max-event-size="102400"
I0508 20:03:11.342966 1 log.go:172] FLAG: --audit-log-version="audit.k8s.io/v1"
I0508 20:03:11.342969 1 log.go:172] FLAG: --audit-policy-file=""
I0508 20:03:11.342972 1 log.go:172] FLAG: --audit-webhook-batch-buffer-size="10000"
I0508 20:03:11.342976 1 log.go:172] FLAG: --audit-webhook-batch-initial-backoff="10s"
I0508 20:03:11.342979 1 log.go:172] FLAG: --audit-webhook-batch-max-size="400"
I0508 20:03:11.342982 1 log.go:172] FLAG: --audit-webhook-batch-max-wait="30s"
I0508 20:03:11.342986 1 log.go:172] FLAG: --audit-webhook-batch-throttle-burst="15"
I0508 20:03:11.342989 1 log.go:172] FLAG: --audit-webhook-batch-throttle-enable="true"
I0508 20:03:11.342993 1 log.go:172] FLAG: --audit-webhook-batch-throttle-qps="10"
I0508 20:03:11.342996 1 log.go:172] FLAG: --audit-webhook-config-file=""
I0508 20:03:11.343000 1 log.go:172] FLAG: --audit-webhook-initial-backoff="10s"
I0508 20:03:11.343003 1 log.go:172] FLAG: --audit-webhook-mode="batch"
I0508 20:03:11.343006 1 log.go:172] FLAG: --audit-webhook-truncate-enabled="false"
I0508 20:03:11.343009 1 log.go:172] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
I0508 20:03:11.343012 1 log.go:172] FLAG: --audit-webhook-truncate-max-event-size="102400"
I0508 20:03:11.343016 1 log.go:172] FLAG: --audit-webhook-version="audit.k8s.io/v1"
I0508 20:03:11.343019 1 log.go:172] FLAG: --authentication-kubeconfig=""
I0508 20:03:11.343022 1 log.go:172] FLAG: --authentication-skip-lookup="false"
I0508 20:03:11.343026 1 log.go:172] FLAG: --authentication-token-webhook-cache-ttl="***REDACTED***"
I0508 20:03:11.343291 1 log.go:172] FLAG: --authentication-tolerate-lookup-failure="false"
I0508 20:03:11.343299 1 log.go:172] FLAG: --authorization-always-allow-paths="[]"
I0508 20:03:11.343303 1 log.go:172] FLAG: --authorization-kubeconfig=""
I0508 20:03:11.343306 1 log.go:172] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0508 20:03:11.343310 1 log.go:172] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0508 20:03:11.343314 1 log.go:172] FLAG: --bind-address="0.0.0.0"
I0508 20:03:11.343317 1 log.go:172] FLAG: --burst="1000000"
I0508 20:03:11.343320 1 log.go:172] FLAG: --bypass-validating-webhook-xray="false"
I0508 20:03:11.343324 1 log.go:172] FLAG: --cert-dir="apiserver.local.config/certificates"
I0508 20:03:11.343327 1 log.go:172] FLAG: --client-ca-file=""
I0508 20:03:11.343330 1 log.go:172] FLAG: --contention-profiling="false"
I0508 20:03:11.343333 1 log.go:172] FLAG: --enable-analytics="true"
I0508 20:03:11.343462 1 log.go:172] FLAG: --enable-mutating-webhook="true"
I0508 20:03:11.343475 1 log.go:172] FLAG: --enable-status-subresource="true"
I0508 20:03:11.343656 1 log.go:172] FLAG: --enable-swagger-ui="false"
I0508 20:03:11.344786 1 log.go:172] FLAG: --enable-validating-webhook="true"
I0508 20:03:11.344793 1 log.go:172] FLAG: --governing-service="kubedb"
I0508 20:03:11.344800 1 log.go:172] FLAG: --help="false"
I0508 20:03:11.344807 1 log.go:172] FLAG: --http2-max-streams-per-connection="1000"
I0508 20:03:11.344813 1 log.go:172] FLAG: --kubeconfig=""
I0508 20:03:11.344828 1 log.go:172] FLAG: --label-key-blacklist="[app.kubernetes.io/name,app.kubernetes.io/version,app.kubernetes.io/instance,app.kubernetes.io/managed-by]"
I0508 20:03:11.344839 1 log.go:172] FLAG: --log-flush-frequency="5s"
I0508 20:03:11.344846 1 log.go:172] FLAG: --log_backtrace_at=":0"
I0508 20:03:11.344851 1 log.go:172] FLAG: --log_dir=""
I0508 20:03:11.344856 1 log.go:172] FLAG: --logtostderr="false"
I0508 20:03:11.344862 1 log.go:172] FLAG: --profiling="true"
I0508 20:03:11.345262 1 log.go:172] FLAG: --qps="1e+06"
I0508 20:03:11.345287 1 log.go:172] FLAG: --rbac="true"
I0508 20:03:11.345296 1 log.go:172] FLAG: --requestheader-allowed-names="[]"
I0508 20:03:11.345300 1 log.go:172] FLAG: --requestheader-client-ca-file=""
I0508 20:03:11.345305 1 log.go:172] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0508 20:03:11.345310 1 log.go:172] FLAG: --requestheader-group-headers="[x-remote-group]"
I0508 20:03:11.345467 1 log.go:172] FLAG: --requestheader-username-headers="[x-remote-user]"
I0508 20:03:11.345507 1 log.go:172] FLAG: --restrict-to-operator-namespace="false"
I0508 20:03:11.345513 1 log.go:172] FLAG: --resync-period="10m0s"
I0508 20:03:11.345517 1 log.go:172] FLAG: --secure-port="8443"
I0508 20:03:11.345521 1 log.go:172] FLAG: --stderrthreshold="0"
I0508 20:03:11.345524 1 log.go:172] FLAG: --tls-cert-file="/var/serving-cert/tls.crt"
I0508 20:03:11.345531 1 log.go:172] FLAG: --tls-cipher-suites="[]"
I0508 20:03:11.345534 1 log.go:172] FLAG: --tls-min-version=""
I0508 20:03:11.345640 1 log.go:172] FLAG: --tls-private-key-file="/var/serving-cert/tls.key"
I0508 20:03:11.345646 1 log.go:172] FLAG: --tls-sni-cert-key="[]"
I0508 20:03:11.345650 1 log.go:172] FLAG: --use-kubeapiserver-fqdn-for-aks="true"
I0508 20:03:11.345756 1 log.go:172] FLAG: --v="3"
I0508 20:03:11.345776 1 log.go:172] FLAG: --vmodule=""
I0508 20:03:11.446419 1 run.go:24] Starting kubedb-server...
I0508 20:03:11.734189 1 lib.go:112] Kubernetes version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
I0508 20:03:11.736313 1 controller.go:72] Ensuring CustomResourceDefinition...
NAME CREATED AT
dormantdatabases.kubedb.com 2019-05-08T20:00:30Z
elasticsearches.kubedb.com 2019-05-08T20:00:28Z
elasticsearchversions.catalog.kubedb.com 2019-05-08T20:00:31Z
etcds.kubedb.com 2019-05-08T20:00:28Z
etcdversions.catalog.kubedb.com 2019-05-08T20:00:31Z
memcacheds.kubedb.com 2019-05-08T20:00:30Z
memcachedversions.catalog.kubedb.com 2019-05-08T20:00:32Z
mongodbs.kubedb.com 2019-05-08T20:00:29Z
mongodbversions.catalog.kubedb.com 2019-05-08T20:00:31Z
mysqls.kubedb.com 2019-05-08T20:00:29Z
mysqlversions.catalog.kubedb.com 2019-05-08T20:00:31Z
postgreses.kubedb.com 2019-05-08T20:00:28Z
postgresversions.catalog.kubedb.com 2019-05-08T20:00:31Z
redises.kubedb.com 2019-05-08T20:00:30Z
redisversions.catalog.kubedb.com 2019-05-08T20:00:32Z
snapshots.kubedb.com 2019-05-08T20:00:31Z
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-74lkr 1/1 Running 1 7m51s
coredns-fb8b8dccf-k6rqq 1/1 Running 1 7m51s
default-http-backend-6864bbb7db-ms42m 1/1 Running 0 7m49s
etcd-minikube 1/1 Running 0 6m44s
kube-addon-manager-minikube 1/1 Running 0 6m35s
kube-apiserver-minikube 1/1 Running 0 6m50s
kube-controller-manager-minikube 1/1 Running 0 6m58s
kube-proxy-tjgxz 1/1 Running 0 7m51s
kube-scheduler-minikube 1/1 Running 0 6m34s
kubedb-operator-6b58bb8954-8znbl 0/1 Running 4 5m37s
nginx-ingress-controller-586cdc477c-8tx5l 1/1 Running 0 7m47s
storage-provisioner 1/1 Running 0 7m47s
tiller-deploy-c48485567-f4zmx 1/1 Running 0 5m55s
bug
apobbati
👍7
All 16 comments
Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.71. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
![issue-label-bot[bot] picture](https://avatars.githubusercontent.com/in/27079?v=4&s=40)
issue-label-bot[bot]
on 8 May 2019
👍4
I'm seeing the same thing on a reasonably vanilla kops cluster
starkers
on 9 May 2019
👍1
Any ideas what's going on here?
apobbati
on 13 May 2019
Got the same issue. Installing via the script works however..
Croissong
on 18 May 2019
Same issue here...
dave08
on 10 Jun 2019
--wait flag is making the problem here.
In details, Kubedb-Operator deployment patches APIService ie, v1alpha1.mutators.kubedb.com and v1alpha1.validators.kubedb.com as a part of it's initialization. But, helm upgrade --wait creates a deadlock here.
- The workaround, ignore
--waitflag while installing kubedb-operator chart.
the-redback
on 11 Jun 2019
the-redback
on 11 Jun 2019
- Another workaround is if user want to sacrifice healthcheck to use
--waitflag, they can do it by,
helm upgrade kubedb-operator appscode/kubedb --install --wait --version 0.12.0 --namespace kube-system --set apiserver.healthcheck.enabled=false
👍2
Got the same issue(0.12.0)
I use helm template to generate raw yaml, and then kubectl -n kube-system apply
log:
I0822 05:53:17.201264 1 run.go:24] Starting kubedb-server...
I0822 05:53:17.497542 1 lib.go:112] Kubernetes version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.4", GitCommit:"a87e9a978f65a8303aa9467537aa59c18122cbf9", GitTreeState:"clean", BuildDate:"2019-07-08T08:43:10Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
I0822 05:53:17.505146 1 controller.go:72] Ensuring CustomResourceDefinition...
Error: timed out waiting for the condition
crds:
[root@node1]# kubectl get crds -l app=kubedb -w
NAME AGE
dormantdatabases.kubedb.com 56m
elasticsearches.kubedb.com 56m
elasticsearchversions.catalog.kubedb.com 56m
etcds.kubedb.com 56m
etcdversions.catalog.kubedb.com 56m
memcacheds.kubedb.com 56m
memcachedversions.catalog.kubedb.com 56m
mongodbs.kubedb.com 56m
mongodbversions.catalog.kubedb.com 56m
mysqls.kubedb.com 56m
mysqlversions.catalog.kubedb.com 56m
postgreses.kubedb.com 56m
postgresversions.catalog.kubedb.com 56m
redises.kubedb.com 56m
redisversions.catalog.kubedb.com 56m
snapshots.kubedb.com 56m
lixianyang
on 22 Aug 2019
👀2
👍2
Got the same issue with v0.12.0 on kubernetes 1.16
yujunz
on 15 Nov 2019
👍2
same issue with v0.12.0 on kubernetes 1.13.11, it's a fundamental check. It's difficult to skip it
rtrive
on 12 Feb 2020
helm --wait flag is fixed by https://github.com/kubedb/installer/pull/57
tamalsaha
on 22 Apr 2020
Bug is still present with 0.13.0-rc.0, same behavior I got with 0.12.0
I install using Helm 3 from Terraform:
resource "helm_release" "kubedb" {
name = "kubedb-operator"
namespace = "kube-system"
version = "0.13.0-rc.0"
repository = "https://charts.appscode.com/stable/"
chart = "kubedb"
}
But the pod will just hang:
[kubedb-operator-6d9cb9cb4-zxlsv operator] I0617 07:10:53.577439 1 run.go:24] Starting kubedb-server...
[kubedb-operator-6d9cb9cb4-zxlsv operator] I0617 07:10:54.873915 1 lib.go:112] Kubernetes version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
[kubedb-operator-6d9cb9cb4-zxlsv operator] I0617 07:10:54.900537 1 controller.go:75] Ensuring CustomResourceDefinition...
But CRDs had already been created (they were installed yesterday by 0.12.0, today I tried 0.13.0-rc.0
(base) laurent-mbp:~ laurent$ kubectl get crds -l app=kubedb
NAME CREATED AT
dormantdatabases.kubedb.com 2020-06-16T18:51:44Z
elasticsearches.kubedb.com 2020-06-16T18:51:37Z
elasticsearchversions.catalog.kubedb.com 2020-06-16T18:51:45Z
etcds.kubedb.com 2020-06-16T18:51:38Z
etcdversions.catalog.kubedb.com 2020-06-16T18:51:45Z
memcacheds.kubedb.com 2020-06-16T18:51:44Z
memcachedversions.catalog.kubedb.com 2020-06-16T18:51:46Z
mongodbs.kubedb.com 2020-06-16T18:51:41Z
mongodbversions.catalog.kubedb.com 2020-06-16T18:51:46Z
mysqls.kubedb.com 2020-06-16T18:51:40Z
mysqlversions.catalog.kubedb.com 2020-06-16T18:51:46Z
postgreses.kubedb.com 2020-06-16T18:51:39Z
postgresversions.catalog.kubedb.com 2020-06-16T18:51:45Z
redises.kubedb.com 2020-06-16T18:51:42Z
redisversions.catalog.kubedb.com 2020-06-16T18:51:46Z
snapshots.kubedb.com 2020-06-16T18:51:45Z
I'm running Kubernetes 1.18.3 on managed service Scaleway Kapsule.
Thanks!
debackerl
on 17 Jun 2020
👍2
It's also hanging for me - also whilst using the helm3 terraform provider - specifying wait = false in the helm_release resource is a work around for now
vCra
on 27 Jun 2020
I'm trying to manage this installation using ArgoCD, so I don't seem to have any control over this --wait flag. Is a new release planned soon?
bitfactory-henno-schooljan
on 29 Jun 2020
👍1
This has been fixed in latest release.
tamalsaha
on 19 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings
Related issues
botzill
路
5Comments
david-ecc
路
3Comments
david-ecc
路
5Comments
mauritsvdvijgh
路
5Comments
DamiaPoquet
路
3Comments
Most helpful comment
Issue-Label Bot is automatically applying the label
bugto this issue, with a confidence of 0.71. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!Links: app homepage, dashboard and code for this bot.