Project: [postgres] Error retrieving resource lock - causes db restarts

[olksdr]

We're running few testing clusters in google cloud and it happens sometimes that the postgres operator gets restarted and take the DBs down as well.
The errors:

I0928 03:57:18.072148       1 log.go:172] Get https://10.11.240.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.11.240.1:443: i/o timeout

I0928 03:54:11.981350       1 leaderelection.go:263] failed to renew lease default/pg-leader-lock: timed out waiting for the condition

See the logs below:

Kubedb operator logs

I0928 03:56:47.771658       1 log.go:172] FLAG: --alsologtostderr="false"
I0928 03:56:47.771704       1 log.go:172] FLAG: --audit-dynamic-configuration="false"
I0928 03:56:47.771711       1 log.go:172] FLAG: --audit-log-batch-buffer-size="10000"
I0928 03:56:47.771715       1 log.go:172] FLAG: --audit-log-batch-max-size="1"
I0928 03:56:47.771720       1 log.go:172] FLAG: --audit-log-batch-max-wait="0s"
I0928 03:56:47.771724       1 log.go:172] FLAG: --audit-log-batch-throttle-burst="0"
I0928 03:56:47.771728       1 log.go:172] FLAG: --audit-log-batch-throttle-enable="false"
I0928 03:56:47.771733       1 log.go:172] FLAG: --audit-log-batch-throttle-qps="0"
I0928 03:56:47.771737       1 log.go:172] FLAG: --audit-log-format="json"
I0928 03:56:47.771742       1 log.go:172] FLAG: --audit-log-maxage="0"
I0928 03:56:47.771751       1 log.go:172] FLAG: --audit-log-maxbackup="0"
I0928 03:56:47.771755       1 log.go:172] FLAG: --audit-log-maxsize="0"
I0928 03:56:47.771760       1 log.go:172] FLAG: --audit-log-mode="blocking"
I0928 03:56:47.771765       1 log.go:172] FLAG: --audit-log-path="-"
I0928 03:56:47.771769       1 log.go:172] FLAG: --audit-log-truncate-enabled="false"
I0928 03:56:47.771777       1 log.go:172] FLAG: --audit-log-truncate-max-batch-size="10485760"
I0928 03:56:47.771782       1 log.go:172] FLAG: --audit-log-truncate-max-event-size="102400"
I0928 03:56:47.771786       1 log.go:172] FLAG: --audit-log-version="audit.k8s.io/v1"
I0928 03:56:47.771790       1 log.go:172] FLAG: --audit-policy-file=""
I0928 03:56:47.771794       1 log.go:172] FLAG: --audit-webhook-batch-buffer-size="10000"
I0928 03:56:47.771799       1 log.go:172] FLAG: --audit-webhook-batch-initial-backoff="10s"
I0928 03:56:47.771804       1 log.go:172] FLAG: --audit-webhook-batch-max-size="400"
I0928 03:56:47.771808       1 log.go:172] FLAG: --audit-webhook-batch-max-wait="30s"
I0928 03:56:47.771812       1 log.go:172] FLAG: --audit-webhook-batch-throttle-burst="15"
I0928 03:56:47.771817       1 log.go:172] FLAG: --audit-webhook-batch-throttle-enable="true"
I0928 03:56:47.771822       1 log.go:172] FLAG: --audit-webhook-batch-throttle-qps="10"
I0928 03:56:47.771826       1 log.go:172] FLAG: --audit-webhook-config-file=""
I0928 03:56:47.771830       1 log.go:172] FLAG: --audit-webhook-initial-backoff="10s"
I0928 03:56:47.771834       1 log.go:172] FLAG: --audit-webhook-mode="batch"
I0928 03:56:47.771839       1 log.go:172] FLAG: --audit-webhook-truncate-enabled="false"
I0928 03:56:47.771843       1 log.go:172] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
I0928 03:56:47.771853       1 log.go:172] FLAG: --audit-webhook-truncate-max-event-size="102400"
I0928 03:56:47.771857       1 log.go:172] FLAG: --audit-webhook-version="audit.k8s.io/v1"
I0928 03:56:47.771861       1 log.go:172] FLAG: --authentication-kubeconfig=""
I0928 03:56:47.771866       1 log.go:172] FLAG: --authentication-skip-lookup="false"
I0928 03:56:47.771873       1 log.go:172] FLAG: --authentication-token-webhook-cache-ttl="***REDACTED***"
I0928 03:56:47.771878       1 log.go:172] FLAG: --authentication-tolerate-lookup-failure="false"
I0928 03:56:47.771885       1 log.go:172] FLAG: --authorization-always-allow-paths="[]"
I0928 03:56:47.771892       1 log.go:172] FLAG: --authorization-kubeconfig=""
I0928 03:56:47.771897       1 log.go:172] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0928 03:56:47.771902       1 log.go:172] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I0928 03:56:47.771906       1 log.go:172] FLAG: --bind-address="0.0.0.0"
I0928 03:56:47.771911       1 log.go:172] FLAG: --burst="1000000"
I0928 03:56:47.771916       1 log.go:172] FLAG: --bypass-validating-webhook-xray="false"
I0928 03:56:47.771930       1 log.go:172] FLAG: --cert-dir="apiserver.local.config/certificates"
I0928 03:56:47.771950       1 log.go:172] FLAG: --client-ca-file=""
I0928 03:56:47.771955       1 log.go:172] FLAG: --contention-profiling="false"
I0928 03:56:47.771959       1 log.go:172] FLAG: --enable-analytics="true"
I0928 03:56:47.771964       1 log.go:172] FLAG: --enable-mutating-webhook="true"
I0928 03:56:47.771968       1 log.go:172] FLAG: --enable-status-subresource="true"
I0928 03:56:47.771972       1 log.go:172] FLAG: --enable-swagger-ui="false"
I0928 03:56:47.771977       1 log.go:172] FLAG: --enable-validating-webhook="true"
I0928 03:56:47.771982       1 log.go:172] FLAG: --governing-service="kubedb"
I0928 03:56:47.771987       1 log.go:172] FLAG: --help="false"
I0928 03:56:47.771992       1 log.go:172] FLAG: --http2-max-streams-per-connection="1000"
I0928 03:56:47.771996       1 log.go:172] FLAG: --kubeconfig=""
I0928 03:56:47.772006       1 log.go:172] FLAG: --label-key-blacklist="[app.kubernetes.io/name,app.kubernetes.io/version,app.kubernetes.io/instance,app.kubernetes.io/managed-by]"
I0928 03:56:47.772018       1 log.go:172] FLAG: --log-flush-frequency="5s"
I0928 03:56:47.772025       1 log.go:172] FLAG: --log_backtrace_at=":0"
I0928 03:56:47.772030       1 log.go:172] FLAG: --log_dir=""
I0928 03:56:47.772034       1 log.go:172] FLAG: --logtostderr="true"
I0928 03:56:47.772039       1 log.go:172] FLAG: --profiling="true"
I0928 03:56:47.772051       1 log.go:172] FLAG: --qps="1e+06"
I0928 03:56:47.772056       1 log.go:172] FLAG: --rbac="true"
I0928 03:56:47.772062       1 log.go:172] FLAG: --requestheader-allowed-names="[]"
I0928 03:56:47.772068       1 log.go:172] FLAG: --requestheader-client-ca-file=""
I0928 03:56:47.772074       1 log.go:172] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I0928 03:56:47.772081       1 log.go:172] FLAG: --requestheader-group-headers="[x-remote-group]"
I0928 03:56:47.772086       1 log.go:172] FLAG: --requestheader-username-headers="[x-remote-user]"
I0928 03:56:47.772091       1 log.go:172] FLAG: --restrict-to-operator-namespace="false"
I0928 03:56:47.772095       1 log.go:172] FLAG: --resync-period="10m0s"
I0928 03:56:47.772100       1 log.go:172] FLAG: --secure-port="8443"
I0928 03:56:47.772104       1 log.go:172] FLAG: --stderrthreshold="0"
I0928 03:56:47.772109       1 log.go:172] FLAG: --tls-cert-file="/var/serving-cert/tls.crt"
I0928 03:56:47.772115       1 log.go:172] FLAG: --tls-cipher-suites="[]"
I0928 03:56:47.772120       1 log.go:172] FLAG: --tls-min-version=""
I0928 03:56:47.772125       1 log.go:172] FLAG: --tls-private-key-file="/var/serving-cert/tls.key"
I0928 03:56:47.772129       1 log.go:172] FLAG: --tls-sni-cert-key="[]"
I0928 03:56:47.772137       1 log.go:172] FLAG: --use-kubeapiserver-fqdn-for-aks="true"
I0928 03:56:47.772141       1 log.go:172] FLAG: --v="3"
I0928 03:56:47.772145       1 log.go:172] FLAG: --vmodule=""
I0928 03:56:47.814272       1 run.go:24] Starting kubedb-server...
Error: Get https://10.11.240.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.11.240.1:443: i/o timeout
Usage:
  kubedb-operator run [flags]

Flags:
      --audit-dynamic-configuration                             Enables dynamic audit configuration. This feature also requires the DynamicAuditing feature flag
      --audit-log-batch-buffer-size int                         The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-log-batch-max-size int                            The maximum size of a batch. Only used in batch mode. (default 1)
      --audit-log-batch-max-wait duration                       The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.
      --audit-log-batch-throttle-burst int                      Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
      --audit-log-batch-throttle-enable                         Whether batching throttling is enabled. Only used in batch mode.
      --audit-log-batch-throttle-qps float32                    Maximum average number of batches per second. Only used in batch mode.
      --audit-log-format string                                 Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json. (default "json")
      --audit-log-maxage int                                    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      --audit-log-maxbackup int                                 The maximum number of old audit log files to retain.
      --audit-log-maxsize int                                   The maximum size in megabytes of the audit log file before it gets rotated.
      --audit-log-mode string                                   Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "blocking")
      --audit-log-path string                                   If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      --audit-log-truncate-enabled                              Whether event and batch truncating is enabled.
      --audit-log-truncate-max-batch-size int                   Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      --audit-log-truncate-max-event-size int                   Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      --audit-log-version string                                API group and version used for serializing audit events written to log. (default "audit.k8s.io/v1")
      --audit-policy-file string                                Path to the file that defines the audit policy configuration.
      --audit-webhook-batch-buffer-size int                     The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-webhook-batch-max-size int                        The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-webhook-batch-max-wait duration                   The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      --audit-webhook-batch-throttle-burst int                  Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      --audit-webhook-batch-throttle-enable                     Whether batching throttling is enabled. Only used in batch mode. (default true)
      --audit-webhook-batch-throttle-qps float32                Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-webhook-config-file string                        Path to a kubeconfig formatted file that defines the audit webhook configuration.
      --audit-webhook-initial-backoff duration                  The amount of time to wait before retrying the first failed request. (default 10s)
      --audit-webhook-mode string                               Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "batch")
      --audit-webhook-truncate-enabled                          Whether event and batch truncating is enabled.
      --audit-webhook-truncate-max-batch-size int               Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      --audit-webhook-truncate-max-event-size int               Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      --audit-webhook-version string                            API group and version used for serializing audit events written to webhook. (default "audit.k8s.io/v1")
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authentication-tolerate-lookup-failure                  If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
      --authorization-always-allow-paths strings                A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --burst int                                               The maximum burst for throttle (default 1000000)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
      --enable-mutating-webhook                                 If true, enables mutating webhooks for KubeDB CRDs. (default false)
      --enable-status-subresource                               If true, uses sub resource for KubeDB crds. (default false)
      --enable-validating-webhook                               If true, enables validating webhooks for KubeDB CRDs. (default false)
      --governing-service string                                Governing service for database statefulset (default "kubedb")
  -h, --help                                                    help for run
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. (default 1000)
      --kubeconfig string                                       kubeconfig file pointing at the 'core' kubernetes server.
      --label-key-blacklist strings                             list of keys that are not propagated from a CRD object to its offshoots (default [app.kubernetes.io/name,app.kubernetes.io/version,app.kubernetes.io/instance,app.kubernetes.io/managed-by])
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --qps float                                               The maximum QPS to the master from this client (default 1e+06)
      --rbac                                                    Enable RBAC for operator & offshoot Kubernetes objects (default true)
      --requestheader-allowed-names strings                     List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      --requestheader-extra-headers-prefix strings              List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers strings                     List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers strings                  List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --restrict-to-operator-namespace                          If true, KubeDB operator will only handle Kubernetes objects in its own namespace.
      --resync-period duration                                  If non-zero, will re-list this often. Otherwise, re-list will be delayed aslong as possible (until the upstream source closes the watch or times out. (default 10m0s)
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 443)
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites strings                               Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
      --tls-min-version string                                  Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])

Global Flags:
      --alsologtostderr                  log to standard error as well as files
      --bypass-validating-webhook-xray   if true, bypasses validating webhook xray checks
      --enable-analytics                 Send analytical events to Google Analytics (default true)
      --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --logtostderr                      log to standard error instead of files (default true)
      --stderrthreshold severity         logs at or above this threshold go to stderr
      --use-kubeapiserver-fqdn-for-aks   if true, uses kube-apiserver FQDN for AKS cluster to workaround https://github.com/Azure/AKS/issues/522 (default true)
  -v, --v Level                          log level for V logs (default 0)
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

I0928 03:57:18.072148       1 log.go:172] Get https://10.11.240.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.11.240.1:443: i/o timeout

postgres logs

I0927 07:01:12.284050       1 log.go:172] FLAG: --alsologtostderr="false"
I0927 07:01:12.284163       1 log.go:172] FLAG: --bypass-validating-webhook-xray="false"
I0927 07:01:12.284169       1 log.go:172] FLAG: --enable-analytics="true"
I0927 07:01:12.284174       1 log.go:172] FLAG: --help="false"
I0927 07:01:12.284179       1 log.go:172] FLAG: --log-flush-frequency="5s"
I0927 07:01:12.284184       1 log.go:172] FLAG: --log_backtrace_at=":0"
I0927 07:01:12.284190       1 log.go:172] FLAG: --log_dir=""
I0927 07:01:12.284205       1 log.go:172] FLAG: --logtostderr="true"
I0927 07:01:12.284213       1 log.go:172] FLAG: --stderrthreshold="0"
I0927 07:01:12.284221       1 log.go:172] FLAG: --use-kubeapiserver-fqdn-for-aks="true"
I0927 07:01:12.284227       1 log.go:172] FLAG: --v="3"
I0927 07:01:12.284235       1 log.go:172] FLAG: --vmodule=""
We want "pg-0" as our leader
I0927 07:01:12.414052       1 leaderelection.go:217] attempting to acquire leader lease  default/pg-leader-lock...
I0927 07:01:12.425235       1 leaderelection.go:227] successfully acquired lease default/pg-leader-lock
Got leadership, now do your jobs
I0927 07:01:12.460562       1 pod.go:56] Patching Pod default/pg-0 with {"metadata":{"labels":{"kubedb.com/role":"primary"}}}
Running as Primary

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
ALTER ROLE


/scripts/primary/start.sh: ignoring /var/initdb/*

2019-09-27 07:01:14.506 UTC [18] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-09-27 07:01:14.506 UTC [18] LOG:  listening on IPv6 address "::", port 5432
2019-09-27 07:01:14.515 UTC [18] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-09-27 07:01:14.799 UTC [63] LOG:  database system was shut down at 2019-09-27 07:01:14 UTC
2019-09-27 07:01:14.814 UTC [18] LOG:  database system is ready to accept connections
2019-09-27 07:49:45.266 UTC [146] ERROR:  schema "public" already exists
2019-09-27 07:49:45.266 UTC [146] STATEMENT:  CREATE SCHEMA public
E0928 03:52:51.982561       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:01.984162       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:11.984617       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:21.984379       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:31.984339       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:41.984228       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:53:51.984257       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
E0928 03:54:01.984162       1 leaderelection.go:306] error retrieving resource lock default/pg-leader-lock: Get https://10.11.240.1:443/api/v1/namespaces/default/configmaps/pg-leader-lock: dial tcp 10.11.240.1:443: connect: connection refused
I0928 03:54:11.981350       1 leaderelection.go:263] failed to renew lease default/pg-leader-lock: timed out waiting for the condition
Lost leadership, now quit
 ```
</details>

The current leader election settings are following:
```bash
  leaderElection:
    leaseDurationSeconds: 120
    renewDeadlineSeconds: 80
    retryPeriodSeconds: 10

which gives some breathing room, but at the end it doesn't help as you can see in logs.

One more thing to notice that during this time the API on the cluster is not accessible with with similar errors:

"github.com/GoogleCloudPlatform/k8s-stackdriver/event-exporter/watchers/watcher.go:50: Failed to watch *v1.Event: Get https://10.59.240.1:443/api/v1/events?resourceVersion=2029&timeoutSeconds=528&watch=true: dial tcp 10.59.240.1:443: getsockopt: connection refused"

```bash
"k8s.io/dns/pkg/dns/dns.go:192: Failed to watch *v1.Service: Get https://10.59.240.1:443/api/v1/services?resourceVersion=302399&timeoutSeconds=517&watch=true: dial tcp 10.59.240.1:443: connect: connection refused"

But all of those services got recovered  without pods being restarted but the kubedb operator could not handle these issues:

See pods statuses:
```bash
event-exporter-v0.2.4-5f88c66fb7-nb42g               2/2     Running   0          2d7h
fluentd-gcp-scaler-59b7b75cd7-4pzpt                  1/1     Running   0          2d7h
fluentd-gcp-v3.2.0-7m9kn                             2/2     Running   0          2d7h
fluentd-gcp-v3.2.0-8r4tj                             2/2     Running   0          2d7h
fluentd-gcp-v3.2.0-g47fn                             2/2     Running   0          7h34m
heapster-v1.6.1-7db8b55fbc-4dd2d                     3/3     Running   0          5h21m
kube-dns-6987857fdb-kjqp2                            4/4     Running   0          2d7h
kube-dns-6987857fdb-zlm2r                            4/4     Running   0          2d7h
kube-dns-autoscaler-bb58c6784-zghql                  1/1     Running   0          2d7h
kube-proxy-gke-tilegen6-default-pool-de3673c0-j2gp   1/1     Running   0          2d7h
kube-proxy-gke-tilegen6-default-pool-de3673c0-wmg9   1/1     Running   0          7h34m
kube-proxy-gke-tilegen6-large-nodes-4760ec35-kf3d    1/1     Running   0          2d7h
kubedb-operator-7bd7cd488f-tsvgb                     1/1     Running   13         2d7h
l7-default-backend-fd59995cd-nnx75                   1/1     Running   0          2d7h
metrics-server-v0.3.1-57c75779f-76nf8                2/2     Running   0          2d7h
prometheus-to-sd-m6lww                               1/1     Running   0          2d7h
prometheus-to-sd-mccbc                               1/1     Running   0          7h34m
prometheus-to-sd-pbbps                               1/1     Running   0          2d7h
tiller-deploy-5bdc5b88f7-wmz7h                       1/1     Running   0          2d7h

---

• Are there maybe some suggestions to adjust the settings to avoid the kubedb operator and postgres restarting ?

• It might be a bug and some re-tries should be added to avoid this situation

It would be nice to hear from core developers on this issue. Thank you!

[sharkymcdongles]

+1 I am also experiencing the same issue on AKS. It seems when it can't connect to API server even intermittently to verify CRDs and catalog, the operator restarts taking all managed objects with it. Would be nice to have a retry mechanism maybe?

[sharkymcdongles]

So I tried to prevent this by setting lease duration since we dont use replicas atm:
leaseDurationSeconds: 315569520

However the behavior remains the same. Would be great if someone could chime in on this as it causes downtimes.

[stephan2012]

Looks like this happens when one of the Kubernetes API service is (temporarily) unavailable, maybe due to a master node failure . It can take some time until kube-proxy updates the endpoints on all nodes. Depending on how the API server faded away connection attempts result in connection reset (node up, API server down) or connection timeout (node down).

So probably a better solution is to retry connecting to the Kubernetes API server for some (tunable) time before giving up.

Also, when one replica is unable to connect to the API server the other probably hits the same issue essentially taking the database offline.

[akaipham]

Any new updates?

[tazhate]

This bug is currently blocker for me to use kubedb in production. Switched to cranchy until it will be fixed.

[hossainemruz]

@tazhate, @akaipham In the latest release, we have removed dependency on Kubernetes for PostgreSQL leader election. We now use raft algorithm for PostgreSQL clustering. So, this problem should be fixed now.