K8s-config-connector: configconnector-operator-0 keeps getting OOMKilled, Error during reconciliation: error applying manifest: error from running kubectl apply: signal: killed
Describe the bug
We're running Config Connector via GKE addon, and the configconnector-operator-0 pod in the configconnector-operator-system namespace keeps getting OOMKilled. The STS sets 100Mi request and 200Mi, and it's not possible to modify without the changes getting overwritten pretty fast.
ConfigConnector Version
1.19.1, the one that comes with kubernetes version 1.16.13-gke.401 in GKE.
To Reproduce
I'm not sure how to reproduce this. It just doesn't have enough memory and there's no way for us to modify it. When I edit the STS and delete the pod, I can see its logs doing stuff for a while until the STS reverts back.
In case you need more context, this is a staging GKE cluster where we run full environments of our app, one version in each namespace we create randomly. Each namespace has a bunch of things we manage via Config Connector. These are all the logs we see on that pod (as far as I can tell, they always end at the exact same line):
2020-09-21T21:19:39.403Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-09-21T21:19:41.510Z INFO setup starting manager
2020-09-21T21:19:41.510Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-09-21T21:19:41.510Z INFO controller-runtime.controller Starting EventSource {"controller": "configconnector-controller", "source": "kind source: /, Kind="}
2020-09-21T21:19:41.611Z INFO controller-runtime.controller Starting EventSource {"controller": "configconnector-controller", "source": "kind source: /, Kind="}
2020-09-21T21:19:41.711Z INFO controller-runtime.controller Starting EventSource {"controller": "configconnector-controller", "source": "channel source: 0xc0001160a0"}
2020-09-21T21:19:41.711Z INFO controller-runtime.controller Starting Controller {"controller": "configconnector-controller"}
2020-09-21T21:19:41.711Z INFO controller-runtime.controller Starting workers {"controller": "configconnector-controller", "worker count": 1}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "dawn-cloud"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "green-fire"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "summer-surf"}
2020-09-21T21:19:41.712Z INFO NameChecker preflight check before reconciling ConfigConnector {"name": "configconnector.core.cnrm.cloud.google.com"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "delicate-sunset"}
2020-09-21T21:19:41.712Z INFO UpgradeChecker preflight check before reconciling ConfigConnector {"name": "configconnector.core.cnrm.cloud.google.com"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "delicate-water"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "shy-haze"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "default"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "still-frost"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "black-wildflower"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "falling-dust"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "misty-surf"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "spring-dawn"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "white-shadow"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "wild-mountain"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "quiet-pine"}
2020-09-21T21:19:41.712Z INFO mapping ConfigConnectorContext request events to ConfigConnector kind {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "long-silence"}
2020-09-21T21:19:41.812Z INFO LocalRepository loading channel {"base": "/configconnector-operator/channels", "name": "stable"}
2020-09-21T21:19:41.813Z INFO UpgradeChecker ConfigConnector {"name": "configconnector.core.cnrm.cloud.google.com", "current version": "1.19.1"}
2020-09-21T21:19:41.813Z INFO UpgradeChecker ConfigConnector {"name": "configconnector.core.cnrm.cloud.google.com", "version to deploy": "1.19.1"}
2020-09-21T21:19:41.813Z INFO UpgradeChecker reconciling ConfigConnector {"name": "configconnector.core.cnrm.cloud.google.com", "version": "1.19.1"}
2020-09-21T21:19:41.813Z INFO reconciling {"object": "/configconnector.core.cnrm.cloud.google.com"}
2020-09-21T21:19:41.813Z INFO ManifestLoader resolving manifest {"name": "configconnector.core.cnrm.cloud.google.com"}
2020-09-21T21:19:41.813Z INFO LocalRepository loading channel {"base": "/configconnector-operator/channels", "name": "stable"}
2020-09-21T21:19:41.813Z INFO ManifestLoader resolved version from channel {"channel": "stable", "version": "1.19.1"}
2020-09-21T21:19:41.815Z INFO LocalRepository loading manifest {"component": "configconnector", "version": "1.19.1", "mode": "namespaced"}
2020-09-21T21:19:41.958Z INFO configconnector-controller removing controller manager components for cluster mode
2020-09-21T21:19:41.977Z INFO configconnector-controller processing ConfigConnectorContext {"name": "configconnectorcontext.core.cnrm.cloud.google.com", "namespace": "wild-mountain"}
There aren't even that many namespaces, we expected to use this cluster for quite a few more.
Is there any suggestion you might have in the interim? Other than removing GKE's addon version of Config Connector and deploying it ourselves?
Thank you for your time.
mrsimo
All 6 comments
Not sure if this bit will be useful, but below certain number of namespaces it seems to not crash entirely, but I can see this in the pod's logs:
2020-09-22T08:14:24.495Z ERROR applying manifest {"error": "error from running kubectl apply: signal: killed"}
cnrm.googlesource.com/configconnector-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/github.com/go-logr/zapr/zapr.go:128
cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/kubebuilder-declarative-pattern/pkg/patterns/declarative.(*Reconciler).reconcileExists
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/kubebuilder-declarative-pattern/pkg/patterns/declarative/reconciler.go:163
cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/kubebuilder-declarative-pattern/pkg/patterns/declarative.(*Reconciler).Reconcile
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/kubebuilder-declarative-pattern/pkg/patterns/declarative/reconciler.go:106
cnrm.googlesource.com/configconnector-operator/pkg/controllers.(*ConfigConnectorReconciler).Reconcile
/go/src/cnrm.googlesource.com/configconnector-operator/pkg/controllers/configconnector_controller.go:305
cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256
cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232
cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211
cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until
/go/src/cnrm.googlesource.com/configconnector-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
2020-09-22T08:14:24.495Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"ConfigConnector","name":"configconnector.core.cnrm.cloud.google.com","uid":"dd4c2d14-1d70-487a-aacc-59838896e57b","apiVersion":"core.cnrm.cloud.google.com/v1beta1","resourceVersion":"220564086"}, "reason": "UpdateFailed", "message": "error during reconciliation: error applying manifest: error from running kubectl apply: signal: killed"}
2020-09-22T08:14:24.503Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "configconnector-controller", "request": "/configconnector.core.cnrm.cloud.google.com"}
Hi @mrsimo , I haven't attempted to reproduce your errors yet, but in the meantime I wanted to let you know that we made some memory optimizations in the operator in version 1.20.0. Unfortunately, that version isn't available on the add-on yet (it usually takes 2-3 weeks to merge into GKE), so you will probably need to manually install the latest version to test it out. Sorry about this inconvenience, and let me know if upgrading your version helps!
caieo
on 23 Sep 2020
Hi @caieo! Sorry I didn't reply earlier. It's not very straightforwad for us to just switch to a manual deployment of Config Connector, so for now we're trying to reduce the amount of namespaces per staging cluster by having more staging clusters. We'll wait until the newer version is available in GKE.
mrsimo
on 28 Sep 2020
I'm getting the same error with 1.29.0, configconnector-operator-0 keeps getting OOM thus CrashLoopBackOff
Bobgy
on 10 Nov 2020
Post an update on this thread.
We have another customer run into the operator scalability issue. The operator pod itself didn't get OOM-killed but the child process kubectl has constantly been killed with the following error message:
Error during reconciliation: error applying manifest: error from running kubectl apply: signal: killed
We have increased the cpu/memory limit of the operator with version 1.38.0 to be able handle more ConfigConnectorContexts/namespaces. At the same time, we are evaluating some long-term approaches to increase scalability of the operator for good. If you have found yourself running into this similar issue, try to increase the cpu/memory limit if you are using the manually-installed operators.
xiaobaitusi
on 6 Feb 2021
I'll close this issue for now, but if it's still happening with 1.38 and higher, please ping me and we can re-open.
There's a larger issue around resource limits (#240) so we can track operator scaling in that ticket.
toumorokoshi
on 19 Apr 2021
Related issues
Sechmann
路
7Comments
RoryShively
路
4Comments
averzicco
路
5Comments
micnncim
路
6Comments
govinda-attal
路
6Comments
Most helpful comment
Hi @mrsimo , I haven't attempted to reproduce your errors yet, but in the meantime I wanted to let you know that we made some memory optimizations in the operator in version 1.20.0. Unfortunately, that version isn't available on the add-on yet (it usually takes 2-3 weeks to merge into GKE), so you will probably need to manually install the latest version to test it out. Sorry about this inconvenience, and let me know if upgrading your version helps!