Gardener: Distribute rollout of Gardenlets for shooted seeds

[rfranzke]

How to categorize this issue?

/area auto-scaling
/kind enhancement
/priority normal
/topology seed
/platform all
/exp intermediate

What would you like to be added:
Right now, the shooted seed controller in the gardenlet is deploying further gardenlet instances into all shooted seeds simultaneously. Especially for large landscapes, and in combination with the CachedRuntimeClient feature gate, this may lead to an extraordinary load on the Gardener API server, as multiple gardenlets restart at the ~same time, start WATCHes and sync their caches (which might be very large).
Eventually, this may lead to the Gardener API server being scaled up and we also saw a few instabilities there, though, they can hopefully be tweaked independently and are unrelated to this issue (/cc @ggaurav10 @amshuman-kr):

the server has received too many requests and has asked us to try again later (get backupentries.core.gardener.cloud shoot--foo--bar--fc5d98ec-17d4-4282-98e7-3df98de42cc4)

See also, the logs of the seed's kube-apiserver:

I0807 19:24:25.896791       1 trace.go:116] Trace[47731262]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.311089747 +0000 UTC m=+18173.463996177) (total time: 585.661655ms):
Trace[47731262]: [373.104993ms] [373.096345ms] Listing from storage done
Trace[47731262]: [585.658915ms] [212.553922ms] Writing http response done count:937
I0807 19:24:25.905080       1 trace.go:116] Trace[685463327]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.311446082 +0000 UTC m=+18173.464352443) (total time: 593.595835ms):
Trace[685463327]: [372.52188ms] [372.512824ms] Listing from storage done
Trace[685463327]: [593.593663ms] [221.071783ms] Writing http response done count:937
I0807 19:24:25.914522       1 trace.go:116] Trace[1398900000]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.309871141 +0000 UTC m=+18173.462777642) (total time: 604.612888ms):
Trace[1398900000]: [381.012212ms] [380.996992ms] Listing from storage done
Trace[1398900000]: [604.6105ms] [223.598288ms] Writing http response done count:937
I0807 19:24:25.971456       1 trace.go:116] Trace[725568575]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.309131689 +0000 UTC m=+18173.462038054) (total time: 662.295353ms):
Trace[725568575]: [394.881731ms] [394.871316ms] Listing from storage done
Trace[725568575]: [662.293721ms] [267.41199ms] Writing http response done count:937
I0807 19:24:25.974179       1 trace.go:116] Trace[1564344665]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.310907358 +0000 UTC m=+18173.463813722) (total time: 663.176563ms):
Trace[1564344665]: [380.916076ms] [380.906309ms] Listing from storage done
Trace[1564344665]: [663.174562ms] [282.258486ms] Writing http response done count:937
I0807 19:24:25.976885       1 trace.go:116] Trace[1702660395]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.30891259 +0000 UTC m=+18173.461818942) (total time: 667.936961ms):
Trace[1702660395]: [389.8659ms] [389.844421ms] Listing from storage done
Trace[1702660395]: [667.935197ms] [278.069297ms] Writing http response done count:937
I0807 19:24:25.979131       1 trace.go:116] Trace[1161997248]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.310517268 +0000 UTC m=+18173.463423642) (total time: 668.555233ms):
Trace[1161997248]: [381.554749ms] [381.538591ms] Listing from storage done
Trace[1161997248]: [668.553202ms] [286.998453ms] Writing http response done count:937
I0807 19:24:25.990076       1 trace.go:116] Trace[473982311]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:25.308437984 +0000 UTC m=+18173.461344344) (total time: 681.601746ms):
Trace[473982311]: [390.369084ms] [390.35783ms] Listing from storage done
Trace[473982311]: [681.600123ms] [291.231039ms] Writing http response done count:937
E0807 19:24:32.245720       1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started)
goroutine 57246120 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x4b714e0, 0xc00029c260)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc01efcdc90, 0x1, 0x1)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x4b714e0, 0xc00029c260)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc0a7e7fd60, 0xc00c18a000)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1cf
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00412a7e0, 0x6410820, 0xc04e1f4cb0, 0xc07b016900)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x310
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016800)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0x10f
net/http.HandlerFunc.ServeHTTP(0xc01f291ce0, 0x6410820, 0xc04e1f4cb0, 0xc07b016800)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc01f291d70, 0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc00412a800, 0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2ca
net/http.HandlerFunc.ServeHTTP(0xc00412a820, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc00412a840, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc01f291e00, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc0183f5ea0, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x6425d60, 0xc068194a50, 0xc0681c1880, 0xc0183f5ea0, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:3366 +0x8d
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc048d7c780, 0xc095491560, 0xc08c6f9a00, 0xc0a584c9a0)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2149 +0x9f
created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1883 +0x4eb
E0807 19:24:32.245767       1 wrap.go:39] apiserver panic'd on GET /api/v1/secrets?limit=500&resourceVersion=0
I0807 19:24:32.245852       1 log.go:172] http2: panic serving 10.44.9.192:60530: killing connection/stream because serving request timed out and response had been started
goroutine 57246120 [running]:
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler.func1(0xc095491560, 0xc01efcdf67, 0xc048d7c780)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2142 +0x16b
panic(0x4b714e0, 0xc00029c260)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc01efcdc90, 0x1, 0x1)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x4b714e0, 0xc00029c260)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc0a7e7fd60, 0xc00c18a000)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1cf
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00412a7e0, 0x6410820, 0xc04e1f4cb0, 0xc07b016900)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x310
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016800)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0x10f
net/http.HandlerFunc.ServeHTTP(0xc01f291ce0, 0x6410820, 0xc04e1f4cb0, 0xc07b016800)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc01f291d70, 0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc00412a800, 0x6410820, 0xc04e1f4cb0, 0xc07b016700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2ca
net/http.HandlerFunc.ServeHTTP(0xc00412a820, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc00412a840, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc01f291e00, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc0183f5ea0, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x6425d60, 0xc068194a50, 0xc0681c1880, 0xc0183f5ea0, 0x6403f60, 0xc095491560, 0xc08c6f9a00)
    /usr/local/go/src/net/http/server.go:3366 +0x8d
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc048d7c780, 0xc095491560, 0xc08c6f9a00, 0xc0a584c9a0)
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2149 +0x9f
created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
    /workspace/anago-v1.16.12-rc.1.1+29e53413f6c4f5/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1883 +0x4eb
E0807 19:24:32.246623       1 writers.go:105] apiserver was unable to write a JSON response: http: Handler timeout
E0807 19:24:32.246653       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}
E0807 19:24:32.247805       1 writers.go:118] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0807 19:24:32.249026       1 trace.go:116] Trace[2123614904]: "List" url:/api/v1/secrets (started: 2020-08-07 19:23:32.246110857 +0000 UTC m=+18120.399017237) (total time: 1m0.002870486s):
Trace[2123614904]: [1m0.002868031s] [59.99176228s] Writing http response done count:10709
I0807 19:24:37.285063       1 trace.go:116] Trace[1545440572]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:36.698530121 +0000 UTC m=+18184.851436534) (total time: 586.501922ms):
Trace[1545440572]: [484.381379ms] [397.289921ms] Transaction prepared
I0807 19:24:37.298143       1 trace.go:116] Trace[1434900149]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/i538593-hxtp/status (started: 2020-08-07 19:24:36.508542423 +0000 UTC m=+18184.661448787) (total time: 789.540798ms):
Trace[1434900149]: [186.67329ms] [108.721836ms] Conversion done
Trace[1434900149]: [776.558688ms] [589.873104ms] Object stored in database
I0807 19:24:37.300411       1 trace.go:116] Trace[1230403920]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/i531571-hxtp/status (started: 2020-08-07 19:24:36.574483704 +0000 UTC m=+18184.727390090) (total time: 725.8843ms):
Trace[1230403920]: [224.208805ms] [125.12357ms] Conversion done
Trace[1230403920]: [712.181117ms] [487.962131ms] Object stored in database
I0807 19:24:45.704057       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0807 19:24:45.863797       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 429, Body: Too many requests, please try again later.
, Header: map[Content-Length:[43] Content-Type:[text/plain; charset=utf-8] Date:[Fri, 07 Aug 2020 19:24:45 GMT] Retry-After:[1] X-Content-Type-Options:[nosniff]]
I0807 19:24:45.863829       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0807 19:24:50.312875       1 trace.go:116] Trace[1017677750]: "Get" url:/apis/machine.sapcloud.io/v1alpha1/namespaces/shoot--foo--bar/machines/shoot--foo--bar-edge-z1-78f6f5c57d-mlzqf (started: 2020-08-07 19:24:49.787302898 +0000 UTC m=+18197.940209265) (total time: 525.539508ms):
Trace[1017677750]: [525.538307ms] [421.223934ms] Transformed response object
I0807 19:24:50.313027       1 trace.go:116] Trace[386557678]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/controlplanes/k534798-hdl/status (started: 2020-08-07 19:24:49.795049726 +0000 UTC m=+18197.947956101) (total time: 517.727846ms):
Trace[386557678]: [517.415761ms] [417.701777ms] Object stored in database
I0807 19:24:50.413846       1 trace.go:116] Trace[78370987]: "Get" url:/api/v1/namespaces/shoot--foo--bar/secrets/gardener (started: 2020-08-07 19:24:49.793916066 +0000 UTC m=+18197.946822432) (total time: 619.891ms):
Trace[78370987]: [284.903284ms] [284.894303ms] About to write a response
Trace[78370987]: [619.889162ms] [334.985878ms] Transformed response object
I0807 19:24:50.427645       1 trace.go:116] Trace[1721646086]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.872001001 +0000 UTC m=+18198.024907369) (total time: 555.579132ms):
Trace[1721646086]: [302.822116ms] [302.812817ms] Listing from storage done
Trace[1721646086]: [555.576118ms] [252.754002ms] Writing http response done count:11
I0807 19:24:50.427856       1 trace.go:116] Trace[1298674710]: "Get" url:/api/v1/namespaces/shoot--foo--bar/secrets/gardener (started: 2020-08-07 19:24:49.893029306 +0000 UTC m=+18198.045935665) (total time: 534.774785ms):
Trace[1298674710]: [187.593738ms] [187.585147ms] About to write a response
Trace[1298674710]: [534.772954ms] [347.179216ms] Transformed response object
I0807 19:24:50.428066       1 trace.go:116] Trace[1029140692]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.873157144 +0000 UTC m=+18198.026063514) (total time: 554.841208ms):
Trace[1029140692]: [200.947222ms] [200.938115ms] Listing from storage done
Trace[1029140692]: [554.839607ms] [353.892385ms] Writing http response done count:11
I0807 19:24:50.430575       1 trace.go:116] Trace[580036880]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.795300851 +0000 UTC m=+18197.948207213) (total time: 635.240763ms):
Trace[580036880]: [318.29267ms] [318.283779ms] Listing from storage done
Trace[580036880]: [635.238833ms] [316.946163ms] Writing http response done count:11
I0807 19:24:50.430760       1 trace.go:116] Trace[1472128221]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.793799137 +0000 UTC m=+18197.946705494) (total time: 636.905784ms):
Trace[1472128221]: [383.517722ms] [383.507843ms] Listing from storage done
Trace[1472128221]: [636.904001ms] [253.386279ms] Writing http response done count:11
I0807 19:24:50.430945       1 trace.go:116] Trace[289967890]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.879617113 +0000 UTC m=+18198.032523472) (total time: 551.223859ms):
Trace[289967890]: [260.699117ms] [260.689915ms] Listing from storage done
Trace[289967890]: [551.222291ms] [290.523174ms] Writing http response done count:5
I0807 19:24:50.430644       1 trace.go:116] Trace[1072806701]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.886611713 +0000 UTC m=+18198.039518093) (total time: 543.99791ms):
Trace[1072806701]: [389.246143ms] [389.237267ms] Listing from storage done
Trace[1072806701]: [543.995977ms] [154.749834ms] Writing http response done count:11
I0807 19:24:50.520800       1 trace.go:116] Trace[1883772546]: "List etcd3" key:/secrets/shoot--foo--bar,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.876677829 +0000 UTC m=+18198.029584195) (total time: 643.149694ms):
Trace[1883772546]: [643.149694ms] [643.149694ms] END
I0807 19:24:50.521068       1 trace.go:116] Trace[708900635]: "List" url:/api/v1/namespaces/shoot--foo--bar/pods (started: 2020-08-07 19:24:49.883073652 +0000 UTC m=+18198.035980027) (total time: 637.955424ms):
Trace[708900635]: [492.583094ms] [492.573875ms] Listing from storage done
Trace[708900635]: [637.953958ms] [145.370864ms] Writing http response done count:11
I0807 19:24:50.584991       1 trace.go:116] Trace[2095985047]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/i341015x-orc/status (started: 2020-08-07 19:24:49.778992305 +0000 UTC m=+18197.931898670) (total time: 805.960151ms):
Trace[2095985047]: [192.047137ms] [192.047137ms] About to convert to expected version
Trace[2095985047]: [738.284654ms] [546.237517ms] Conversion done
I0807 19:24:50.619923       1 trace.go:116] Trace[1207658373]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/clusters/shoot--foo--bar (started: 2020-08-07 19:24:49.973075357 +0000 UTC m=+18198.125981715) (total time: 646.818507ms):
Trace[1207658373]: [600.703001ms] [583.646093ms] Object stored in database
I0807 19:24:50.712969       1 trace.go:116] Trace[669126164]: "List" url:/api/v1/namespaces/shoot--foo--bar-orc/pods (started: 2020-08-07 19:24:50.072928024 +0000 UTC m=+18198.225834392) (total time: 640.00553ms):
Trace[669126164]: [141.272242ms] [141.262574ms] Listing from storage done
Trace[669126164]: [640.002801ms] [498.730559ms] Writing http response done count:11
I0807 19:24:50.713842       1 trace.go:116] Trace[2005772426]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:50.186506008 +0000 UTC m=+18198.339412404) (total time: 527.311786ms):
Trace[2005772426]: [222.577178ms] [210.370074ms] Transaction committed
Trace[2005772426]: [527.311786ms] [304.734608ms] END
I0807 19:24:50.714157       1 trace.go:116] Trace[1203094967]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/controlplanes/tiac9e5f3e-hdl/status (started: 2020-08-07 19:24:50.099580637 +0000 UTC m=+18198.252487011) (total time: 614.556713ms):
Trace[1203094967]: [614.298601ms] [527.490497ms] Object stored in database
I0807 19:24:50.728315       1 trace.go:116] Trace[891846232]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/store1-hxtp/status (started: 2020-08-07 19:24:49.996975404 +0000 UTC m=+18198.149881773) (total time: 731.284577ms):
Trace[891846232]: [208.847108ms] [208.847108ms] About to convert to expected version
Trace[891846232]: [506.870168ms] [298.02306ms] Conversion done
Trace[891846232]: [635.212281ms] [128.332881ms] Object stored in database
I0807 19:24:50.770351       1 trace.go:116] Trace[912174690]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:50.179069951 +0000 UTC m=+18198.331976325) (total time: 591.245167ms):
Trace[912174690]: [591.245167ms] [585.836487ms] END
I0807 19:24:50.770482       1 trace.go:116] Trace[937248174]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/controlplanes/3877f3-haas/status (started: 2020-08-07 19:24:50.125996929 +0000 UTC m=+18198.278903300) (total time: 644.46778ms):
Trace[937248174]: [644.46778ms] [591.722075ms] END
I0807 19:24:50.796456       1 trace.go:116] Trace[1588554393]: "List" url:/api/v1/namespaces/shoot--foo--bar/secrets (started: 2020-08-07 19:24:49.876661781 +0000 UTC m=+18198.029568139) (total time: 919.712984ms):
Trace[1588554393]: [644.189265ms] [644.180084ms] Listing from storage done
Trace[1588554393]: [919.710631ms] [275.521366ms] Writing http response done count:54
I0807 19:24:50.800447       1 trace.go:116] Trace[1398610206]: "List etcd3" key:/secrets/garden,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.194264456 +0000 UTC m=+18198.347170837) (total time: 606.152642ms):
Trace[1398610206]: [606.152642ms] [606.152642ms] END
I0807 19:24:50.815939       1 trace.go:116] Trace[711480816]: "List" url:/api/v1/namespaces/garden/secrets (started: 2020-08-07 19:24:50.193397039 +0000 UTC m=+18198.346303432) (total time: 622.510587ms):
Trace[711480816]: [607.325203ms] [606.469042ms] Listing from storage done
I0807 19:24:51.279644       1 trace.go:116] Trace[421084962]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.596671243 +0000 UTC m=+18197.749577614) (total time: 1.682940481s):
Trace[421084962]: [1.682940481s] [1.682940481s] END
I0807 19:24:51.280494       1 trace.go:116] Trace[1825693581]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.101553209 +0000 UTC m=+18198.254459581) (total time: 1.178914555s):
Trace[1825693581]: [1.178914555s] [1.178914555s] END
I0807 19:24:51.304026       1 trace.go:116] Trace[387398049]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.974730311 +0000 UTC m=+18198.127636689) (total time: 1.329261194s):
Trace[387398049]: [1.329261194s] [1.329261194s] END
I0807 19:24:51.402280       1 trace.go:116] Trace[2125397741]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.995018025 +0000 UTC m=+18198.147924409) (total time: 1.407224267s):
Trace[2125397741]: [1.407224267s] [1.407224267s] END
I0807 19:24:51.419423       1 trace.go:116] Trace[2083130318]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.596834727 +0000 UTC m=+18197.749741096) (total time: 1.822508692s):
Trace[2083130318]: [1.822508692s] [1.822508692s] END
I0807 19:24:51.480331       1 trace.go:116] Trace[1762403465]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.789886583 +0000 UTC m=+18197.942792962) (total time: 1.690406561s):
Trace[1762403465]: [1.690406561s] [1.690406561s] END
I0807 19:24:51.587507       1 trace.go:116] Trace[1268118727]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.880539882 +0000 UTC m=+18198.033446249) (total time: 1.706935655s):
Trace[1268118727]: [1.706935655s] [1.706935655s] END
I0807 19:24:51.595536       1 trace.go:116] Trace[358510776]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.301348758 +0000 UTC m=+18198.454255178) (total time: 1.29415572s):
Trace[358510776]: [1.29415572s] [1.29415572s] END
I0807 19:24:51.674028       1 trace.go:116] Trace[1962934474]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.774689148 +0000 UTC m=+18197.927595536) (total time: 1.899309291s):
Trace[1962934474]: [1.899309291s] [1.899309291s] END
I0807 19:24:51.688856       1 trace.go:116] Trace[1482255018]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.596832113 +0000 UTC m=+18197.749738477) (total time: 2.091993557s):
Trace[1482255018]: [2.091993557s] [2.091993557s] END
I0807 19:24:51.691870       1 trace.go:116] Trace[3345093]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.285856864 +0000 UTC m=+18198.438763233) (total time: 1.405982055s):
Trace[3345093]: [1.405982055s] [1.405982055s] END
I0807 19:24:51.695446       1 trace.go:116] Trace[305492974]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.788903833 +0000 UTC m=+18197.941810205) (total time: 1.890178932s):
Trace[305492974]: [1.890178932s] [1.890178932s] END
I0807 19:24:51.776634       1 trace.go:116] Trace[266183398]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.292719245 +0000 UTC m=+18198.445625617) (total time: 1.483882038s):
Trace[266183398]: [1.483882038s] [1.483882038s] END
I0807 19:24:51.786154       1 trace.go:116] Trace[1911136836]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.78322567 +0000 UTC m=+18197.936132054) (total time: 2.002826472s):
Trace[1911136836]: [2.002826472s] [2.002826472s] END
I0807 19:24:51.788496       1 trace.go:116] Trace[1835524988]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.180637872 +0000 UTC m=+18198.333544258) (total time: 1.60782794s):
Trace[1835524988]: [1.60782794s] [1.60782794s] END
I0807 19:24:51.798750       1 trace.go:116] Trace[1498057545]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.59721048 +0000 UTC m=+18197.750116848) (total time: 2.201510106s):
Trace[1498057545]: [2.201510106s] [2.201510106s] END
I0807 19:24:51.970376       1 trace.go:116] Trace[357061465]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar-hxtp/operatingsystemconfigs (started: 2020-08-07 19:24:51.29246449 +0000 UTC m=+18199.445370852) (total time: 677.874345ms):
Trace[357061465]: [393.468919ms] [393.458947ms] Listing from storage done
Trace[357061465]: [677.871777ms] [284.402858ms] Writing http response done count:34
I0807 19:24:51.973387       1 trace.go:116] Trace[250927194]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.87776767 +0000 UTC m=+18198.030674040) (total time: 2.09559002s):
Trace[250927194]: [2.09559002s] [2.09559002s] END
I0807 19:24:51.987330       1 trace.go:116] Trace[1725347136]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:51.480792454 +0000 UTC m=+18199.633698890) (total time: 506.501041ms):
Trace[1725347136]: [408.121065ms] [383.315494ms] Transaction prepared
I0807 19:24:52.074996       1 trace.go:116] Trace[694776285]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.596646888 +0000 UTC m=+18197.749553277) (total time: 2.47831472s):
Trace[694776285]: [1.687200587s] [1.687184552s] Listing from storage done
Trace[694776285]: [2.478312352s] [791.111765ms] Writing http response done count:937
I0807 19:24:52.079199       1 trace.go:116] Trace[1547630726]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:51.498983816 +0000 UTC m=+18199.651890188) (total time: 580.186261ms):
Trace[1547630726]: [488.182333ms] [487.374109ms] Transaction prepared
I0807 19:24:52.083929       1 trace.go:116] Trace[277756365]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar-hxtp/operatingsystemconfigs (started: 2020-08-07 19:24:51.291546592 +0000 UTC m=+18199.444452983) (total time: 792.342495ms):
Trace[277756365]: [286.475858ms] [286.467731ms] Listing from storage done
Trace[277756365]: [792.3402ms] [505.864342ms] Writing http response done count:34
I0807 19:24:52.086201       1 trace.go:116] Trace[1595318055]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.101527892 +0000 UTC m=+18198.254434253) (total time: 1.984639511s):
Trace[1595318055]: [1.298082178s] [1.298067144s] Listing from storage done
Trace[1595318055]: [1.984638004s] [686.555826ms] Writing http response done count:937
I0807 19:24:52.177635       1 trace.go:116] Trace[78211085]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.171472715 +0000 UTC m=+18198.324379084) (total time: 2.00612094s):
Trace[78211085]: [2.00612094s] [2.00612094s] END
I0807 19:24:52.178953       1 trace.go:116] Trace[2033575372]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:50.074228463 +0000 UTC m=+18198.227134830) (total time: 2.104698778s):
Trace[2033575372]: [2.104698778s] [2.104698778s] END
I0807 19:24:52.189226       1 trace.go:116] Trace[401268375]: "List etcd3" key:/extensions.gardener.cloud/backupentries,resourceVersion:,limit:0,continue: (started: 2020-08-07 19:24:49.798239004 +0000 UTC m=+18197.951145373) (total time: 2.390953142s):
Trace[401268375]: [2.390953142s] [2.390953142s] END
I0807 19:24:52.196878       1 trace.go:116] Trace[1108184684]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-08-07 19:24:51.481573853 +0000 UTC m=+18199.634480273) (total time: 715.27331ms):
Trace[1108184684]: [692.327836ms] [689.163734ms] Transaction prepared
I0807 19:24:52.391106       1 trace.go:116] Trace[465301783]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar-hdl/workers/i335320-h-hdl/status (started: 2020-08-07 19:24:51.484907275 +0000 UTC m=+18199.637813683) (total time: 906.164719ms):
Trace[465301783]: [594.326851ms] [580.477273ms] Object stored in database
Trace[465301783]: [906.164719ms] [311.837868ms] END
I0807 19:24:52.392448       1 trace.go:116] Trace[212481877]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/i063543-hxtp/status (started: 2020-08-07 19:24:51.406852283 +0000 UTC m=+18199.559758641) (total time: 985.566742ms):
Trace[212481877]: [580.511973ms] [507.227068ms] Object stored in database
Trace[212481877]: [985.566742ms] [405.054769ms] END
I0807 19:24:52.398605       1 trace.go:116] Trace[483486805]: "Update" url:/apis/extensions.gardener.cloud/v1alpha1/namespaces/shoot--foo--bar/workers/store1-hxtp/status (started: 2020-08-07 19:24:51.294100205 +0000 UTC m=+18199.447006577) (total time: 1.104460918s):
Trace[483486805]: [186.094201ms] [185.1818ms] Conversion done
Trace[483486805]: [903.072052ms] [716.971971ms] Object stored in database
Trace[483486805]: [1.104460918s] [201.388866ms] END
I0807 19:24:52.592898       1 trace.go:116] Trace[891554121]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.596816803 +0000 UTC m=+18197.749723260) (total time: 2.996045505s):
Trace[891554121]: [2.095679343s] [2.095668764s] Listing from storage done
Trace[891554121]: [2.996043333s] [900.36399ms] Writing http response done count:937
I0807 19:24:52.593138       1 trace.go:116] Trace[1813022330]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.788883833 +0000 UTC m=+18197.941790193) (total time: 2.804204886s):
Trace[1813022330]: [1.982483365s] [1.98247105s] Listing from storage done
Trace[1813022330]: [2.804202962s] [821.719597ms] Writing http response done count:937
I0807 19:24:52.593368       1 trace.go:116] Trace[1954816369]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.974708154 +0000 UTC m=+18198.127614525) (total time: 2.618632703s):
Trace[1954816369]: [1.333865475s] [1.33385272s] Listing from storage done
Trace[1954816369]: [2.618631325s] [1.28476585s] Writing http response done count:937
I0807 19:24:52.594305       1 trace.go:116] Trace[356926739]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.285284631 +0000 UTC m=+18198.438191021) (total time: 2.308992113s):
Trace[356926739]: [1.410010437s] [1.409450293s] Listing from storage done
Trace[356926739]: [2.308990517s] [898.98008ms] Writing http response done count:937
I0807 19:24:52.595278       1 trace.go:116] Trace[622209314]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.880519269 +0000 UTC m=+18198.033425644) (total time: 2.714731351s):
Trace[622209314]: [1.711990008s] [1.711976808s] Listing from storage done
Trace[622209314]: [2.714729876s] [1.002739868s] Writing http response done count:937
I0807 19:24:52.596934       1 trace.go:116] Trace[617503620]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.789867478 +0000 UTC m=+18197.942773844) (total time: 2.807029609s):
Trace[617503620]: [1.695931107s] [1.695918832s] Listing from storage done
Trace[617503620]: [2.807027972s] [1.111096865s] Writing http response done count:937
I0807 19:24:52.671275       1 trace.go:116] Trace[1462246196]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.292696768 +0000 UTC m=+18198.445603138) (total time: 2.378540519s):
Trace[1462246196]: [1.489296377s] [1.489282205s] Listing from storage done
Trace[1462246196]: [2.378538783s] [889.242406ms] Writing http response done count:937
I0807 19:24:52.681594       1 trace.go:116] Trace[1270540189]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.180612386 +0000 UTC m=+18198.333518745) (total time: 2.500944618s):
Trace[1270540189]: [1.611102402s] [1.611087592s] Listing from storage done
Trace[1270540189]: [2.500942928s] [889.840526ms] Writing http response done count:937
I0807 19:24:52.683927       1 trace.go:116] Trace[1839349151]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.596811168 +0000 UTC m=+18197.749717528) (total time: 3.087085277s):
Trace[1839349151]: [1.825825019s] [1.825813074s] Listing from storage done
Trace[1839349151]: [3.087083415s] [1.261258396s] Writing http response done count:937
I0807 19:24:52.685186       1 trace.go:116] Trace[649667573]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.774647282 +0000 UTC m=+18197.927553649) (total time: 2.910506996s):
Trace[649667573]: [1.902859875s] [1.902831373s] Listing from storage done
Trace[649667573]: [2.910505049s] [1.007645174s] Writing http response done count:937
I0807 19:24:52.688074       1 trace.go:116] Trace[76437489]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.877734585 +0000 UTC m=+18198.030640945) (total time: 2.810299623s):
Trace[76437489]: [2.10580234s] [2.105790534s] Listing from storage done
Trace[76437489]: [2.810297818s] [704.495478ms] Writing http response done count:937
I0807 19:24:52.688306       1 trace.go:116] Trace[1803557759]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.994994085 +0000 UTC m=+18198.147900461) (total time: 2.693282482s):
Trace[1803557759]: [1.411471375s] [1.41145767s] Listing from storage done
Trace[1803557759]: [2.693281162s] [1.281809787s] Writing http response done count:937
I0807 19:24:52.690211       1 trace.go:116] Trace[1526825091]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.29343752 +0000 UTC m=+18198.446343889) (total time: 2.396740789s):
Trace[1526825091]: [1.306892199s] [1.298995677s] Listing from storage done
Trace[1526825091]: [2.39673931s] [1.089847111s] Writing http response done count:937
I0807 19:24:52.690378       1 trace.go:116] Trace[190648918]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.597194076 +0000 UTC m=+18197.750100424) (total time: 3.093157339s):
Trace[190648918]: [2.204336462s] [2.204326111s] Listing from storage done
Trace[190648918]: [3.093156309s] [888.819847ms] Writing http response done count:937
I0807 19:24:52.691490       1 trace.go:116] Trace[1626282767]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.783189581 +0000 UTC m=+18197.936095941) (total time: 2.908271263s):
Trace[1626282767]: [2.012540197s] [2.012512282s] Listing from storage done
Trace[1626282767]: [2.908269902s] [895.729705ms] Writing http response done count:937
I0807 19:24:52.700919       1 trace.go:116] Trace[1345945493]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:49.798217634 +0000 UTC m=+18197.951124009) (total time: 2.902666731s):
Trace[1345945493]: [2.394656191s] [2.394642528s] Listing from storage done
Trace[1345945493]: [2.902665001s] [508.00881ms] Writing http response done count:937
I0807 19:24:52.702505       1 trace.go:116] Trace[1260218343]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.171426774 +0000 UTC m=+18198.324333132) (total time: 2.531044283s):
Trace[1260218343]: [2.102928527s] [2.102892683s] Listing from storage done
Trace[1260218343]: [2.53104264s] [428.114113ms] Writing http response done count:937
I0807 19:24:52.702927       1 trace.go:116] Trace[566197244]: "List" url:/apis/extensions.gardener.cloud/v1alpha1/backupentries (started: 2020-08-07 19:24:50.074206106 +0000 UTC m=+18198.227112468) (total time: 2.628677629s):
Trace[566197244]: [2.109857704s] [2.109844329s] Listing from storage done
Trace[566197244]: [2.628675984s] [518.81828ms] Writing http response done count:937
I0807 19:24:52.706172       1 trace.go:116] Trace[630794712]: "List" url:/api/v1/namespaces/shoot--foo--bar/secrets (started: 2020-08-07 19:24:51.796387595 +0000 UTC m=+18199.949293958) (total time: 909.721834ms):
Trace[630794712]: [909.720111ms] [812.459532ms] Writing http response done count:59
I0807 19:24:54.974956       1 trace.go:116] Trace[988874541]: "List" url:/api/v1/namespaces/shoot--foo--bar/secrets (started: 2020-08-07 19:24:54.47428889 +0000 UTC m=+18202.627195248) (total time: 500.626095ms):



429 too many requests

/cc @timebertt

Why is this needed:
Less load spikes to the Gardener API server, more stability of the landscapes, smoother deployments.

[timuthy]

Maybe something like jitterDuration can easily help for enqueuing.

[timebertt]

Today, we hit this issue again while rolling out a different gardenlet configuration and it required manual operator actions to resolve the situation.

To progress on this topic, I would suggest to indeed use some jitter to spread enqueueing the shooted seeds similar to how it is done in the shoot care controller:
https://github.com/gardener/gardener/blob/e03a995c79fcaae59697269ef9473c7d70359234/pkg/gardenlet/controller/shoot/shoot_care_control.go#L58-L63

We could add an additional field in the gardenlet config to specify how long the interval to use should be.
WDYT?

[rfranzke]

/touch
/assign