Project: Creating redis quickstart fails with error "the server is currently unable to handle the request"

Created on 20 Jun 2019 · 3Comments · Source: kubedb/project

Kubernetes: 1.13.1
KubeDB: 0.12.0
StorageOS: 1.2.1

Steps to reproduce:

Install kubedb via helm to namespace kubedb
Install StorageOS as storage class provider via helm
Create the redis resource
kubedb create -f https://raw.githubusercontent.com/kubedb/cli/0.12.0/docs/examples/redis/quickstart/demo-1.yaml

Expected Result:
redis.kubedb.com/redis-quickstart created

Actual Result:
Error from server (InternalError): error when creating "https://raw.githubusercontent.com/kubedb/cli/0.12.0/docs/examples/redis/quickstart/demo-1.yaml": Internal error occurred: failed calling webhook "redis.mutators.kubedb.com": the server is currently unable to handle the request

Additional information:
Observe that the kubedb-operator pod does not appear to display any information related to the creation attempt.

bug

Source

acceleratxr

Most helpful comment

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.98. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 20 Jun 2019

👍4

All 3 comments

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.98. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on 20 Jun 2019

👍4

Looking at the kubedb-operator log further I found this warning. Perhaps related somehow?

W0620 21:10:11.000978       1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.

acceleratxr on 20 Jun 2019

I can confirm the issue is still showing up in version 0.14.0. It only happens with redis. Mysql, mongoDB and Postgres were created without any issues. Here is some debug information:

E1107 12:01:04.093674       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 7574 [running]:
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest.func1.1(0xc00207f080)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:233 +0xd5
panic(0x306d300, 0x5241310)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
kubedb.dev/apimachinery/apis/kubedb/v1alpha2.(*Redis).setDefaultAffinity(0xc000581000, 0xc000581180, 0xc002d0fef0, 0x0)
    /src/vendor/kubedb.dev/apimachinery/apis/kubedb/v1alpha2/redis_helpers.go:246 +0x208
kubedb.dev/apimachinery/apis/kubedb/v1alpha2.(*Redis).SetDefaults(0xc000581000, 0x0)
    /src/vendor/kubedb.dev/apimachinery/apis/kubedb/v1alpha2/redis_helpers.go:195 +0x3f1
kubedb.dev/redis/pkg/admission.setDefaultValues(0xc000581000, 0x0, 0x500, 0x34e748d, 0xa, 0x34e3a03)
    /src/vendor/kubedb.dev/redis/pkg/admission/mutator.go:129 +0xb5
kubedb.dev/redis/pkg/admission.(*RedisMutator).Admit(0xc000a0e690, 0xc0006b0ea0, 0x0)
    /src/vendor/kubedb.dev/redis/pkg/admission/mutator.go:95 +0x1c5
kmodules.xyz/webhook-runtime/registry/admissionreview/v1beta1.(*REST).Create(0xc002f89220, 0x3a4dd20, 0xc002d0fc20, 0x39ec8a0, 0xc002d0fd40, 0x3608210, 0xc0010f1860, 0x0, 0x0, 0x0, ...)
    /src/vendor/kmodules.xyz/webhook-runtime/registry/admissionreview/v1beta1/admission_review.go:61 +0x4c
k8s.io/apiserver/pkg/endpoints/handlers.(*namedCreaterAdapter).Create(0xc0022310f0, 0x3a4dd20, 0xc002d0fc20, 0x0, 0x0, 0x39ec8a0, 0xc002d0fd40, 0x3608210, 0xc0010f1860, 0x3, ...)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:208 +0x7f
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1.1(0x30c0da0, 0x0, 0x0, 0x0)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:145 +0x13b
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1.2(0xc0006219a0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:166 +0xf5
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest.func1(0xc00207f080, 0xc00207eea0, 0xc00207f020, 0xc00207ef60)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:241 +0x62
created by k8s.io/apiserver/pkg/endpoints/handlers.finishRequest
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:222 +0xd4

goroutine 7572 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0015ac1e0)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x113
panic(0x2f0a3c0, 0xc002231290)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest(0x6fc23ac00, 0xc00207eea0, 0x21, 0x0, 0x0, 0x0)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:259 +0x505
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0x3a3e160, 0xc000206800, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:153 +0x180d
k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc002d0faa0, 0xc00032ed90)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1104 +0xe4
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc002d0faa0, 0xc00032ed90)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:323 +0x255
github.com/emicklei/go-restful.(*Container).dispatch(0xc000874a20, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/emicklei/go-restful.(*Container).Dispatch(...)
    /src/vendor/github.com/emicklei/go-restful/container.go:199
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x34e9788, 0xb, 0xc000874a20, 0xc000656d90, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4bc
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x523
net/http.HandlerFunc.ServeHTTP(0xc000623ec0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x4d4
net/http.HandlerFunc.ServeHTTP(0xc00086d320, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x22bd
net/http.HandlerFunc.ServeHTTP(0xc000623f00, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x652
net/http.HandlerFunc.ServeHTTP(0xc0006219a0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0015ac1e0, 0xc000869860, 0x3a4e520, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xb8
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cc

goroutine 7571 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2f0a3c0, 0xc0022312d0)
    /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc000ec5c98, 0x1, 0x1)
    /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x2f0a3c0, 0xc0022312d0)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc000869860, 0x3a3e320, 0xc00032ecb0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:119 +0x448
k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x3a3e320, 0xc00032ecb0, 0xc001388a00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc00086d350, 0x3a3e320, 0xc00032ecb0, 0xc001388a00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
net/http.HandlerFunc.ServeHTTP(0xc00086d380, 0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc000869880, 0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2c9
net/http.HandlerFunc.ServeHTTP(0xc0008698a0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc0008698c0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc00086d3b0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc0026560e0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x3a4dd20, 0xc002674030, 0xc002672e00, 0xc0026560e0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
golang.org/x/net/http2.(*serverConn).runHandler(0xc001bb1800, 0xc0002067e8, 0xc001388800, 0xc002b19380)
    /src/vendor/golang.org/x/net/http2/server.go:2147 +0x8b
created by golang.org/x/net/http2.(*serverConn).processHeaders
    /src/vendor/golang.org/x/net/http2/server.go:1881 +0x505
E1107 12:01:04.093785       1 wrap.go:39] apiserver panic'd on POST /apis/mutators.kubedb.com/v1alpha1/redises?timeout=30s
I1107 12:01:04.093956       1 log.go:181] http2: panic serving 10.244.0.0:49224: runtime error: invalid memory address or nil pointer dereference
goroutine 7574 [running]:
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest.func1.1(0xc00207f080)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:233 +0xd5
panic(0x306d300, 0x5241310)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
kubedb.dev/apimachinery/apis/kubedb/v1alpha2.(*Redis).setDefaultAffinity(0xc000581000, 0xc000581180, 0xc002d0fef0, 0x0)
    /src/vendor/kubedb.dev/apimachinery/apis/kubedb/v1alpha2/redis_helpers.go:246 +0x208
kubedb.dev/apimachinery/apis/kubedb/v1alpha2.(*Redis).SetDefaults(0xc000581000, 0x0)
    /src/vendor/kubedb.dev/apimachinery/apis/kubedb/v1alpha2/redis_helpers.go:195 +0x3f1
kubedb.dev/redis/pkg/admission.setDefaultValues(0xc000581000, 0x0, 0x500, 0x34e748d, 0xa, 0x34e3a03)
    /src/vendor/kubedb.dev/redis/pkg/admission/mutator.go:129 +0xb5
kubedb.dev/redis/pkg/admission.(*RedisMutator).Admit(0xc000a0e690, 0xc0006b0ea0, 0x0)
    /src/vendor/kubedb.dev/redis/pkg/admission/mutator.go:95 +0x1c5
kmodules.xyz/webhook-runtime/registry/admissionreview/v1beta1.(*REST).Create(0xc002f89220, 0x3a4dd20, 0xc002d0fc20, 0x39ec8a0, 0xc002d0fd40, 0x3608210, 0xc0010f1860, 0x0, 0x0, 0x0, ...)
    /src/vendor/kmodules.xyz/webhook-runtime/registry/admissionreview/v1beta1/admission_review.go:61 +0x4c
k8s.io/apiserver/pkg/endpoints/handlers.(*namedCreaterAdapter).Create(0xc0022310f0, 0x3a4dd20, 0xc002d0fc20, 0x0, 0x0, 0x39ec8a0, 0xc002d0fd40, 0x3608210, 0xc0010f1860, 0x3, ...)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:208 +0x7f
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1.1(0x30c0da0, 0x0, 0x0, 0x0)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:145 +0x13b
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1.2(0xc0006219a0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:166 +0xf5
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest.func1(0xc00207f080, 0xc00207eea0, 0xc00207f020, 0xc00207ef60)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:241 +0x62
created by k8s.io/apiserver/pkg/endpoints/handlers.finishRequest
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:222 +0xd4

goroutine 7572 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0015ac1e0)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x113
panic(0x2f0a3c0, 0xc002231290)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apiserver/pkg/endpoints/handlers.finishRequest(0x6fc23ac00, 0xc00207eea0, 0x21, 0x0, 0x0, 0x0)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:259 +0x505
k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0x3a3e160, 0xc000206800, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:153 +0x180d
k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc002d0faa0, 0xc00032ed90)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1104 +0xe4
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc002d0faa0, 0xc00032ed90)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:323 +0x255
github.com/emicklei/go-restful.(*Container).dispatch(0xc000874a20, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/emicklei/go-restful.(*Container).Dispatch(...)
    /src/vendor/github.com/emicklei/go-restful/container.go:199
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x34e9788, 0xb, 0xc000874a20, 0xc000656d90, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4bc
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x523
net/http.HandlerFunc.ServeHTTP(0xc000623ec0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x4d4
net/http.HandlerFunc.ServeHTTP(0xc00086d320, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x22bd
net/http.HandlerFunc.ServeHTTP(0xc000623f00, 0x7fa058947cc8, 0xc0002067f0, 0xc001388c00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x652
net/http.HandlerFunc.ServeHTTP(0xc0006219a0, 0x7fa058947cc8, 0xc0002067f0, 0xc001388b00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0015ac1e0, 0xc000869860, 0x3a4e520, 0xc0002067f0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xb8
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cc

goroutine 7571 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1(0xc0002067e8, 0xc000ec5f8e, 0xc001bb1800)
    /src/vendor/golang.org/x/net/http2/server.go:2140 +0x16f
panic(0x2f0a3c0, 0xc0022312d0)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc000ec5c98, 0x1, 0x1)
    /src/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x2f0a3c0, 0xc0022312d0)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc000869860, 0x3a3e320, 0xc00032ecb0, 0xc001388b00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:119 +0x448
k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x3a3e320, 0xc00032ecb0, 0xc001388a00)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc00086d350, 0x3a3e320, 0xc00032ecb0, 0xc001388a00)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
net/http.HandlerFunc.ServeHTTP(0xc00086d380, 0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /src/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc000869880, 0x3a3e320, 0xc00032ecb0, 0xc001388900)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2c9
net/http.HandlerFunc.ServeHTTP(0xc0008698a0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc0008698c0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc00086d3b0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /src/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc0026560e0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x3a4dd20, 0xc002674030, 0xc002672e00, 0xc0026560e0, 0x3a313e0, 0xc0002067e8, 0xc001388800)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
golang.org/x/net/http2.(*serverConn).runHandler(0xc001bb1800, 0xc0002067e8, 0xc001388800, 0xc002b19380)
    /src/vendor/golang.org/x/net/http2/server.go:2147 +0x8b
created by golang.org/x/net/http2.(*serverConn).processHeaders
    /src/vendor/golang.org/x/net/http2/server.go:1881 +0x505