Metrics-server: Error: "cluster doesn't provide requestheader-client-ca-file"

Created on 6 Oct 2017  路  18Comments  路  Source: kubernetes-sigs/metrics-server

follow up https://github.com/kubernetes-incubator/metrics-server/issues/18#issuecomment-334164036

picture piosz

All 18 comments

cc @RRAlex @ncdc @deads2k @cheftako @sttts

piosz picture piosz on 6 Oct 2017

My kube-apiserver doesn't have a --requestheader-client-ca-file=<...> set.
Should this basically point to the same CA file as all the options point to the ca.pem?

Here is what I use:

ExecStart=/usr/bin/docker run --rm \
  -v /var/lib/kubernetes:/var/lib/kubernetes \
  -p 6443:6443 \
  -p 8080:8080 \
  --net=host \
  --name kube-apiserver \
  gcr.io/google-containers/kube-apiserver-amd64:v1.8.0 \
   /usr/local/bin/kube-apiserver \
    --admission-control=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds \
    --advertise-address=172.27.19.228 \
    --allow-privileged=true \
    --apiserver-count=3 \
    --audit-log-maxage=30 \
    --audit-log-maxbackup=3 \
    --audit-log-maxsize=100 \
    --audit-log-path=/var/log/audit.log \
    --authorization-mode=Node,RBAC \
    --bind-address=0.0.0.0 \
    --client-ca-file=/var/lib/kubernetes/ca.pem \
    --enable-swagger-ui=true \
    --etcd-cafile=/var/lib/kubernetes/ca.pem \
    --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
    --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
    --etcd-servers=https://172.27.19.228:2379,https://172.27.19.232:2379,https://172.27.19.234:2379 \
    --event-ttl=1h \
    --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yml \
    --insecure-bind-address=127.0.0.1 \
    --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
    --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
    --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
    --kubelet-https=true \
    --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
    --service-account-key-file=/var/lib/kubernetes/ca-key.pem \
    --service-cluster-ip-range=10.0.0.0/24 \
    --service-node-port-range=30000-32767 \
    --tls-ca-file=/var/lib/kubernetes/ca.pem \
    --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
    --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
    --v=2

EDIT: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/#enable-apiserver-flags ?

RRAlex picture RRAlex on 6 Oct 2017

I would agree with the note on the configuration page, which suggests using the different CA than the master. If you reuse the master ca.pem file you are blurring the trust boundaries. This sort of blurring potentially allows for attack avenues which otherwise do not exist. Without a compelling reason to make these trust relationships the same, I would create a separate CA for the aggregation use cases.

cheftako picture cheftako on 6 Oct 2017
👍1

Anything special to be aware of before generating those certificates?
Like in RBAC with nodes, does it need something like an "O": "system:aggregator" field or will any CA+certs do?

RRAlex picture RRAlex on 10 Oct 2017

I would look at the generate-aggregator-certs function https://github.com/kubernetes/kubernetes/blob/master/cluster/common.sh#L1117. Essentially nothing special about the CA cert but the signed cert needs a CN of aggregator.

cheftako picture cheftako on 10 Oct 2017
👍1

Documentation on what certificates mean what can be found at https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md. I would read over that document so that you understand what the different certificates do.

DirectXMan12 picture DirectXMan12 on 16 Oct 2017

So I've created another CA (CN=aggregator) with a single cert I've spread to my 3 kube-apiserver.

I've added this to the kube-apiserver configurations:

   ...
    --requestheader-client-ca-file=/var/lib/kubernetes/aggregator-ca/aggregator-ca.pem \
    --proxy-client-cert-file=/var/lib/kubernetes/aggregator-ca/aggregator-proxy-client.pem \
    --proxy-client-key-file=/var/lib/kubernetes/aggregator-ca/aggregator-proxy-client-key.pem \
    --requestheader-allowed-names=aggregator \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --enable-aggregator-routing=true \
   ...

The metrics server seems to be deploying fine:

Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.248854       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.248908       1 heapster.go:72] Metrics Server version v0.2.0
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.249215       1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version 
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.249237       1 configs.go:62] Using kubelet port 10255
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.250832       1 heapster.go:128] Starting with Metric Sink
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.560455       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.799193       1 heapster.go:101] Starting Heapster API server...
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: [restful] 2017/10/30 14:43:04 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: [restful] 2017/10/30 14:43:04 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
Oct 30 10:43:04 localhost docker/k8s_metrics-server_metrics-server-859cb9bd4b-4lpw7_kube-system_9ec6a903-bd80-11e7-ae4a-fa163e7471c5_0[5227]: I1030 14:43:04.800798       1 serve.go:85] Serving securely on 0.0.0.0:443

But I see these errors in the API's log:

Oct 30 10:42:57 localhost docker/kube-apiserver[944]: E1030 14:42:57.783389       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Oct 30 10:42:57 localhost docker/kube-apiserver[944]: I1030 14:42:57.788226       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (3.694343ms) 409 [[kube-apiserver/v1.8.2 (linux/amd64) kubernetes/bdaeafa] 127.0.0.1:55080]
Oct 30 10:42:57 localhost docker/kube-apiserver[944]: E1030 14:42:57.788874       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again

also:

Oct 30 10:45:07 localhost docker/kube-apiserver[944]: E1030 14:45:07.828129       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.1.192.30:443: i/o timeout'
Oct 30 10:45:07 localhost docker/kube-apiserver[944]: Trying to reach: 'https://10.1.192.30:443/swagger.json', Header: map[]
Oct 30 10:45:07 localhost docker/kube-apiserver[944]: I1030 14:45:07.828148       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

I might have missed others...
Anything obvious I might (still) be doing wrong?
Once working, is something like kubectl top node supposed to output some data?

thanks!

RRAlex picture RRAlex on 30 Oct 2017

Shortly after, kubectl actually stopped responding and I see these lines in the apiserver logs:

Oct 30 11:06:08 localhost docker/kube-apiserver[944]: I1030 15:06:08.749199       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1: (30.000646885s) 503
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: goroutine 78824 [running]:
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42d7e9420, 0x1f7)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42d7e9420, 0x1f7)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: net/http/httputil.(*ReverseProxy).ServeHTTP(0xc42abdf9c0, 0x8129dc0, 0xc42d7e9420, 0xc427d22200)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/usr/local/go/src/net/http/httputil/reverseproxy.go:246 +0x795
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).ServeHTTP(0xc427db6de0, 0x8129dc0, 0xc42d7e9420, 0xc427d22100)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:231 +0x838
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc4241ad180, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:150 +0x690
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4271a6b80, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x567
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4231dfc00, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x38c8c84, 0xf, 0xc428790480, 0xc4231dfc00, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42878aa20, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011<autogenerated>:69 +0x86
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: net/http.HandlerFunc.ServeHTTP(0xc42818b2c0, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/usr/local/go/src/net/http/server.go:1942 +0x44
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: net/http.HandlerFunc.ServeHTTP(0xc428455a40, 0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/usr/local/go/src/net/http/server.go:1942 +0x44
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1(0x8129dc0, 0xc42d7e9420, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: net/http.HandlerFunc.ServeHTTP(0xc428455ac0, 0x7f96c1f4b608, 0xc424f9a1c8, 0xc427d22000)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/usr/local/go/src/net/http/server.go:1942 +0x44
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc428455ae0, 0x812db00, 0xc424f9a1c8, 0xc427d22000, 0xc427db6d80)
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: 
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: logging error output: "Error: 'dial tcp 10.1.192.30:443: i/o timeout'\nTrying to reach: 'https://10.1.192.30:443/apis/metrics.k8s.io/v1beta1'"
Oct 30 11:06:08 localhost docker/kube-apiserver[944]:  [[kubectl/v1.8.1 (linux/amd64) kubernetes/f38e43b] 127.0.0.1:56488]
Oct 30 11:06:08 localhost docker/kube-apiserver[944]: I1030 15:06:08.760024       1 wrap.go:42] GET /api/v1: (1.246987ms) 200 [[kubectl/v1.8.1 (linux/amd64) kubernetes/f38e43b] 127.0.0.1:56488]
RRAlex picture RRAlex on 30 Oct 2017

that looks like you've got cluster network issues. Can your API server talk to service IPs? Some cluster network setups don't support that OOTB.

DirectXMan12 picture DirectXMan12 on 30 Oct 2017

The controller nodes are separated from workers and thus don't have kubelet/kube-proxy running.
They thus don't have access to the overlay network (weave in this case).
So indeed, I don't see how they could reach those pod & svc IP, unless it's possible via kubelet/kube-proxy in some way?

But I thought that was the idea behind the following option:

If you are not running kube-proxy on a host running the API server, 
then you must make sure that the system is enabled with the following apiserver flag:

--enable-aggregator-routing=true

? :)

RRAlex picture RRAlex on 31 Oct 2017

I believe EnableAggregatorRouting just uses pod IPs instead of service IPs:

https://github.com/kubernetes/kubernetes/blob/d94592707701438fece7ab7e49ea9e6a90fe7993/cmd/kube-apiserver/app/options/options.go#L243

You'll still need to have pod IPs accessible. I'd take that up with the Weave folks.

DirectXMan12 picture DirectXMan12 on 31 Oct 2017

Weave, like any overlay networks of any kind, will only be on the worker nodes, not on any of the controller (unless they are wearing both hats!).
It is also my understanding that separating controller from workers isn't an anti-pattern and actually improves security & stability.
Also, Kubernetes the hard way actually proposes this pattern to set up clusters, and a lot of people use this guide in production.

So It seems a lot of people will not be able to use metrics-server if this is in fact the problem?
So, I'm not sure what to do, and someone correct me if i'm getting this wrong, but the metrics server can only work in some configurations of K8s clusters which are, in my sense, less scalable or sub-optimal?

Or I'm missing the big picture...
cheers! :)

RRAlex picture RRAlex on 1 Nov 2017

It's not just metrics-server. No aggregated API server (e.g. service catalog, custom metrics API server, etc) will work without the master being able to contact at least pod IPs. Furthermore, legacy HPA will most likely break as well with that setup as well, because it uses the API server's HTTP proxy.

You may wish to file a bug with upstream Kubernetes, but for the moment, it's a requirement that masters can talk to pod IPs to use aggregated API servers.

It is also my understanding that separating controller from workers isn't an anti-pattern and actually improves security & stability.

A number of things break when you do this. I would not recommend it. In OpenShift we have masters join the SDN by become unschedulable nodes, for similar reasons.

Also, Kubernetes the hard way actually proposes this pattern to set up clusters, and a lot of people use this guide in production.
So It seems a lot of people will not be able to use metrics-server if this is in fact the problem?

If "Kubernetes the Hard Way" is recommending this, it's recommending a setup that breaks several assumptions in current-day Kubernetes.

less scalable

It should not have any effect on scalability.

DirectXMan12 picture DirectXMan12 on 2 Nov 2017

The only other thing I noticed not working right now, is that my nodes/proxy access (say when using kubectl exec -it --namespace=staging <pod_name> bash) is indeed not working right now.

First because my worker nodes are not in the local cloud DNS, but if I set them statically in the API-server container /etc/hosts, I'm getting to what seems like an RBAC issue talking to kubelet.

I'm just mentioning this in case that could be related at all...?

API:

Nov  3 10:24:56 localhost docker/kube-apiserver[944]: I1103 14:24:56.095078       1 rbac.go:116] RBAC DENY: user "kubernetes" groups ["Kubernetes" "system:authenticated"] cannot "create" resource "nodes/proxy" named "staging-worker-004" cluster-wide

kubelet:

Nov  3 10:12:06 staging-worker-004 kubelet[22414]: I1103 10:12:06.608046   22414 server.go:245] Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: I1103 10:12:06.608222   22414 server.go:779] POST /exec/staging/api/api?command=bash&input=1&output=1&tty=1: (430.945碌s) 403
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: goroutine 4587493 [running]:
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc421f4a5b0, 0x193)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc421f4a5b0, 0x193)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc422c540c0, 0x193)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:201 +0x41
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc422c540c0, 0x193, 0xc42167f130, 0x4b, 0x4, 0xc42167f130)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:181 +0x46
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1(0xc4220d5f80, 0xc422c540c0, 0xc422392c30)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:246 +0x4fa
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc422392c30, 0xc4220d5f80, 0xc422c540c0)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19 +0x68
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc420b885a0, 0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:274 +0x991
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: net/http.HandlerFunc.ServeHTTP(0xc420243110, 0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/usr/local/go/src/net/http/server.go:1942 +0x44
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: net/http.(*ServeMux).ServeHTTP(0xc420b562a0, 0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/usr/local/go/src/net/http/server.go:2238 +0x130
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc420b885a0, 0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292 +0x4d
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc420b6e280, 0x51f4160, 0xc421f4a5b0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:778 +0x110
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: net/http.serverHandler.ServeHTTP(0xc420ad6000, 0x51f4a20, 0xc42105b0a0, 0xc422c5b500)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/usr/local/go/src/net/http/server.go:2568 +0x92
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: net/http.(*conn).serve(0xc421a39680, 0x51f6fa0, 0xc4224c4580)
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/usr/local/go/src/net/http/server.go:1825 +0x612
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: created by net/http.(*Server).Serve
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: #011/usr/local/go/src/net/http/server.go:2668 +0x2ce
Nov  3 10:12:06 staging-worker-004 kubelet[22414]: logging error output: "Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)"
Nov  3 10:12:06 staging-worker-004 kubelet[22414]:  [[kubectl/v1.8.2 (linux/amd64) kubernetes/bdaeafa] 172.27.39.86:57552]

But the above problem is happening while connecting to kubelet ports, not pods directly, and thus it does not require to be aware of the overlay network. That's what I find really confusing...
I can understand it could require access to some of kubelet ports as above (ie: 4194, 10248, 10250, 10255 (deprecated)), but why would it need to reach the pods? Especially since they don't have to expose anything.

Am I missing the point of metrics-server, or could it be that it serves a dual purpose here?
ie: Could it both gather high level metrics (cadvisor-like) and then also custom ones from pods exposing tracing/profiling ports/API?

I though metrics-server was going to allow me to run things like kubectl top (and eventually HPA) by using metrics from cadvisor built into kubelets, reached through the API that would then go talk to kubelet(s) (To get some non-prometheus formatted equivalent to port:4194)?

I don't mind opening an issue with the core if that's indeed a necessary discussion, and I'm sorry if I have a hard time grasping this architecture, but it seems weird to me as I thought the main use case was to get global cluster/nodes/pods metrics from the servers to be able to start doing HPA and such...

I might need some enlightenment... :-)

RRAlex picture RRAlex on 3 Nov 2017

Comments a little out of order:

Am I missing the point of metrics-server, or could it be that it serves a dual purpose here?
ie: Could it both gather high level metrics (cadvisor-like) and then also custom ones from pods exposing tracing/profiling ports/API?

I though metrics-server was going to allow me to run things like kubectl top (and eventually HPA) by using metrics from cadvisor built into kubelets, reached through the API that would then go talk to kubelet(s) (To get some non-prometheus formatted equivalent to port:4194)?

metrics-server just gathers from the summary API on the node, and stores the gathered information in-memory. The master (the API server) needs to be able to access pods because the API that metrics-server serves is an aggregated API -- when you try to get metrics.k8s.io, the API server doesn't actually serve that directly. Instead it proxies the result to metrics-server, which responds to the request. Since metrics-server is just running as a pod on the cluster, the master needs to be able to reach that pod via it's pod IP. Basically, on the API server, we say "when you get a request for metrics.k8s.io, proxy it to any pods in the service metrics-server" (that's not hard-coded BTW -- there's an instance of the APIService object in the apiregistration.k8s.io group for metrics.k8s.io).

I'm just mentioning this in case that could be related at all...?

That's probably not related, since the error you were getting before was a connect timeout. For that error, I'd make sure your policy is properly reconciled (kubectl policy reconcile-*).

DirectXMan12 picture DirectXMan12 on 3 Nov 2017

Oh ok, thanks a lot for explaining all this, I now understand how, and to what, it needs to connect.
I thought you were saying it was gathering from any pods, not metrics-server, thus my confusion!

So, then...
Conceptually, could the metrics-server's pod be exposed as a port, via a service, on an external IP that could be reached by kube-apiserver, without access to the worker's overlay network?

This would solve a problem for those with controllers who are separated from your workers.

[parenthesis]
... Which, I still think is a great pattern for separation of concerns...
If you don't think so, you might want to raise this issue with @kelseyhightower on his guides which clearly doesn't run kubelet on the controller nodes (and thus no available overlay from the kube-apiserver), see:

[/parenthesis]

So exposing metrics-server and giving that IP to the kube-apiserver is probably not a supported feature right now, but if the above is realistically feasible, and if I "transform" this issue as a /kind feature request, I would say:

It'd be very useful to those with the above architecture, If we had an option to expose a port on metrics-server pods, which would become an alternative way to connect, making an overlay mode like you described and this suggested approach both feasible.

If everything above still stands, would this feature depend of something in K8s-core for which I should open an issue there?
Or should I open a new issue here as a feature request?

Cheers!

EDIT: I've fixed my kubectl exec issue with a clusterrole + /etc/hosts worker aliases.

RRAlex picture RRAlex on 3 Nov 2017

Any change in this regard would probably need to be a change to Kubernetes itself, since the issue lies in the kube-aggregator component, not the metrics-server. I'm going to close this for the moment, feel free to CC me if you create a Kubernetes feature request.

DirectXMan12 picture DirectXMan12 on 6 Nov 2017
👍1

@DirectXMan12
So, I'm reopening this because I've added kubelet/kube-proxy and thus the weave network device to all the nodes (including controllers with the apiservers!).

Any idea what is now going on by Nov 21 17:24:45?

Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.327639       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (5.12395ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.328952       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.332223       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (2.323296ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.339954       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (3.352474ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.421626       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (4.097894ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.423560       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.432152       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (5.594283ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.471791       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (14.25075ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.473203       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.478075       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (2.201772ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.487308       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (7.405118ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.973669       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (23.105986ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: E1121 22:21:07.974848       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.990372       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (10.307827ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.993020       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (1.674686ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: I1121 22:21:11.361404       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: E1121 22:21:11.390341       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: I1121 22:21:11.390397       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: I1121 22:22:11.391155       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: E1121 22:22:11.395587       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: I1121 22:22:11.395604       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: I1121 22:24:11.395958       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: E1121 22:24:11.398517       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: I1121 22:24:11.398543       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:24:45 localhost docker/kube-apiserver[891]: I1121 22:24:45.689134       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (1.802252ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: I1121 22:24:49.146330       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: E1121 22:24:49.148977       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: I1121 22:24:49.149034       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: I1121 22:25:49.149473       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: E1121 22:25:49.153288       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: I1121 22:25:49.153311       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: I1121 22:27:49.153602       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: E1121 22:27:49.157182       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: I1121 22:27:49.157200       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

It seems to be able to talk to the API but then gets:
OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists

kubectl top pod / node still say:
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

cheers :)

RRAlex picture RRAlex on 21 Nov 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

system-dev-formations picture system-dev-formations  路  8Comments
cdenneen picture cdenneen  路  7Comments
gjmzj picture gjmzj  路  7Comments
ajatkj picture ajatkj  路  9Comments
godomainz picture godomainz  路  6Comments