metrics-server request canceled while waiting for connection

Created on 20 Sep 2018  路  15Comments  路  Source: kubernetes-sigs/metrics-server

I have deployed a metrics-server in kubernetes following https://github.com/kubernetes-incubator/metrics-server/tree/v0.2.1/deploy, and my metrics-server image is from gcr.io/google_containers/metrics-server-amd64:v0.2.1.
Metric server is running fine and but I am not able to get metrics from it. And I'm using Kubernetes version is 1.11.0.
The problem I met as followings:

[root@k8s-master metrics-server]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: 2018-09-12T09:29:13Z
  name: v1beta1.metrics.k8s.io
  resourceVersion: "828496"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 4fc0035b-b66e-11e8-aa6e-000c29985c64
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2018-09-12T09:29:13Z
    message: 'no response from https://169.169.50.19:443: Get https://169.169.50.19:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

And I use curl https://169.169.50.19:443, but nothing return.

metircs-server logs:

I0920 06:40:56.625243       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet_client.crt
I0920 06:40:56.625285       1 heapster.go:72] Metrics Server version v0.2.1
I0920 06:40:56.625439       1 configs.go:61] Using Kubernetes client with master "https://169.169.0.1:443" and version
I0920 06:40:56.625447       1 configs.go:62] Using kubelet port 10250
I0920 06:40:56.626126       1 heapster.go:128] Starting with Metric Sink
I0920 06:40:56.988875       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0920 06:40:57.446216       1 heapster.go:101] Starting Heapster API server...
[restful] 2018/09/20 06:40:57 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/09/20 06:40:57 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0920 06:40:57.448570       1 serve.go:85] Serving securely on 0.0.0.0:443

Kubernetes master:
apiserver:

ETCD_SERVER="--etcd-servers=http://127.0.0.1:2379"
INSECURE_BIND_ADDRESS="--insecure-bind-address=0.0.0.0"
INSECURE_PORT="--insecure-port=8080"
SECURE_PORT="--secure-port=6443"
SERVICE_CLUSTER_IP_RANGE="--service-cluster-ip-range=169.169.0.0/16"
SERVICE_NODE_PORT_RANGE="--service-node-port-range=1-65535"
ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
CLIENT_CA_FILE="--client-ca-file=/etc/kubernetes/ssl/ca.crt"
TLS_PRIVATE_KEY="--tls-private-key-file=/etc/kubernetes/ssl/server.key"
TLS_CERT_FILE="--tls-cert-file=/etc/kubernetes/ssl/server.crt"
SERVICE_ACCOUNT_KEY_FILE="--service-account-key-file=/etc/kubernetes/ssl/server.key"
REQUESTHEADER_CLIENT_CA_FILE="--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.crt"
REQUESTHEADER_ALLOWED_NAMES="--requestheader-allowed-names=aggregator"
REQUESTHEADER_EXTRA_HEADERS_PREFIX="--requestheader-extra-headers-prefix=X-Remote-Extra-"
REQUESTHEADER_GROUP_HEADERS="--requestheader-group-headers=X-Remote-Group"
REQUESTHEADER_USERNAME_HEADERS="--requestheader-username-headers=X-Remote-User"
KUBE_API_ARGS="--enable-swagger-ui=true"

controller-manager:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
SERVICE_ACCOUNT_KEY_FILE="--service-account-private-key-file=/etc/kubernetes/ssl/server.key"
CLUSTER_SIGNING_CERT_FILE=""
CLUSTER_SIGNING_KEY_FILE=""
ROOT_CA_FILE="--root-ca-file=/etc/kubernetes/ssl/ca.crt"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_CONTROLLER_MANAGER_ARGS='--cloud-provider=""'

kubeconfig:

apiVersion: v1
kind: Config
users:
- name: controllermanager
  user:
    client-certificate: /etc/kubernetes/ssl/cs_client.crt
    client-key: /etc/kubernetes/ssl/cs_client.key
  clusters:
  - name: local
    cluster:
      certificate-authority: /etc/kubernetes/ssl/ca.crt
contexts:
- context:
    cluster: local
    user: controllermanager
  name: my-context
current-context: my-context

proxy:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_PROXY_ARGS=""

Kubernetes node:
kubelet:

ADDRESS="--address=10.0.83.63"
HOSTNAME_OVERRIDE="--hostname-override=10.0.83.63"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=5"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
PAUSE_IMAGE="--pod_infra_container_image=kubernetes/pause"
KUBELET_ARGS='--cloud-provider=""'

proxy:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_PROXY_ARGS=""

kubeconfig:

apiVersion: v1
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/kubelet_client.crt
    client-key: /etc/kubernetes/ssl/kubelet_client.key
clusters:
- cluster:
    certificate-authority: ssl/ca.crt
    server: https://k8s-master:6443
  name: local
contexts:
- context:
    cluster: local
    user: kubelet
  name: my-context
current-context: my-context
kind: Config
preferences: {}

And Metrics-server YAML metrics-server-deployment.yaml l I just modified

apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/kubernetes/ssl
          name: ca-ssl
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250
        - --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet_client.crt
      volumes:
       - name: ca-ssl
         hostPath:
          path: /etc/kubernetes/ssl

So, is there any configuration fault? Anyone has suggestion to solve this problem?

picture f4ct0r

Most helpful comment

@zonyes

I ran into the same problem and have been trying different things to get metrics-server working on EKS with weave.

And all the comments above are valid.

The only things you need to do is ( after following all the standard procedures mentioned above, I used helm chart to install metrics-server without much customization):

kubectl -n metrics-server edit deployment YOUR_METRICS_SERVER_DEPLOYMENT

Add hostNetwork:true under spec:template:spec (the same level with containers)

Save it and then, it all works like a charm....

picture allansun on 8 Apr 2019
👍12 ❤1

All 15 comments

Are you running some overlay network on your cluster that prevents the main API server from talking to the the service proxies or pods?

DirectXMan12 picture DirectXMan12 on 24 Sep 2018

@DirectXMan12 Thanks for your reply. overlay network? you mean CNI or other network plugin?

f4ct0r picture f4ct0r on 25 Sep 2018

yeah, some CNI plugins create isolation between different parts of the system. Sometimes this isolation breaks Kubernetes features

DirectXMan12 picture DirectXMan12 on 25 Sep 2018

I have not installed any network plugins before, so master cannot access pods on other nodes?
When I installed CNI plugin sucessfully, then this problem can be solved, right?
Could you please give me an User Guide to install CNI?

f4ct0r picture f4ct0r on 26 Sep 2018

You'll have to check the kubernetes documentation -- there's a guide here: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/

DirectXMan12 picture DirectXMan12 on 27 Sep 2018

I have installed CNI plugin, but metrics-server pod cannot be created, and error return

Name:           metrics-server-68c7fb9fd4-jtv6h
Namespace:      kube-system
Node:           worker-node/10.0.83.63
Start Time:     Sun, 30 Sep 2018 17:10:32 +0800
Labels:         k8s-app=metrics-server
                pod-template-hash=2473965980
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicaSet/metrics-server-68c7fb9fd4
Containers:
  metrics-server:
    Container ID:
    Image:         docker.tophant.com/google_containers/metrics-server-amd64:v0.2.1
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /metrics-server
      --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250
      --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet.pem
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/kubernetes/ssl from ca-ssl (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-wlzn9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ca-ssl:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/ssl
    HostPathType:
  metrics-server-token-wlzn9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-wlzn9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                  Age                From                  Message
  ----     ------                  ----               ----                  -------
  Warning  FailedCreatePodSandBox  4m (x141 over 1h)  kubelet, worker-node  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d3eee93c3d626d7a37286f7568a0bee6e03ab2684b788ed7ffd2d38c95660a28" network for pod "metrics-server-68c7fb9fd4-jtv6h": NetworkPlugin cni failed to set up pod "metrics-server-68c7fb9fd4-jtv6h_kube-system" network: Get https://[10.1.0.1]:443/api/v1/namespaces/kube-system/pods/metrics-server-68c7fb9fd4-jtv6h: dial tcp 10.1.0.1:443: i/o timeout

It said NetworkPlugin cni failed to set up pod, but I don't know why
And all the pod cannot be create because of this error

f4ct0r picture f4ct0r on 8 Oct 2018

This is a major problem in the whole master -> metrics server design. Masters are not supposed to be on the overlay network for security reasons.
With the old InfluxDB setup I was able to run InfluxDB on the host network, same as the master, but this new Metric Server does not support that setup.

As an enhancement I think the Metric Server should be able to run host network.

mingfang picture mingfang on 17 Oct 2018
👍2

you could run metrics-server in hostnetwork mode. It's not a metrics-server issue -- it's in the core design of the Kubernetes aggregation layer. If you have concerns with this model, please discuss in the main kubernetes repo with the SIG APIMachinery team.

DirectXMan12 picture DirectXMan12 on 8 Nov 2018

I run metrics-server successfully, thanks for these advises :)

f4ct0r picture f4ct0r on 22 Nov 2018

i want to know you how to fix? metrics-server in hostnetwork mode? how to setting? @f4ct0r

zonyes picture zonyes on 2 Apr 2019

@zonyes

I ran into the same problem and have been trying different things to get metrics-server working on EKS with weave.

And all the comments above are valid.

The only things you need to do is ( after following all the standard procedures mentioned above, I used helm chart to install metrics-server without much customization):

kubectl -n metrics-server edit deployment YOUR_METRICS_SERVER_DEPLOYMENT

Add hostNetwork:true under spec:template:spec (the same level with containers)

Save it and then, it all works like a charm....

allansun picture allansun on 8 Apr 2019
👍12 ❤1

The previous answer fixes this issue for me. I've encountered the same issue after upgrade cluster to 1.11.x using kops.
You can just change this line in values of chart:
https://github.com/helm/charts/blob/master/stable/metrics-server/values.yaml#L27

kuzm1ch picture kuzm1ch on 9 Apr 2019

@zonyes

I ran into the same problem and have been trying different things to get metrics-server working on EKS with weave.

And all the comments above are valid.

The only things you need to do is ( after following all the standard procedures mentioned above, I used helm chart to install metrics-server without much customization):

kubectl -n metrics-server edit deployment YOUR_METRICS_SERVER_DEPLOYMENT

Add hostNetwork:true under spec:template:spec (the same level with containers)

Save it and then, it all works like a charm....

Hi @allansun ,
even after enabling the host network to true. it didn't work. i'm using flannel as overlay network. and i'm facing this issue.
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: app=metrics-server
chart=metrics-server-2.8.2
heritage=Tiller
release=metrics-server
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2019-06-11T11:49:33Z
Resource Version: 1681
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
UID: fad12538-8c3e-11e9-a91a-005056a5efb4
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2019-06-11T11:49:33Z
Message: no response from https://10.233.47.243:443: Get https://10.233.47.243:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:

event i'm not able to curl https://10.233.47.243:443. it's not giving any response.
also kube-proxy and kube-flannel pods are running fine.
Please suggest me any possible solution. I'm struck with this.
Please let me know if you require any other info.

RamanPndy picture RamanPndy on 11 Jun 2019

I ran into the same problem on AWS EKS with Secondary Cidr extended.

$ kubectl get apiservice v1beta1.metrics.k8s.io -o yaml                                                                                                                                                                            [15:53:39]
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: "2019-11-20T07:34:35Z"
  labels:
    app: metrics-server
    chart: metrics-server-2.8.8
    heritage: Helm
    release: metrics-server
  name: v1beta1.metrics.k8s.io
  resourceVersion: "1947084"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 33541d8e-0b68-11ea-9755-065377ff5cdc
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2019-11-20T07:34:35Z"
    message: 'no response from https://100.64.54.127:8443: Get https://100.64.54.127:8443:
      Address is not allowed'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Because I enable Custom Network Config for the VPC CNI.
I follow @allansun to add hostNetwork: true, the issue was resolved.

RaymondKYLiu picture RaymondKYLiu on 20 Nov 2019

@f4ct0r I had the same problem, and I struggled for it.

2018-11-27 picture 2018-11-27 on 24 Dec 2019
😄1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Sh4d1 picture Sh4d1  路  6Comments
cdenneen picture cdenneen  路  7Comments
serathius picture serathius  路  9Comments
floryut picture floryut  路  8Comments
CheyiLin picture CheyiLin  路  3Comments