Metrics-server: 0.3.0/0.3.1 unable to fetch metrics from Kubelet: 401 Unauthorized

[gjmzj]

• k8s version: 1.11.3
• metrics-server: 0.3.1

error log of 'metrics-server' POD:

E0930 02:45:31.619297       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:10.xx.yy.41: unable to fetch metrics from Kubelet 10.xx.yy.41 (10.xx.yy.41): request failed - "401 Unauthorized", response: "Unauthorized"

kubelet is running with flags '--anonymous-auth=false'

my question is 'how can the metrics-server get authorized by the kubelet server '

did i miss something ?

[gjmzj]

through manifests in deploy/1.8+
and also i did a little changes in metrics-server-deployment.yaml :

      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      - name: ssl-dir
        secret:
          secretName: metrics-server-secrets
          defaultMode: 0400
      containers:
      - name: metrics-server
        #image: k8s.gcr.io/metrics-server-amd64:v0.3.0
        image: mirrorgooglecontainers/metrics-server-amd64:v0.3.1
        imagePullPolicy: IfNotPresent
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        - --logtostderr=true
        - --tls-cert-file=/etc/ssl/ms-cert
        - --tls-private-key-file=/etc/ssl/ms-key
        - --v=2
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: ssl-dir
          mountPath: /etc/ssl

i created 'metrics-server-secrets' before deployment

kubectl create secret generic -n kube-system metrics-server-secrets \
            --from-file=ca=ca.pem \
            --from-file=ms-key=metrics-server-key.pem \
            --from-file=ms-cert=metrics-server.pem"

[DirectXMan12]

metrics-server attempts to authorize itself using token authentication. Please ensure that you're running your kubelets with webhook token authentication turned on.

[gjmzj]

thanks, i'll check on it.

[erhudy]

I'm experiencing the same issue, but webhook authentication looks to be on to me:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
  webhook:
    enabled: true
  anonymous:
    enabled: false
authorization:
  mode: Webhook
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
kubeReserved:
  cpu: 100m
  memory: 256Mi
rotateCertificates: true
staticPodPath: /etc/kubernetes/manifests

[gjmzj]

it works on my cluster when i add following settings on kubelet:

  --authentication-token-webhook \
  --authorization-mode=Webhook \

thanks @DirectXMan12

[geoxanadu]

Sorry for the stupid question, how can I access/edit the kubelet? @gjmzj

[DirectXMan12]

you'll need to figure out how your kubelet is being run, unfortunately I can't help much with that without more information.