Dashboard: Couldn't read CA certificate: open : no such file or directory

[dreamlover]

Environment

I installed a single-node kubernetes cluster on CentOS7 using kubeadm according to this manual, then installed the kubernetes-dashboard extension.

Dashboard version:  1.7.1
Kubernetes version: 1.7.5
Operating system: CentOS 7
Node.js version:
Go version: 1.8.3

Observed result

[root@ay pki]# kubectl get pods -n kube-system
NAME                                    READY     STATUS             RESTARTS   AGE
etcd-ay                                 1/1       Running            0          12d
kube-apiserver-ay                       1/1       Running            0          12d
kube-controller-manager-ay              1/1       Running            0          12d
kube-dns-209315428-666w5                3/3       Running            0          12d
kube-proxy-92ss6                        1/1       Running            0          12d
kube-scheduler-ay                       1/1       Running            0          12d
kubernetes-dashboard-1092119393-n9ww6   0/1       CrashLoopBackOff   185        15h
weave-net-wtf68                         2/2       Running            22         1d

[root@ay run]# kubectl logs kubernetes-dashboard-1092119393-n9ww6 -n kube-system
2017/10/25 00:57:16 Using in-cluster config to connect to apiserver
2017/10/25 00:57:16 Starting overwatch
2017/10/25 00:57:16 Using service account token for csrf signing
2017/10/25 00:57:16 No request provided. Skipping authorization
2017/10/25 00:57:16 Successful initial request to the apiserver, version: v1.7.5
2017/10/25 00:57:16 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2017/10/25 00:57:16 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2017/10/25 00:57:16 Initializing secret synchronizer synchronously using secret kubernetes-dashboard-key-holder from namespace kube-system
2017/10/25 00:57:16 Initializing JWE encryption key from synchronized object
2017/10/25 00:57:16 Creating in-cluster Heapster client
2017/10/25 00:57:16 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2017/10/25 00:57:16 Serving securely on HTTPS port: 8443
2017/10/25 00:57:16 Couldn't read CA certificate: open : no such file or directory

[maciaszczykm]

Could you paste output from kubectl describe kubernetes-dashboard -n kube-system and kubectl describe secret kubernetes-dashboard-certs - n kube-system?

[dreamlover]

@maciaszczykm

[root@ay ay.k8s.d]# kubectl describe pod kubernetes-dashboard -n kube-system
Name:       kubernetes-dashboard-1092119393-1np60
Namespace:  kube-system
Node:       ay/10.27.183.194
Start Time: Thu, 26 Oct 2017 08:55:07 +0800
Labels:     k8s-app=kubernetes-dashboard
        pod-template-hash=1092119393
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-1092119393","uid":"4f7270cc-b9e8-11e7-b...
Status:     Running
IP:     10.32.0.4
Created By: ReplicaSet/kubernetes-dashboard-1092119393
Controlled By:  ReplicaSet/kubernetes-dashboard-1092119393
Init Containers:
  kubernetes-dashboard-init:
    Container ID:   docker://4d814445778aa41bd2be53fbfe6bfb256c8f83d957f99c709cc11b57b36fb948
    Image:      registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-init-amd64:v1.0.1
    Image ID:       docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-init-amd64@sha256:fb86ae64a1876a73ee68fa6428c94cbc14c85b549ea3e896f19e81eda00181ba
    Port:       <none>
    State:      Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 26 Oct 2017 08:55:09 +0800
      Finished:     Thu, 26 Oct 2017 08:55:09 +0800
    Ready:      True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-bmvm9 (ro)
Containers:
  kubernetes-dashboard:
    Container ID:   docker://93ee8337723b9e4810c3005302c93779e115164d19aa1264f7307557ae015a37
    Image:      registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1
    Image ID:       docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64@sha256:52b1aeb47e56a97e1278fcdede3dd84703e5e7cef8e0129aa26a73b5f4cadb76
    Port:       8443/TCP
    Args:
      --tls-key-file=/certs/dashboard.key
      --tls-cert-file=/certs/dashboard.crt
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 27 Oct 2017 08:30:33 +0800
      Finished:     Fri, 27 Oct 2017 08:30:33 +0800
    Ready:      False
    Restart Count:  281
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (ro)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-bmvm9 (ro)
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  kubernetes-dashboard-certs:
    Type:   Secret (a volume populated by a Secret)
    SecretName: kubernetes-dashboard-certs
    Optional:   false
  tmp-volume:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  kubernetes-dashboard-token-bmvm9:
    Type:   Secret (a volume populated by a Secret)
    SecretName: kubernetes-dashboard-token-bmvm9
    Optional:   false
QoS Class:  BestEffort
Node-Selectors: <none>
Tolerations:    node-role.kubernetes.io/master:NoSchedule
        node.alpha.kubernetes.io/notReady:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From        SubObjectPath               Type        Reason      Message
  --------- --------    -----   ----        -------------               --------    ------      -------
  23h       1m      282 kubelet, ay spec.containers{kubernetes-dashboard}   Normal      Pulled      Container image "registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1" already present on machine
  23h       1m      282 kubelet, ay spec.containers{kubernetes-dashboard}   Normal      Created     Created container
  23h       1m      282 kubelet, ay spec.containers{kubernetes-dashboard}   Normal      Started     Started container
  23h       7s      6578    kubelet, ay spec.containers{kubernetes-dashboard}   Warning     BackOff     Back-off restarting failed container
  23h       7s      6580    kubelet, ay                     Warning     FailedSync  Error syncing pod

[root@ay ay.k8s.d]# kubectl describe secret kubernetes-dashboard-certs -n kube-system
Name:       kubernetes-dashboard-certs
Namespace:  kube-system
Labels:     k8s-app=kubernetes-dashboard
Annotations:    
Type:       Opaque

Data
====
dashboard.crt:  1123 bytes
dashboard.key:  1704 bytes

[maciaszczykm]

At a first glance, It looks like Dashboard cannot open certs/dashboard.crt file, which should be created on a mounted volume (https://github.com/kubernetes/dashboard/blob/master/src/deploy/recommended/kubernetes-dashboard.yaml#L116), but it seems to be there.

It is just a guess, but it might be some Docker problem (https://docs.docker.com/machine/reference/regenerate-certs/).

[colinlabs]

+1 same issue

Dashboard version: 1.7.1
Kubernetes version: 1.8.1
Operating system: Ubuntu 16.04

[pennpeng]

+1 same issue

Dashboard version: 1.7.1
Kubernetes version: v1.7.9+coreos.0
Operating system: centos 7.3

[floreks]

2017/10/25 00:57:16 Couldn't read CA certificate: open : no such file or directory

This is a bit weird error because it does not come from us. I agree with @maciaszczykm because from what I have checked it might indeed be related to docker. Even the path in this error is missing.

Similar issue: https://github.com/boot2docker/osx-installer/issues/126

[Sidney9217]

+1 same issue

[maciaszczykm]

Did anoyone try solution from https://github.com/kubernetes/dashboard/issues/2518#issuecomment-339891654?

[floreks]

Can all of you paste your docker version? I have found this error in one of our vendored deps:
https://github.com/docker/distribution/blob/b6e0cfbdaa1ddc3a17c95142c7bf6e42c5567370/vendor/github.com/docker/libtrust/key_manager.go#L141

It definitely looks like issue or conflict with docker daemon.

[colinlabs]

@floreks

docker version

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

docker info

Containers: 10
 Running: 10
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Backing Filesystem: extfs
 Dirs: 50
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: iZ2ze1zz9zmbbh6ah52701Z
ID: 2WHB:6Y7P:YLW7:FKDB:2GEK:TU4L:VS6K:ODQE:5JLD:APC3:Q7W4:MYBI
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
Live Restore Enabled: false

WARNING: No swap limit support

[shenkonghui]

今天刚遇到这个问题，折腾了好久才发现是阿里云这个镜像有问题
registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.7.1
换这个镜像
registry.cn-hangzhou.aliyuncs.com/kube_containers/kubernetes-dashboard-amd64:v1.7.1

[Ascendance]

Same issue. Upgraded K8S from 1.7 to 1.8. Biggest mistake ever. Everything broken, dashboard crashloopbackoff, Cassandra crashes and restarts without logs, etc etc

[floreks]

Looks like some core issue or conflict with docker daemon. I'm afraid that we can't fix that directly in Dashboard.

[pingod]

@SILLKY 我改镜像也没用

[maciaszczykm]

Closing as stale.

/close

[k8s-ci-robot]

@maciaszczykm: Closing this issue.

In response to this:

Closing as stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.