Rke: Failed to apply the ServiceAccount needed for job execution

[nii236]

RKE version:
Master branch
Docker version: (docker version,docker info preferred)

Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 5
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: centos-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 2.168 GB
 Data Space Total: 4.48 GB
 Data Space Available: 2.312 GB
 Metadata Space Used: 495.6 kB
 Metadata Space Total: 33.55 MB
 Metadata Space Available: 33.06 MB
 Thin Pool Minimum Free Space: 447.7 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host overlay null
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.693 GiB
Name: centos-base
ID: 4BHK:NGSY:Z3TH:7PZ3:APTG:XPJ7:Y4G7:GFDO:HEQU:3LCA:KXRI:U6DN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)



md5-b57ebcebb885207de8661d07ff70e76c



NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"



md5-3ca6a09c38c997aaef039959d2c8217d



3.10.0-693.11.1.el7.x86_64



md5-e846e3c251e0b85c50b83ae362261b55



# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- machine_name: ""
  address: 10.254.26.101
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: node1
  user: raijin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: /home/raijin/.ssh/id_rsa
- machine_name: ""
  address: 10.254.26.102
  internal_address: ""
  role:
  - worker
  - etcd
  hostname_override: node2
  user: raijin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: /home/raijin/.ssh/id_rsa
- machine_name: ""
  address: 10.254.26.103
  internal_address: ""
  role:
  - worker
  - etcd
  hostname_override: node3
  user: raijin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: /home/raijin/.ssh/id_rsa
services:
  etcd:
    image: quay.io/coreos/etcd:latest
    extra_args: {}
  kube-api:
    image: rancher/k8s:v1.8.3-rancher2
    extra_args: {}
    service_cluster_ip_range: 10.233.0.0/18
  kube-controller:
    image: rancher/k8s:v1.8.3-rancher2
    extra_args: {}
    cluster_cidr: 10.233.64.0/18
    service_cluster_ip_range: 10.233.0.0/18
  scheduler:
    image: rancher/k8s:v1.8.3-rancher2
    extra_args: {}
  kubelet:
    image: rancher/k8s:v1.8.3-rancher2
    extra_args: {}
    cluster_domain: raijin.local
    infra_container_image: gcr.io/google_containers/pause-amd64:3.0
    cluster_dns_server: 10.233.0.3
  kubeproxy:
    image: rancher/k8s:v1.8.3-rancher2
    extra_args: {}
network:
  plugin: flannel
  options: {}
authentication:
  strategy: x509
  options: {}
addons: ""
system_images: {}
ssh_key_path: ~/.ssh/id_rsa
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false



md5-887799ffc286d02bfbb1278147d1a732



rke up --config cluster.yml



md5-90d6cca9fa08ca452b48e95e119332bb



...
INFO[0027] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [10.254.26.101]
INFO[0027] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [10.254.26.101]
INFO[0028] [controlplane] Successfully started [kube-api] container on host [10.254.26.101]
INFO[0028] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [10.254.26.101]
INFO[0028] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [10.254.26.101]
INFO[0029] [controlplane] Successfully started [kube-controller] container on host [10.254.26.101]
INFO[0029] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [10.254.26.101]
INFO[0029] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [10.254.26.101]
INFO[0030] [controlplane] Successfully started [scheduler] container on host [10.254.26.101]
INFO[0030] [controlplane] Successfully started Controller Plane..
INFO[0030] [authz] Creating rke-job-deployer ServiceAccount
FATA[0060] [auths] Failed to apply RBAC resources: Failed to apply the ServiceAccount needed for job execution: Post https://10.254.26.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: x509:
 certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")

[nii236]

Is there a way to specify a .crt and .key file for SSL?

I also tried changing to a hostname instead of an IP address and got the following:

INFO[0028] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [kube1.theninja.life]
INFO[0028] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [kube1.theninja.life]
INFO[0028] [controlplane] Successfully started [kube-api] container on host [kube1.theninja.life]
INFO[0028] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [kube1.theninja.life]
INFO[0028] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [kube1.theninja.life]
INFO[0029] [controlplane] Successfully started [kube-controller] container on host [kube1.theninja.life]
INFO[0029] [controlplane] Checking image [rancher/k8s:v1.8.3-rancher2] on host [kube1.theninja.life]
INFO[0029] [controlplane] No pull necessary, image [rancher/k8s:v1.8.3-rancher2] exists on host [kube1.theninja.life]
INFO[0030] [controlplane] Successfully started [scheduler] container on host [kube1.theninja.life]
INFO[0030] [controlplane] Successfully started Controller Plane..
INFO[0030] [authz] Creating rke-job-deployer ServiceAccount
FATA[0060] [auths] Failed to apply RBAC resources: Failed to apply the ServiceAccount needed for job execution: Post https://kube1.theninja.life:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: x509: certificate is valid for localhost, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not kube1.theninja.life

[galal-hussein]

@nii236 The error may indicates that you are trying reuse the hosts without cleaning the certificates on the hosts, currently we generate the certificates and redeploy them on the hosts, can you try using:

rke remove

to clean the cluster components and remove the previously deployed certificates, and then try to re-deploy again

[nii236]

Cool, thanks! I have been executing remove every time before up, but the problem is still there.

I suspect that something may have happened at some point where I updated from 0.0.8-dev to master.

Anyway I will reset my VM and start from scratch, it should be fine now.

[kdealer]

Whats the state of this bug?

[nii236]

Hey @kdealer unfortunately I can't remember, but I did get RKE working eventually, so I probably did a poor job cleaning up the VMs.

This was also for a very very old version so probably does not apply to you.