Kubespray: Clarify whats in each cert dir, and feature request for cert reissuance as an ansible role

[charandas]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Feature request as well as asking for clarification about certs, and certs re-issuance for existing hosts.

I currently have two directories (/etc/kubernetes/ssl/ and /etc/ssl/etcd/ssl/) of certs on all of my deployments (on each node). I have 5+ different clusters deployed using kubespray.

diff /etc/ssl/etcd/ssl/node-node6-key.pem /etc/kubernetes/ssl/node-node6-key.pem

// will show these are two different keys and I am not sure why

In any case, I need to re-issue certs on all my systems, because I wanna start using central DNS, and I mistakenly deployed these clusters with hard-coded ip addresses (as my inventory below would show). As I understand it, each of those TLS certs are now hardcoded to my ip addresses, and will require re-issuance (correct me if I am wrong please).

Environment:

• Cloud provider or hardware configuration:
  vSphere

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 4.14.11-coreos x86_64
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.1.0
VERSION_ID=1632.1.0
BUILD_ID=2018-01-05-0908
PRETTY_NAME="Container Linux by CoreOS 1632.1.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Version of Ansible (ansible --version):
  ansible 2.5.3

Kubespray version (commit) (git rev-parse --short HEAD):
c4b1808983b

Certs in 2 locations

dev-k8s-node-02 core # cd /etc/kubernetes/ssl/
dev-k8s-node-02 ssl # ls
admin-node2-key.pem  ca.pem                      kube-controller-manager-key.pem  kube-proxy-node2.pem      kube-proxy-node5-key.pem  kube-scheduler.pem  node-node3-key.pem  node-node5.pem
admin-node2.pem      front-proxy-ca-key.pem      kube-controller-manager.pem      kube-proxy-node3-key.pem  kube-proxy-node5.pem      node-node1-key.pem  node-node3.pem      node-node6-key.pem
apiserver-key.pem    front-proxy-ca.pem          kube-proxy-node1-key.pem         kube-proxy-node3.pem      kube-proxy-node6-key.pem  node-node1.pem      node-node4-key.pem  node-node6.pem
apiserver.pem        front-proxy-client-key.pem  kube-proxy-node1.pem             kube-proxy-node4-key.pem  kube-proxy-node6.pem      node-node2-key.pem  node-node4.pem      service-account-key.pem
ca-key.pem           front-proxy-client.pem      kube-proxy-node2-key.pem         kube-proxy-node4.pem      kube-scheduler-key.pem    node-node2.pem      node-node5-key.pem

dev-k8s-node-02 ssl # ls /etc/ssl/etcd/ssl/
admin-node2-key.pem  ca-key.pem  member-node2-key.pem  node-node1-key.pem  node-node2-key.pem  node-node3-key.pem  node-node4-key.pem  node-node5-key.pem  node-node6-key.pem
admin-node2.pem      ca.pem      member-node2.pem      node-node1.pem      node-node2.pem      node-node3.pem      node-node4.pem 

Example Inventory File

[all]
node1 ansible_host=192.168.0.56 ip=192.168.0.56 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity
node2 ansible_host=192.168.0.57 ip=192.168.0.57 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity
node3 ansible_host=192.168.0.58 ip=192.168.0.58 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity
node4 ansible_host=192.168.0.59 ip=192.168.0.59 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity
node5 ansible_host=192.168.0.60 ip=192.168.0.60 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity
node6 ansible_host=192.168.0.61 ip=192.168.0.61 ansible_ssh_user=core ansible_ssh_private_key_file=~/.ssh/some-identity

# ## configure a bastion host if your nodes are not directly reachable
# bastion ansible_host=x.x.x.x ansible_user=some_user

[kube-master]
node1
node2

[etcd]
node1
node2
node3

[kube-node]
node2
node3
node4
node5
node6

[kube-ingress]
node2
node3

[k8s-cluster:children]
kube-master
kube-node
kube-ingress

[charandas]

Anyone care to comment? With some clarity, I might be able to take a knack at this.

[kaarolch]

@charandas If you do some dig in doc or code you probably found this:
vault usage and there you have short clarification:

As a result of the Vault role will be create separated Root CA for etcd, kubernetes and vault. Also, if you intend to use a Root or Intermediate CA generated elsewhere, you'll need to copy the certificate and key to the hosts in the vault group prior to running the vault role. By default, they'll be located at:

vault:
......
etcd:
/etc/ssl/etcd/ssl/ca.pem
/etc/ssl/etcd/ssl/ca-key.pem
kubernetes:
/etc/kubernetes/ssl/ca.pem
/etc/kubernetes/ssl/ca-key.pem

This doc is not precise as you would expect but as I understand for different services kubesrapy use different certificates. So you do not have a one global certificate per host but separate certs per services: vault, kubernetes, etcd, calico, etc... This commit 6eb22c5db was changed the behavior.

[charandas]

That helps @kaarolch. Since there would be quite a number of certs since we have them per application, I wonder if the feature-request is pertinent to any other users out there?

[kaarolch]

Two days ago I was thinking that maybe would be easier to users when they get short description of the certificate generation process but I was not sure if anyone need it:) Now I know that would be nice to have it. I can handle it.

[charandas]

@kaarolch to clarify, you will be working on the docs only, or an ansible role as well?

[kaarolch]

I was more thinking about documentation. The gen_cert role is defined for different services i.e.: etc certs
Also I'm not sure what plan for certificates maintainers have. Currently there are a few places where the same script was used:

• kuberentes
• etcd

Btw. I'm not sure if you have to regenerate certificates if you introduce central DNS. Do you have any purpose for it?

[charandas]

Thanks for engaging me on this. So i have a disaster recovery scenario where my VMs are replicated to another environment with different static ips than they were.

Do you see how that would break the certs if they are hard coded to the first set of ip addresses?

[kaarolch]

I guess that there are more place that you need change configuration. I can look on it tomorrow but from my perspective in this case this would be a little tricky if you would like to recover each node from backup as I correctly understand yours scenario.

[charandas]

@kaarolch what would that configuration be? I am asking assuming the ip addresses are the only part in configuration, and I plan on eradicating using common DNS names in both systems.

Lastly, my hypervisor (VMWare) does all this VM syncing in memory (with help of Zerto), and my sysadmin tells me the VM doesn't notice much of a blip. It just thinks its resuming where it left off and there isn't even a power cycle involved.

[charandas]

@kaarolch I actually realized this morning that you are right, there is much more to it than certs if I am transitioning an existing cluster. Those IPs must have made to several config files in the initial run.

[kaarolch]

@charandas Sorry I was quite busy today and forgot to answer. Yes, there would be a lot things to change from my perspective the better option would have separate cluster (A, B) and some switch between or just one kube cluster splitted between DC. The only options that your scenario would works correctly is the same IP inside vmware cluster network. If I understand correctly you would like to achive active/passive infra but passive is like online mirror (cpu, memory, disk, etc.) This probably won't be so easy.

[charandas]

While active/passive is not feasible, I wanted to add here that allocating another non-routable interface helped in our case with keeping the kubernetes clusters more adaptable to internal IP Range changes.

This non-routable interface is almost like a private network for the nodes in a k8s cluster, and we use the ip param in hosts.ini inventory file to bind kube services on that interface. This way, if down the line, the more public interface on which we bind ingress and such for external consumption neeeds to change due to shifting infrastructure, the k8s cluster itself remains in good standing, without needing to reissue certs.

Thanks @kaarolch again for clarifying things last we talked.