Kubespray: Handling Config for Encrypting Secrets at Rest Better

[robottaway]

Feature Request

So I would like to develop this as it has been biting me while managing many clusters here at work. It is also very very nasty to deal with (need deep understanding of k8s, kubespray), and has caused some expensive unplanned work.

My teams do not check in the contents of <inventory>/credentials folder as it contains secrets we do not want in SCM.

When enabling at rest encryption the file <inventory_path>/credentials/kube_encrypt_token.creds contains the token which is base64 encoded and used as secret in apiserver configuration.

If you do not drop a token here then when you run with feature enabled a new one is created and the configuration is updated on your api servers. This is not a good thing to have happen when already you have configured with original secret... it will cause the pre-existing secrets to not be usable and will possibly now create new secrets with a different token creating split.

If you check out a clean copy of the repo and then do some changes to cluster it is very easy to forget to go to secret manager, grab the token/file and put it into this location in inventory path. BUT if you do not do this you can suffer some really nasty consequences.

I am not sure of the best long term way to make managing secrets at rest more user friendly through kubespray, but I would think by default if the secret configuration exists on the master (/etc/kubernetes/ssl/secrets_encryption.yaml) we would not want to just overwrite that file. Really from what I found going through pain of dealing with this we want to extend the functionality of kubespray to manage multiple tokens, and allow automation of process to rotate them while re-encrypting the old secrets.

thoughts?

Environment:

Kubespray version (commit) (git rev-parse --short HEAD):

We are forked off of release-2.8

[rhockenbury]

Agreed. For a quick fix, I ended up writing a short playbook to generate a secret and store it on the masters, and then override kube_encrypt_token with the secret.

# kubespray assumes secret is stored on controller local file system
# this patch will store the secret on the file system on each master node instead
# and set it as a fact for kubespray to consume by overriding `kube_encrypt_token`
- name: Generate the encryption secret
  hosts: kube-master[0]
  tasks:
    - name: Check if etcd encryption secret file exists
      stat:
        path: "{{ kube_cert_dir }}/etcd/{{ encryption_secret_file }}"
      register: stat_secret_file

    - name: Generate new secret and set as fact
      when: not stat_secret_file.stat.exists
      set_fact:
        new_secret: "{{ lookup('password', '/dev/null length=32 chars=ascii_letters,digits') }}"

    - name: Get old secret from master if it exists
      when: stat_secret_file.stat.exists
      shell: cat {{ kube_cert_dir }}/etcd/{{ encryption_secret_file }}
      register: old_secret

    - name: Set secret as fact for kubespray
      set_fact:
        secret: "{{ old_secret.stdout if stat_secret_file.stat.exists else new_secret }}"

- name: Store secrets on masters
  hosts: kube-master
  tasks:
    - name: Check if etcd encryption secret file exists
      stat:
        path: "{{ kube_cert_dir }}/etcd/{{ encryption_secret_file }}"
      register: stat_secret_file

    - name: Store secret if encryption secret is not stored
      when: not stat_secret_file.stat.exists
      block:
        - name: Create the cert directory
          file:
            path: "{{ kube_cert_dir }}/etcd"
            state: directory
            recurse: yes

        - name: Place secret in file
          copy:
            content: "{{ hostvars[groups['kube-master'][0]].secret }}"
            dest: "{{ kube_cert_dir }}/etcd/{{ encryption_secret_file }}"

    - name: Read secret from file
      shell: cat {{ kube_cert_dir }}/etcd/{{ encryption_secret_file }}
      register: secret

    - name: Set secret as fact for kubespray
      set_fact:
        kube_encrypt_token: "{{ secret.stdout | trim }}"

[robottaway]

That looks pretty close to what I'd like. I think the downside is it still is Out-of-Band from the regular flow and could suffer the same fate when a team mate checks out and forgets to run this first. Maybe you have a solution for that though?

[robottaway]

Whoops, ping @rhockenbury

[chadswen]

For more background there was a closed PR related to this, #3964 by @kmadnani not sure why it was closed. We definitely need to fix this in kubespray IMO.

[chadswen]

I think the downside is it still is Out-of-Band from the regular flow and could suffer the same fate when a team mate checks out and forgets to run this first. Maybe you have a solution for that though?

We should work to get a solution in-tree that runs within the existing kubespray playbooks.

[robottaway]

@rhockenbury would you be ok with me taking your playbook and working into the tree? I don't want to subsume that if you are eager to do it.

[rhockenbury]

Go for it. 👍

[mcntrn]

@robottaway I'm working on a solution for this today. Potentially based on the sample from @rhockenbury, since my team needs it ASAP. Unless you have already have solution ready to go, I can go ahead an make a PR.