Microk8s: Documentation for permantly replacing API certificates.

[VladimirAkopyan]

I have microK8S cluster, and expose the API server at my domain.
The server.crt and server.key in /var/snap/microk8s/1079/certs need to be replaced with the ones that include my domain.
Otherwise, as expected, i get the error:

Unable to connect to the server: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local, not mydonaim.com

With the help of cert-manager I have produced certificates and replaced them, the system works fine now.

Problem: every time server is restarted, server.crt and server.key are generated again in
/var/snap/microk8s/1079/certs. My custom certs are deleted, making API server unreachable remotely.
How can I stop the system from doing that all the time?

Workaround?
Should I place my certificates elsewhere and edit config files like /var/snap/microk8s/1079/args/kube-controller-manager with the path to those certificates?
Are those config files auto-replaced as well?
I would be happy to contribute to documentation, if this is the correct approach.

Cluster information:

• Kubernetes version: 1.16.3
• Cloud being used: Bare metal, single-node
• cluster Installation method: Ubuntu Server with Snaps
• Host OS: Ubuntu 18.04.3 LTS

[balchua]

@VladimirAkopyan there is a file called /var/snap/microk8s/current/certs/csr.conf.template where you can add your own domain under the DNS section.

For example:

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = mydomain.com

After changing, the apiserver-kicker will automatically detect the difference , generate new certs and restart the apiserver.
Can you try that?

[VladimirAkopyan]

Thank you, this is great, but Jesus, it would have saved me so much time to know this! I coundn't find this in documentation anywhere! Maybe I am being daft?

You really need a lot more documentation folks,

Have you considered making documentation on https://microk8s.io/docs/ editable, either from git like azure docs, or a form of wiki? Especially since it's a "one-click install" package and will attracts some kubernetes noobs.

I am exploring how to stand up production grade kubernetes on bare metal, and would be great to have guidance in that direction. I would be happy to contribute. Otherwise, i write articles for my blog.

[balchua]

@VladimirAkopyan Thank you for the feedback. Apologies for the missing information.

@ktsakalozos is there anyway for contributors to update the docs as @VladimirAkopyan has mentioned.

[strigona-worksight]

The source for microk8s.io is hosted in this repo: https://github.com/canonical-web-and-design/microk8s.io

[lazzarello]

Hello! I'm also working on this exact issue. Thanks for all the help!

[strigona-worksight]

Also relevant comments here: https://github.com/ubuntu/microk8s/issues/816#issuecomment-558354391

[ktsakalozos]

@VladimirAkopyan it is in our intentions to move the docs to a wiki based system. Until then you can suggest changes in the docs (https://microk8s.io/docs/) by clicking the " edit this page " link. You will be directed to github where a wiki view of the changes is available.

[VladimirAkopyan]

Ow cool, my apologies, i have not noticed it.

[mac2000]

Not sure if related - there is no docs for ingress certs being used

In my case I have deployment with ingress and valid certificates, everything in ok in browser, but trying to talk to service from nodejs gives me unable to verify the first certificate reason seems to be in using default certs instead of provided because of lack of SNI being send, e.g.:

curl -v https://contoso.com

will show valid certs and everything is ok, but:

openssl s_client -showcerts -connect contoso.com:443

I do see

Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

In same time exactly the same ingress configuration inside GKE everything works as expected (but GKE has proprietary ingress controller)

[ktsakalozos]

@mac2000 for the ingress certificate you may want to look into the --default-ssl-certificate of the ingress controller [1]. You will need to microk8s kubectl edit the daemonset to add the argument and then delete the ingress pods so as to force a respawn with the introduced flag.

We recently merged a PR [2] that will allow you to set the ssl certificate when you microk8s enable ingress but this work has not made it to latest/stable yet, you can find it on latest/edge (sudo snap install microk8s --classic --channel=latest/edge).

[1] https://kubernetes.github.io/ingress-nginx/user-guide/tls/
[2] https://github.com/ubuntu/microk8s/pull/1231

[mac2000]

@ktsakalozos after further investigation it is more or less related to wildcard certs

e.g. with almost same ingress config, aka:

# ...
spec:
  rules:
  - host: acme.contoso.com
    http:
      paths:
      - backend:
          serviceName: acme
          servicePort: acme
        path: /*
  tls:
  - hosts:
    - acme.contoso.com
    secretName: wildcard.contoso.com
# ...

in GKE sending request without SNI uses wildcard.contoso.com in microk8s Kubernetes Ingress Controller Fake Certificate and it seems like it is more because of difference in nginx ingress controller used in microk8s and proprietary controller used in google cloud

suggested --default-ssl-certificate wont work here because microk8s handles multiple different domains

PS: had to contact support to figure out that in google cloud we need path to be exactly /* rather than / or /.* in microk8s