Microk8s: Cannot connect to microk8s from another machine

[soapergem]

I just ran a fresh install of microk8s with this command:

sudo snap install microk8s --classic --channel=1.14/stable

I can access everything while I'm SSH'ed into the box this is running on, with commands much like this:

microk8s.kubectl get pods

However when I set my ~/.kube/config file - on another machine on the same network - to this:

apiVersion: v1
clusters:
- cluster:
    server: http://192.168.1.123:8080
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    username: admin

And then try to run this command:

kubectl get pods

I get this error message:

The connection to the server 192.168.1.123:8080 was refused - did you specify the right host or port?

In fact, when I run curl localhost:8080 while on the box, I see valid K8S output, but if I try running curl 192.168.1.123:8080 from outside the box, even that returns:

curl: (7) Failed to connect to 192.168.1.123 port 8080: Connection refused

I've tried a couple of things to get this working, first of which was allowing access to 8080 on the firewall with this command:

sudo ufw allow 8080
sudo ufw enable

Unfortunately that doesn't solve the problem. When I run netstat -an | grep "LISTEN " I can see an entry for port 8080 as follows:

tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN

But because it says 127.0.0.1 instead of 0.0.0.0, this would explain why it's not working off the box. Is there some microk8s.* command I don't know about, to open up access?

[ktsakalozos]

In v1.14 we restricted the insecure access to MicroK8s from port 8080 to local users only.

You have two options:

• Open insecure port 8080 to everyone. To do this you need to edit /var/snap/microk8s/current/args/kube-apiserver and set the --insecure-bind-address=127.0.0.1 to 0.0.0.0. Then restart MicroK8s with microk8s.stop and microk8s.start.

• Use the secure port 16443. Due to a recent bug fix you need to get MicroK8s from edge sudo snap install microk8s --channel=1.14/edge --classic. This bug fix will soon reach stable. Then use the config you get with microk8s.config to reach the cluster.

[soapergem]

Great, thank you @ktsakalozos . However now I've encountered another problem. I was able to get things working on another machine on my LAN. (I had to run sudo ufw allow 16443 first.) I also enabled port forwarding on my router from port 16443 to the Ubuntu server, and unfortunately it does not seem that it is working correctly when I try to access the same instance from outside of my LAN.

I've taken the output of microk8s.config for use in my ~/.kube/config file. And when I'm accessing it externally, I change the IP from 196.168.1.123 to the IP of my router. And I can tell it's getting there because the response is almost instantaneous. However when I try to run something like kubectl get pods from outside the network, this is the response I get:

Error from server (BadRequest): the server rejected our request for an unknown reason

Any ideas?

[ktsakalozos]

Can you edit /var/snap/microk8s/current/args/kube-apiserver and add a -v=9 and restart MicroK8s? We may be able to see what the problem is from the http request/response reaching the API server.

[soapergem]

I think I may have fat-fingered something when I copied my ~/.kube/config file. This time I copied it verbatim from the output of microk8s.config -- and then updated it to use the external IP address, as before -- and now I'm getting a different error:

Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 192.168.1.123, not <my-external-ip>

And here I'm using <my-external-ip> as a placeholder; it does in fact show my external IP on that line. Is there any way for me to re-generate the cert such that it allows me to enter the server's external IP?

P.S. I'm not sure that adding -v=9 to that file did anything. Would that add more information to a log file, or something like that?

[balchua]

@soapergem, i think there is a way to tell kubectl to ignore certs by adding --insecure-skip-tls-verify. I've never tried though. I got it from kubectl reference doc.
https://kubernetes.io/docs/reference/kubectl/kubectl/

[ktsakalozos]

@soapergem please give the suggestion of @balchua a try.

Your request of being able to add external IPs to the certificate seems reasonable but we do not support this yet. The csr.conf.template file https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/certs/csr.conf.template where additional IPs can be added is found under /snap/microk8s/current/certs/csr.conf.template and is read only. A fix would be to place this file under /var/snap/microk8s/current/certs/csr.conf.template by the configure hook https://github.com/ubuntu/microk8s/blob/master/snap/hooks/configure and update this line https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L125 to pick up the template file from the new location. The user would then be able to edit the template file and add any external IPs required.

I am marking this issue as a "feature request" and "help needed" in case anyone wants to work on this. The fix should be easy but I cannot give you an ETA.

[soapergem]

Yes, adding --insecure-skip-tls-verify to my kubectl commands does seem to work. As this is just a development server, I'll do that for now. But I'll also leave this open in case anyone wants to work on the feature request.

Although, one thing important to note: while that flag works for kubectl, it doesn't work for helm. So that just raises the importance of adding extra IPs to the certificate, in my mind.

Update: looks like I can add the insecure-skip-tls-verify flag to my ~/.kube/config file directly (I have to comment out the certificate-authority-data). So I can get this working with Helm after all.

[balchua]

@ktsakalozos i will give this issue a shot.

[soapergem]

Hey @ktsakalozos earlier you said:

This bug fix will soon reach stable. Then use the config you get with microk8s.config to reach the cluster.

How does one track when a new stable release gets cut? Since there's nothing listed on the releases page, how can I tell once a new release has been promoted to stable? Are there release notes anywhere?

[ktsakalozos]

At the moment, @soapergem, the best way to check for new releases is with:

snap info microk8s

The edge channel gets immediately updated with the code we have on this repo. Upon an upstream patch release (eg from 1.12.7 to 1.12.8) the edge channel is pushed to beta and candidate. About a week after a candidate release the new version is pushed to stable.

In the future we would like to automate the release notes authoring and make use of the github release page.

[soapergem]

I see that there's been some movement on this, allowing updates to the csr.conf.template file. However I'm running into some issues now when I try to apply changes. I'm trying to add extra lines to that file for DNS.6 (containing an A record I set up) and also IP.3 (being the public IP that forwards to my microk8s machine). However, when I try to edit the file at /snap/microk8s/current/certs/csr.conf.template it will not let me save. In fact I can't do anything with that file, and this seems to be by design with Ubuntu's snap system. If I try and run sudo rm on it I get: cannot remove: Read-only file system. If I create the file in my home directory and then try to use sudo mv to copy it into place, I get mv: inter-device move failed: unable to remove target: Read-only file system. How do I update this file?

[strigona-worksight]

@soapergem You are correct that /snap/microk8s/current/certs/csr.conf.template is read only. Instead you should be modifying the file in /var/, full path is: /var/snap/microk8s/current/certs/csr.conf.template. Hope that helps.

[soapergem]

Great, that worked perfectly. I'm going to close this issue because what I was hoping for has been accomplished.

[nrvale0]

After reading this thread it looks like there is a template and hook available to allow tuning of the certs but I don't see anything on the docs site/in the docs as to how one might leverage. Am I missing something or does the above need docu'ed?

[ktsakalozos]

@nrvale0, for sure this needs documenting. We mark issues like as Q&A and our intention is to move them to https://discuss.kubernetes.io/ as wiki topics. Your thoughts on this @balchua and everyone?

@evilnick we do not want to miss this one.

[balchua]

I hardly go to discuss.kubernetes.io. 😁. My preference as a microk8s user is to have faq on common issues like today.
Microk8s.io is the first place a user will end up with, followed by github.

[ktsakalozos]

Microk8s.io is the first place a user will end up

Good point, we will have to mirror any posts from discuss.kubernetes.io on microk8s.io.

[lukos]

I don't think you should duplicate effort so unless the intention is to automatically pull in relevant content (by tag?), it might be better to have a cookbook/recipe section hosted at the microk8s site and if the content is identical to Kubernetes running on other platforms, then a simple link would be OK, otherwise if it needs modifying for microk8s, it could be either copied and modified or else something like, "follow the steps found in replacing kubectl with microk8s.kubectl.

[nrvale0]

As a data point for someone who is not new to k8s but is new to microk8s, this was my process:

1) look at mk8s docs site for how to allow external access to API provider
2) look at README in repo
3) repo Issues search
4) back to mk8s docs site
5) back to this Issue to ask

[kawsark]

Great, that worked perfectly. I'm going to close this issue because what I was hoping for has been accomplished.

@soapergem After you edit this file, how do you trigger microk8s to generate the new certs? I tried restarting and then doing sudo microk8s.kubectl config view --raw. However, when I try to connect I still get the error: Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 10.142.0.5, 10.1.38.0, not <external IP>

[soapergem]

@kawsark after editing the file you need to run first microk8s.stop and then microk8s.start. FYI I posted about this, along with another way to regenerate certs, on this post.

[kawsark]

Hello @soapergem Thank you for your help. Unfortunately I am still unable to get this working. For now I am using a workaround of SSH Tunneling. I provided the steps to reproduce the problem below. Any help is greatly appreciated!

1. Installed mircrok8s on a ubuntu 18.04.4 LTS server using sudo snap install microk8s --classic
2. Wait until all services are up, then edit /var/snap/microk8s/current/certs/csr.conf.template to add IP.3 as below

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 10.152.183.1
IP.3 = 34.xx.xx.xx
#MOREIPS

1. Run microk8s.stop and then microk8s.start
2. Save the output from sudo microk8s.kubectl config view --raw locally as and modify server: https://127.0.0.1:16443 to server: https://34.xx.xx.xx:16443
3. Run export KUBECONFIG=./microk8s.yaml and kubectl get nodes. I get the error:

➜  kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 10.142.0.19, 10.1.14.0, not 34.xx.xx.xx

[shoover]

@kawsark check the generated crt conf. When I did this, the MOREIPS tag expanded to overwrite IP.3 with another IP. After changing the template to IP.9 (out of range of those added by MOREIPS), it worked.

[kawsark]

@shoover Thank you so much for that suggestion, that was indeed the issue! The /var/snap/microk8s/current/certs/csr.conf file had up to IP.4. So I added my public IP as IP.5 and was able to connect without issue. Thanks again!

[pfisterer]

@kawsark Thanks a lot, you saved my day.

For anybody interested in this, I've used the above hint to create an Ansible playbook (https://github.com/pfisterer/edsc-microk8s-playbook) that installs microk8s in OpenStack and modifies the CSR accordingly.