Microk8s: ssh tunnel documentation

[asloan7]

Following the tutorial at https://tutorials.ubuntu.com/tutorial/install-a-local-kubernetes-with-microk8s#3, but on a remote server, I have successfully set up an ssh tunnel to the kubernetes-dashboard but am having trouble with grafana

microk8s.kubectl get services -A|grep grafa
kube-system monitoring-grafana ClusterIP 10.152.183.21 80/TCP

microk8s.kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:16443
Grafana is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

I expected these to work:
ssh -L 8000:10.152.183.21:80 -i ~/pemfile.pem [email protected]
http://localhost:8000

ssh -L 16443:10.152.183.21:16443 -i ~/pemfile.pem [email protected]
https://localhost:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

But neither seem to

[balchua]

@asloan7 what i usually do when i need to access remotely grafana is to run kubectl port-forward from my local pc.

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/

Then you can use it like its a localhost.
Maybe this can help.

[asloan7]

thx @balchua , and to be comprehensive that also requires on laptop kubectl, .kube/config (is this the output of microk8s.config?), and on server an addition to .ssh/authorized_keys?
microk8s.config contains a private IP, not a public one, will a swap work?

[balchua]

You will need on your laptop kubectl and the kubeconfig. You can get the kubeconfig by doing this microk8s.kubectl config view --raw. Need to save that to a file. Copy that to your laptop's ~/.kube/config. You don't need to add any ssh key anymore at the server. Then you can do kubectl port-forward.

[asloan7]

@balchua getting there
microk8s.kubectl config view --raw gives 127.0.0.1, so it wouldn't know how to reach the server:

• cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://127.0.0.1:16443
  So got "Unable to connect to the server: dial tcp [127.0.0.1:16443]: connectex: No connection could be made because the target machine actively refused it"

Changing to server: https://(publicIP):16443
gets closer:
kubectl.exe get nodes
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 172.31.7.179, 10.1.16.0, not (publicIP)

[balchua]

You can add extra IP addresses in the /var/snap/microk8s/current/certs/csr.template.conf.

Example:

IP.1 = 127.0.0.1
IP.2 = 10.152.183.1
IP.9 = your_public_ip
#MOREIPS

This will automatically generate new certs and restart the apiserver.
After that you need to get the new kubeconfig again.

[strigona-worksight]

You can update the IP in your config file. To fix the certificate issue:

Add Names to Kubernetes API Certificate

Modify:

/var/snap/microk8s/current/certs/csr.conf.template

Add:

DNS.X =

Add:

IP.Y =

Where X and Y don't conflict with values in csr.conf. Setting a high value such as 20 should be safe. Certificates will be automatically generated after the file is saved. Microk8s needs to be restarted for the certificate to be reloaded.

[asloan7]

@balchua @strigona-worksight Added my public IP to the csr.conf.template, did microk8s.stop, microk8s.start
kubectl.exe get nodes now works from my laptop, connecting to server

Still having trouble figuring out the kubectl port-forward line from this info (hopefully all relevant info is here):

microk8s.kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:16443
Grafana is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

microk8s.kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.152.183.1 443/TCP 6d3h
kube-system kube-dns ClusterIP 10.152.183.10 53/UDP,53/TCP,9153/TCP 6d3h
kube-system kubernetes-dashboard ClusterIP 10.152.183.45 443/TCP 6d3h
kube-system monitoring-grafana ClusterIP 10.152.183.21 80/TCP 6d3h

Try 1:
kubectl port-forward svc/kubernetes 16443:16443
error: cannot attach to *v1.Service: invalid service 'kubernetes': Service is defined without a selector

Try 2:
kubectl port-forward svc/kube-dns 16443:9153 -n kube-system
Forwarding from 127.0.0.1:16443 -> 9153
Forwarding from [::1]:16443 -> 9153
Handling connection for 16443
E1127 14:28:52.842977 16068 portforward.go:385] error copying from local connection to remote stream: read tcp4 127.0.0.1:16443->127.0.0.1:22631: wsarecv: An existing connection was forcibly closed by the remote host.

Firefox's complaint for that error is:
https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
Secure Connection Failed
An error occurred during a connection to 127.0.0.1:16443. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

[strigona-worksight]

What is it that you want to be forwarded to your PC? You don't need to forward the kubernetes internal resources to your PC unless you have some special need to.

Do basic kubectl commands work from your PC to your remote server? Does this succeed? kubectl.exe get nodes

[balchua]

If you are going to access grafana from your local pc using port forward here's what i usually do.
kubectl -n monitoring-grafana port-forward 7777:5555 svc/grafana

Where:
7777 - is the local pc port you want to use. So it could be anything.
5555 - is the port defined in your grafana Service in kubernetes.

How to access it from your laptop.
http://localhost:7777/

[asloan7]

@strigona-worksight I put too much in the last comment, but yes kubectl get nodes, etc. works now

@balchua grafana is the only one I really want, but thought I'd be able to do it over https with this proxy at 127.0.0.1:16443 if I could get the port-forward line correct.

microk8s.kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:16443
Grafana is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

[strigona-worksight]

Ah, I see. I'm not entirely sure, but I believe that is a somewhat special endpoint? That being said, you can probably replace 127.0.0.1 with your public IP:
https://<public IP>:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
and hit it that way without the need for a proxy, though I forget how to authenticate to it.

In reality, you don't gain anything by proxying Grafana locally as http - the traffic is all going through your API server which is https, so it is only HTTP once it reaches your PC.
[browser]<--http-->[kubectl proxy]<--https-->[microk8s]<--http-->[grafana]

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.