Microk8s: Using hostPort

[steveh]

I'm a newbie to k8s and this got me up and running super quick, so thanks for that! I wasn't able to get hostPort working, and came across this comment:

https://github.com/ubuntu/microk8s/blob/11fe17a5c52055eca1959b65d48510eb488ecd3a/microk8s-resources/actions/ingress.yaml#L82

I can't use hostNetwork or nodePort for my particular use case. Is that comment still correct? Digging around, it seems like it can work in newer versions of Calico, but requires a portMap plugin, but I don't really know how to go about installing/enabling that.

[ktsakalozos]

Hi @steveh, It is always a pleasure to see happy kubernauts!

The network plugin we use in microk8s is kubenet, so Calico is out of the picture (at least for now).

I also did not manage to get hostPort working, however since microk8s networks are available to anyone having access to the host machine you can reach any pod or service either from the designated ClusterIP or the Pod IP. For example you can create a pod, get its IP from microk8s.kubectl describe <pod> and use that IP to reach any running services, similarly you can get the cluster IP of a service without exposing it with nodePort. Could you provide some more info on the deployment you are trying to setup? Could you share the yaml manifests? It would be easier for us if we had some concrete use case we could work on.

[steveh]

Sure. I currently use plain Docker containers on my home server. I want to try replace with microk8s to get familiar with it before I try it on some real work.

Say I have a container that listens on 80/tcp and 6881/udp. I want to remap 80/tcp to 8080/tcp on the host.

• hostNetwork is out because you can't specify a different port on the host
• nodePort is out because it only works for ports >32000
• hostPort sounds like a good option but doesn't work
• it looks like an ingress controller might also be suitable - does the version supplied with microk8s support UDP load balancing?

[ktsakalozos]

That was really helpful thank you for providing us with some context.

I am sure you have already read that the use of hostPort and hostNetwork is not recommended because (among others) they limit the pod management options you have with Kubernetes. For example, it is not clear to me how would you upgrade without downtime.

Here is a suggestion that is more aligned with the Kubernetes way. You put the container in a deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: mine
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mine
  template:
    metadata:
      labels:
        app: mine
    spec:
      containers:
      - name: mycontainer
        image: myimage:latest
        ports:
        - containerPort: 80
          name: reliable
        - containerPort: 6883
          name: unreliable

You can run as many pods as you want given the hosted services are stateless.

In front of the deployment you put a service.

apiVersion: v1
kind: Service
metadata:
  name: service-name
spec:
  type: NodePort
  ports:
  - name: reliable
    port: 80
    protocol: TCP
    nodePort: 30080
  - name: unreliable
    port: 6883
    protocol: UDP
    nodePort: 36883
  selector:
    app: mine

At this point you can mange the deployment without any restrictions. What we do not like are the ports. You have two options here:

1. Adjust the nodePort port range. This is done with --service-node-port-range argument in the kube-apiserver (https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/). For microk8s you will need to append the argument to /var/snap/microk8s/current/args/kube-apiserver and restart the api server with sudo systemctl restart snap.microk8s.daemon-apiserver.service. the drawback in this approach is that you may have port conflicts with already running services.

2. Use iptables to forward traffic to the right ports. This is what Kubernetes expects from a loadbalancer. I am sure you are familiar with this approach, just for reference you can see (https://www.cyberciti.biz/faq/linux-port-redirection-with-iptables/). In this approach you can even skip the nodePort entirely. You can expose the service to a fixed ClusterIP (something like 10.152.183.X) and then use iptable rules to forward traffic accordingly.

Important note: port 8080 is already in use by the API server.

[miraculixx]

with the above setup as described by @ktsakalozos you can also use port-forward

# map local:service-port 
microk8s.kubectl port-forward service/service-name 80:30080 6883:36883

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jakecobb]

This may have changed since this issue was opened a couple years ago, but this seems to work today. I have microk8s 1.18.2 and it is using --network=cni with flannel.
The issue linked in the snippet @steveh showed in the original report here indicates hostPort support in the major CNI plugins via the portmap capability and this is present in /var/snap/microk8s/1379/args/cni-network/flannel.conflist:

{
    "name": "microk8s-flannel-network",
    "cniVersion": "0.3.1",
    "plugins": [
      {
        "type": "flannel",
        "name": "flannel-plugin",
        "subnetFile": "/var/snap/microk8s/common/run/flannel/subnet.env",
        "dataDir": "/var/snap/microk8s/common/var/lib/cni/flannel",
        "delegate": {
          "hairpinMode": true,
          "isDefaultGateway": true
        }
      },
      {
        "type": "portmap",
        "capabilities": {"portMappings": true},
        "snat": true
      }
    ]
}

I deployed a pod using hostPort and was able to reach the container successfully using it from the host on 127.0.0.1.

[markmandel]

This may have changed since this issue was opened a couple years ago, but this seems to work today. I have microk8s 1.18.2 and it is using --network=cni with flannel.

Would be great to document this. Agones requires hostPort to be able to connect players to game server Pods, and we've had a few new users get caught by this issue.