Microk8s: How to chage a IP range

[niwashing]

IP range 10.1.1.0/24, 10.152.183.0/24 are userd for cluster or pods by default.
How can I change a cluster IP and IP range that is assigned to nodes?

[ktsakalozos]

Hi @niwashing

These two IP ranges are configured in a couple places:

10.152.183.0/24:

• /var/snap/microk8s/current/args/kube-apiserver: --service-cluster-ip-range=10.152.183.0/24
• /var/snap/microk8s/current/args/kubelet: --non-masquerade-cidr=10.152.183.0/24
• /var/snap/microk8s/current/args/kube-proxy: --cluster-cidr=10.152.183.0/24

10.1.1.0/24:

• /var/snap/microk8s/current/args/cni-network/cni.conf "subnet": "10.1.1.0/24",
• /var/snap/microk8s/current/args/kubelet:--pod-cidr=10.1.1.0/24

I hope I am not missing anything. Remember to stop/start MicroK8s after you update any of those arguments.

Thank you for using MicroK8s

[niwashing]

@ktsakalozos

Thank you! I could change ip range, but microk8s.enable dns still throw error as follows:

$ microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/kube-dns unchanged
configmap/kube-dns unchanged
deployment.extensions/kube-dns configured
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": provided IP is not in the valid range. The range of valid IPs is 172.22.183.0/24
Failed to enable dns

Although I found /snap/microk8s/354/actions/dns.yaml and /snap/microk8s/354/actions/enable.dns.sh still include a default IP range, they cannot be changed due to snap permission.

Do I have to build microk8s from source by using snapcraft?

[ktsakalozos]

You have the option to recompile MicroK8s and produce your own .snap file.

I suspect that after microk8s.enable dns you can also microk8s.kubectl edit the part of the dns manifest that is failing.

[niwashing]

I suspect that after microk8s.enable dns you can also microk8s.kubectl edit the part of the dns >manifest that is failing.

Sorry, I'm not very familier with kubernetes,
but kubectl edit is not available because kube-dns has not been deployed yet due to ip range error?

I would appreciate if you could show me entier commands.

[ktsakalozos]

Sure, here is what I have:

> microk8s.kubectl get all --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   pod/kube-dns-6ccd496668-qx5m4   3/3     Running   0          41s

NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.152.183.1    <none>        443/TCP         72s
kube-system   service/kube-dns     ClusterIP   10.152.183.10   <none>        53/UDP,53/TCP   41s

NAMESPACE     NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/kube-dns   1/1     1            1           41s

NAMESPACE     NAME                                  DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/kube-dns-6ccd496668   1         1         1       42s

At this point I have kube-dns running, it should be failing in your case. I suspect you can go and edit the kube-dns service clusterIP with:

microk8s.kubectl edit -n kube-system service/kube-dns

If this is does not work you will need to download and edit this file https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/dns.yaml and then microk8s.kubectl apply -f ./dns.yaml

[fquinner]

I can confirm I just hit this too because i didn't enable dns before adding pods so there was an IP conflict. I can also confirm that the installation of the dns.yaml definition was required since dns never started.

[AdamIsrael]

What happens when the microk8s snap refreshes? Since the files to be edited are in /var/snap/microk8s/current/, I suspect the changes will revert back to the defaults.

What about storing the subnet to use in /var/snap/microk8s/common, and modifying the configuration to get the value from that file?

[ktsakalozos]

@AdamIsrael I do not think this is what happens during a refresh. ${SNAP_DATA} is backed up and could be reverted (contrary to ${SNAP_COMMON}) but the contents are preserved. If you configure your daemons in a specific way we respect your configuration, we do not overwrite the configuration with the defaults. Have a look here https://forum.snapcraft.io/t/proper-way-to-simulate-a-snap-refresh-release/5565 on how you can simulate a refresh and check this yourself.

[AdamIsrael]

@ktsakalozos That's good to know, thanks!

I still think having the subnet defined in a single location would be better. Also perhaps a command for changing it to a specific or random subnet?

The specific use-case I have is that I may install microk8s to lxd, and thus have multiple microk8s on a host. I can create new networks in lxd for each, but I then need for each microk8s to use the appropriate network.

[evilhamsterman]

I agree with @AdamIsrael, I started playing with K8s and decided that since we were and Ubuntu shop microk8s would be a great way to get started. However we also happen to use the subnet 10.1.1.0/24 and so the cbr0 interface caused issues for me accessing portions of our network.

[camille-rodriguez]

Hello everyone, @ktsakalozos,
I followed these steps to deploy microk8s on a different range (simply 10.152.182.0/24). However, I run into a similar issue when trying to enable istio. It tries to re-deploy kube-dns (even if it is already deployed) and gets stuck because of the clusterIP spec again.

$ microk8s.kubectl get all --all-namespaces
NAMESPACE            NAME                                        READY   STATUS             RESTARTS   AGE
container-registry   pod/registry-d7d7c8bc9-g86qw                0/1     Pending            0          15h
kube-system          pod/coredns-9b8997588-v2hs6                 0/1     Running            3          16h
kube-system          pod/hostpath-provisioner-7b9cb5cdb4-c2z2l   0/1     CrashLoopBackOff   22         15h
kube-system          pod/kube-dns-579bd8fb8d-gh2m6               0/3     InvalidImageName   0          15h

NAMESPACE            NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
container-registry   service/registry     NodePort    10.152.182.65   <none>        5000:32000/TCP   15h
default              service/kubernetes   ClusterIP   10.152.182.1    <none>        443/TCP          17h
kube-system          service/kube-dns     ClusterIP   10.152.182.10   <none>        53/UDP,53/TCP    15h

NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
container-registry   deployment.apps/registry               0/1     1            0           15h
kube-system          deployment.apps/coredns                0/1     1            0           16h
kube-system          deployment.apps/hostpath-provisioner   0/1     1            0           15h
kube-system          deployment.apps/kube-dns               0/1     1            0           15h

NAMESPACE            NAME                                              DESIRED   CURRENT   READY   AGE
container-registry   replicaset.apps/registry-d7d7c8bc9                1         1         0       15h
kube-system          replicaset.apps/coredns-9b8997588                 1         1         0       16h
kube-system          replicaset.apps/hostpath-provisioner-7b9cb5cdb4   1         1         0       15h
kube-system          replicaset.apps/kube-dns-579bd8fb8d               1         1         0       15h

$ microk8s.enable istio
Enabling Istio
Enabling DNS
Applying manifest
serviceaccount/coredns unchanged
configmap/coredns unchanged
deployment.apps/coredns unchanged
clusterrole.rbac.authorization.k8s.io/coredns unchanged
clusterrolebinding.rbac.authorization.k8s.io/coredns unchanged
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": field is immutable
Failed to enable dns
Failed to enable istio

Could you recommend what I should do with this, or which files I should download and edit? Thank you!

[sadoMasupilami]

Hello,
this problem is still relevant now with flannel.
we do a lot of work from home (using VPN) and many of our internal services run in 10.1.0.0/16

is there any way to change the adresses with flannel?
ui tried to change /var/snap/microk8s/current/args/flannel-network-mgr-config and after a reset and reboot the new range is used. But sometimes it just stops working as flannel daemon can not start with:

Jul 02 09:50:48 xxx microk8s.daemon-flanneld[1747]: error #0: dial tcp: lookup none on 127.0.0.53:53: server misbehaving
Jul 02 09:50:53 xxx microk8s.daemon-flanneld[2174]: Error: dial tcp: lookup none on 127.0.0.53:53: server misbehaving

we want to replace minikube with microk8s for local development on linux. So it would be really important to us.

Thanks for any good tips

[strigona-worksight]

Microk8s 1.16+

Modify:

/var/snap/microk8s/current/args/flannel-network-mgr-config

Change: "10.1.0.0/16" to: "10.8.0.0/16" (or any other range)

Then restart microk8s:

microk8s.stop

microk8s.start

[uGiFarukh]

Right now, what is the best option to achieve this local IP range change for the latest 1.19 microk8s without breaking any addons or functionality? And is it possible to use custom ip ranges outside of the standard ip ranges used in local? for example: 1.1.1.1? Or will this conflict with the internet?

[uGiFarukh]

@ktsakalozos any idea on how to achieve this properly without breaking anything?

[ktsakalozos]

@uGiFarukh for 1.19 we have the following:

There are two main IP ranges you may want to change.

1. The range where cluster IPs are from. By default this is set to 10.152.183.1/24. To change the cluster ip range you need to:
   
   
   
   • Stop all services with microk8s.stop
   
   
   • Clean the current datastore and CNI with:
     
     (cd /var/snap/microk8s/current/var/kubernetes/backend/; rm -v !(cluster.key|cluster.crt) ) echo "Address: 127.0.0.1:19001" > /var/snap/microk8s/current/var/kubernetes/backend/init.yaml rm /var/snap/microk8s/current/args/cni-network/calico-kubeconfig
   
   
   • Edit /var/snap/microk8s/current/args/kube-apiserver and update the --service-cluster-ip-range=10.152.183.0/24 argument of the API server.
   
   
   • Edit /var/snap/microk8s/current/certs/csr.conf.template and replace IP.2 = 10.152.183.1 with the the new IP the kubernetes service will have in the new IP range.
   
   
   • If you are also setting up a proxy update /var/snap/microk8s/current/args/containerd-env with the respective IP ranges
   
   
   • Start all services with microk8s.start
   
   
   • Reload the CNI with microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml
   
   
   • To enable dns you should not use the packaged addon. Instead you should:
     
     
     
     
     
     • make a copy of the dns manifest with cp /snap/microk8s/current/actions/coredns.yaml /tmp/.
     
     
     
     • In this manifest copy update the clusterIP: 10.152.183.10 to an IP in the new range and replace the $iALLOWESCALATION string with false.
     
     
     
     • Apply the manifest with microk8s kubectl apply -f /tmp/coredns.yaml
     
     
     
     • Add the following two arguments on the kubelt arguments at `/var/snap/microk8s/current/args/kubelet:
       
       
       --cluster-domain cluster.local --cluster-dns <the cluster ip of the dns service you put in the coredns.yaml>
     
     
     
     • Restart MicroK8s with microk8s stop; microk8s start.
     
     
     
   
   

1. The IP range pods get their IPs from. By default this is set to 10.1.0.0/16. To change this IP range you need to:
   
   
   
   • Edit /var/snap/microk8s/current/args/kube-proxy and update the --cluster-cidr=10.1.0.0/16 argument.
   
   
   • If you are also setting up a proxy update /var/snap/microk8s/current/args/containerd-env with the respective IP ranges
   
   
   • Restart MicroK8s with microk8s stop; microk8s start.
   
   
   • Edit /var/snap/microk8s/current/args/cni-network/cni.yaml and replace the new IP range in.
     
     - name: CALICO_IPV4POOL_CIDR value: "10.1.0.0/16"
   
   
   • Apply the above yaml with microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml.
   
   

[baritono]

@ktsakalozos I tried your solution for 1.19 mentioned above, but there are still some issues:

1. There is a line in /var/snap/microk8s/current/args/cni-network/calico-kubeconfig that says "server: https://[10.152.183.1]:443", and even if I edit it with sudo, it would be restored to its original content automatically after I restart microk8s. Does that matter?
2. I found reference to "10.1.0.0/16" and "10.152.183.0/24" in /var/snap/microk8s/current/args/containerd-env. Should I also update that?
3. I found reference to "10.1.0.0/16" in /var/snap/microk8s/current/args/flannel-network-mgr-config. Should I also update that?

Thanks a lot!

[ktsakalozos]

@baritono I revised and tested the the instructions in the above comments for the service range. Please have another look. To your questions:

• There is a line in /var/snap/microk8s/current/args/cni-network/calico-kubeconfig that says "server: https://[10.152.183.1]:443", and even if I edit it with sudo, it would be restored to its original content automatically after I restart microk8s. Does that matter?

This is now covered y the revised version of the instructions above.

• I found reference to "10.1.0.0/16" and "10.152.183.0/24" in /var/snap/microk8s/current/args/containerd-env. Should I also update that?

If you are using a proxy you should update this file accordingly.

• I found reference to "10.1.0.0/16" in /var/snap/microk8s/current/args/flannel-network-mgr-config. Should I also update that?

Flannel is not used in 1.19 anymore. It is here only for backwards compatibility with the non-HA setup.

[baritono]

@ktsakalozos thank you so much! Now microk8s is up and running, and can happily co-exist with my Cisco VPN.

Some follow-up questions:

1. Now when I microk8s enable dashboard then microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443, I cannot access the dashboard at http://localhost:10443. Got the following error

$ microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443
Forwarding from 127.0.0.1:10443 -> 8443
Forwarding from [::1]:10443 -> 8443
Handling connection for 10443
Handling connection for 10443
E0910 12:32:18.674538   29339 portforward.go:400] an error occurred forwarding 10443 -> 8443: error forwarding port 8443 to pod c20523e09aa81f81d8448079efeb79369bc89d9d47b1ecaf29db9126d5544f67, uid : failed to execute portforward in network namespace "/var/run/netns/cni-371b6a26-e533-5db1-0c43-a6bdcaa84643": socat command returns error: exit status 1, stderr: "2020/09/10 12:32:18 socat[29527] E connect(5, AF=2 127.0.0.1:8443, 16): Connection refused\n"
E0910 12:32:18.675504   29339 portforward.go:400] an error occurred forwarding 10443 -> 8443: error forwarding port 8443 to pod c20523e09aa81f81d8448079efeb79369bc89d9d47b1ecaf29db9126d5544f67, uid : failed to execute portforward in network namespace "/var/run/netns/cni-371b6a26-e533-5db1-0c43-a6bdcaa84643": socat command returns error: exit status 1, stderr: "2020/09/10 12:32:18 socat[29528] E connect(5, AF=2 127.0.0.1:8443, 16): Connection refused\n"

1. My main goal is to run kubeflow on microk8s, if I microk8s enable kubeflow, it seems to depend on the dns addon. Since I do not want to enable the packaged dns addon, what's the recommended way of enabling kubeflow in this setting? Should I follow the generic deploying kubeflow on existing kubernetes cluster instructions here?

Thank you again!

[baritono]

@ktsakalozos since dashboard was not working, I disabled it microk8s disable dashboard and restarted microk8s microk8s stop ; microk8s start.

Now the pods are not healthy. For example, log from pod calico-kube-controllers-847c8c99d-dg4rj in deployment calico-kube-controllers (namespace kube-system)

2020-09-10 20:43:02.341 [INFO][1] main.go 88: Loaded configuration from environment config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m", CompactionPeriod:"10m", EnabledControllers:"node", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", HealthEnabled:true, SyncNodeLabels:true, DatastoreType:"kubernetes"}
W0910 20:43:02.343059       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-09-10 20:43:02.343 [INFO][1] main.go 109: Ensuring Calico datastore is initialized
2020-09-10 20:43:05.399 [ERROR][1] client.go 255: Error getting cluster information config ClusterInformation="default" error=Get https://192.168.64.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 192.168.64.1:443: connect: no route to host
2020-09-10 20:43:05.399 [FATAL][1] main.go 114: Failed to initialize Calico datastore error=Get https://192.168.64.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 192.168.64.1:443: connect: no route to host

[ktsakalozos]

@baritono could you attach the microk8s inspect tarball?

BTW who is @ keshavdv ?

[baritono]

Sorry, @ktsakalozos I misspelled your ID! Auto-completion somehow gave me @ keshavdv .

$ microk8s inspect
[sudo] password for haosong: 
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

WARNING:  Docker is installed. 
File "/etc/docker/daemon.json" does not exist. 
You should create it and add the following lines: 
{
    "insecure-registries" : ["localhost:32000"] 
}
and then restart docker with: sudo systemctl restart docker
Building the report tarball
  Report tarball is at /var/snap/microk8s/1668/inspection-report-20200911_124340.tar.gz

inspection-report-20200911_124340.tar.gz

[Bessonov]

Wow, that's a lot of steps! Is there any movement to make it configurable (and working?). If it's a not on the roadmap, then may be set better defaults can resolve most of use cases? It seems like 10.1.1/24 is often leads to a conflict. And I'm wonder that /24 is enough, especially for "production-grade Kubernetes". Why not just use 10.152.0.0/16 and 10.153.0.0/16 as default? Or I like /12. Or look at other project defaults like rancher

Don't forget to upvote the first post to reflect the need of an easy configuration.