K3s: Questions regarding k3s use in ARM and IoT for production use case.

[uGiFarukh]

Hello,

I have couple of questions regarding k3s and its use case in small ARM based IoT devices with <512MB RAM and 1CPU Core.

I am planning to deploy a production kubernetes cluster using k3s for IoT devices and some web services to host IoT APIs.

Question 1: Is k3s suitable for production use for this particular use case?

Question 2: How does it compare to docker swarm in-terms of memory footprint and overall resource utilization under same scenarios?

Question 3: Will k3s work perfectly fine on ARMv6 architecture of the Raspberry Pi Zero? What about A7 Cortex ARM CPU's?

Question 4: Should I use any kind of swap for these k3s nodes or should I totally stay away from swap? What about swaps with lower swappiness level and maybe lower cache pressure? Just for exception cases so that I don't have out of memory exceptions on those small IoT Pi Zeros.

Question 5: I want to deploy the master node on AWS EC2 t2.micro instance. Which has 1vCPU and 1GB RAM. I want to use this master server node only as the orchestrator or scheduler and maybe have some proxy networking and metrics services over this node. Other than that, this node will not host any of my actual services. So, how should I go about deploying this master node? Should I go the master only approach? Where this node is not part of the worker nodes cluster? Or should I deploy it normally using the default installation process?

Question 6: Will I be able to deploy the kubernetes dashboard on this master node and use it to manage everything? Or will it be resource heavy for this EC2 instance? and if I can, how do I deploy the dashboard or any kubernetes upstream services in k3s?

Question 7: Can I get an explanation of the services that are deployed by default?

`ubuntu@ip-172-31-16-191:~$ sudo kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/local-path-provisioner-6d59f47c7-v2rvv 1/1 Running 0 7m57s
kube-system pod/metrics-server-7566d596c8-xgrz7 1/1 Running 0 7m57s
kube-system pod/helm-install-traefik-skx9f 0/1 Completed 0 7m57s
kube-system pod/coredns-7944c66d8d-4wzfd 1/1 Running 0 7m57s
kube-system pod/svclb-traefik-r66dh 2/2 Running 0 7m30s
kube-system pod/traefik-758cd5fc85-xq2f2 1/1 Running 0 7m30s

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 443/TCP 8m13s
kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 8m10s
kube-system service/metrics-server ClusterIP 10.43.251.195 443/TCP 8m8s
kube-system service/traefik-prometheus ClusterIP 10.43.36.233 9100/TCP 7m30s
kube-system service/traefik LoadBalancer 10.43.74.93 172.31.16.191 80:31781/TCP,443:30121/TCP 7m30s

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/svclb-traefik 1 1 1 1 1 7m30s

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/local-path-provisioner 1/1 1 1 8m10s
kube-system deployment.apps/metrics-server 1/1 1 1 8m9s
kube-system deployment.apps/coredns 1/1 1 1 8m11s
kube-system deployment.apps/traefik 1/1 1 1 7m30s

NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/local-path-provisioner-6d59f47c7 1 1 1 7m57s
kube-system replicaset.apps/metrics-server-7566d596c8 1 1 1 7m57s
kube-system replicaset.apps/coredns-7944c66d8d 1 1 1 7m57s
kube-system replicaset.apps/traefik-758cd5fc85 1 1 1 7m30s

NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik 1/1 28s 8m8s`

Want to know about each of them and what exactly are they doing in terms of the kubernetes cluster.

Question 8: Is it possible for me to strip down some of the default services which I don't need and get lesser memory footprint of the k3s by doing so? Also, what about if I use an external MySQL database instead of the sqlite to get HA. Will that reduce the resource usage on the master node? Will it be significant or minor?

Question 9: What are the other ways, I can strip down the k3s to fit exactly my use case and get overall less resource utilization out of the system?

Question 10: How can I change the default ClusterIP Pool Range and Pods IP Range of k3s to a different local ip range properly without breaking anything? Is it possible to do so during installation process? What is the method to achieve this?

Question 11: When will kubernetes v1.19 come to k3s? Any ETA?

Question 12: Basically want to know everything I can do to strip down k3s even more to suit my particular use case in both the server and agent deployments. Any suggestions regarding that would be really helpful.

Question 13: Is there any initial steps I should follow to secure k3s more before going to production with the cluster setup? Anything on the agent or server part? BTW, the agent server connection automatically uses TLS right? Any tweaks for security?

Thank You

[uGiFarukh]

Anyone?

[cjellick]

big list! I'll try to get these answered soon. Thanks :-)