K3s: After first start without `--no-deploy=local-storage` flag, subsequent starts with the flag still deploy local storage

[sgielen]

Version:
k3s version v0.11.0-alpha2 (405f85aa)
root 1917 118 5.3 355120 203940 ? Ssl 09:24 0:07 /sbin/k3s server --no-deploy=servicelb --no-deploy=local-storage

Describe the bug
After adding --no-deploy=local-storage, I expect no local-path provisioner to be running, but it is and the service class is registered.

To Reproduce

• Have an existing installation of k3s with local-storage.
• Provide my own storage class and delete the local-path storage class.
• Add --no-deploy=local-storage to the flags of k3s and restart.

(See https://github.com/rancher/k3s/issues/1048#issuecomment-553864526 for more information and screenshots.)

Expected behavior

• No local-path service class is created and no local-path-provisioner is running as they are disabled.

Actual behavior

• The local-path service class is recreated and the local-path-provisioner is running.

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
[...]
kube-system   local-path-provisioner-58fb86bdfd-487rl   1/1     Running     1          45h
default       nfs-client-provisioner-6b9fb56458-nmbvj   1/1     Running     1          22h
[...]
$ kubectl get sc
NAME                            PROVISIONER                   AGE
managed-nfs-storage (default)   k8s.sjor.sg/nfs-provisioner   22h
local-path (default)            rancher.io/local-path         3m13s

[sgielen]

I also added --no-deploy=servicelb which did cause the svclb not to be deployed after a reboot. Then, I added --no-deploy=traefik but traefik is actually deployed after reboot... Weird?

[davidnuzik]

What does the k3s service look like? How did you install k3s exactly?

[galal-hussein]

@sgielen I cant reproduce the issue, I started k3s with --no-deploy local-storage and I can see that neither the storage class nor the local path provisioner deployment got deployed

CC @davidnuzik

[galal-hussein]

@sgielen can you describe exactly how did you start k3s with the --no-deploy, if you started without --no-deloy and then tried to add it after it won't delete the deployment that already has been installed

[sgielen]

I have these steps to reproduce in the task description:

• Already have k3s running without any --no-deploy flags
• Delete the storage class (and, probably irrelevant, provide a default alternative)
• Add the --no-deploy=local-storage flag and restart k3s
• The local storage provisioner is still started and the storage class recreated

@galal-hussein this sounds reasonable, that the deployment isn't automatically removed – but it did trigger me that when you add --no-deploy=servicelb, the svclb is automatically removed. Is there a difference between how local-storage and servicelb are deployed that would explain this difference? Perhaps this is something to make consistent, or else, something to document?

[sgielen]

I set up k3s in an x86_64 VM with default args (just k3s server) and svclb started:

Then, added --no-deploy=servicelb to /k3os/system/config.yaml and rebooted. Initially, svclb was running but then (without me doing anything) it was set to Terminating, then removed:

Then, I also added --no-deploy=local-storage to /k3os/system/config.yaml and rebooted again, but the local path provisioner does not auto-terminate like the svclb did:

Fwiw, the config.yaml looks like this at this point:

k3os:
  password: rancher
  k3s_args:
  - server
  - --no-deploy=servicelb
  - --no-deploy=local-storage
sshAuthorizedKeys:
- github:sgielen

I think at least this difference could be better documented, but one could argue that the principle of least surprise says that k3s should comply with its --no-deploy flags regardless of whether they are given at first start or subsequent starts.

[sgielen]

Worse, I removed the local path provisioner and the storage class and rebooted and they come back even though --no-deploy=local-storage is given:

(reboot)

The same behaviour is observed with --no-deploy=servicelb,local-storage

[sgielen]

Looks like this issue is caused by the file /var/lib/rancher/k3s/server/manifests/local-storage.yaml which exists even after --no-deploy is added. (Related issue: #717) Indeed, after removing the deployment and storage class and removing the file and adding --no-deploy=local-storage, the local path provisioner is not recreated. It's not obvious to me why this is the case for local-storage but not for servicelb.

I would consider k3s' behaviour less surprising if it would not load the local-storage.yaml manifest (and perhaps even kubectl delete whatever it added before) if --no-deploy=local-storage is given? To me, as an end user, the automatic creation of this yaml is perhaps more of an implementation detail than something I normally expect to have to know about in order to change the setup of my cluster...

[Ehco1996]

i found this too
i want a clean version that do NOT contain any pre install component such as traefik or local-path-storage

---

also the cli args such as --no-deploy traefik is so strange, as a user, i dont't need no-deploy
i need deploy-xxx cmd

[jonfairbanks]

also the cli args such as --no-deploy traefik is so strange, as a user, i dont't need no-deploy
i need deploy-xxx cmd

+1 to this request. Opt in would be preferred. I suppose that would make it less seamless for new users coming in though.

[casidiablo]

Yeah, seems like deleting /var/lib/rancher/k3s/server/manifests/traefik.yaml also fixes the problem from traefik

[casidiablo]

Actually, this is happening again even without /manifests/ file in my server

[cjellick]

We introduced a --disable flag to accomplish what is being asked here.