Microk8s: Failed to disable prometheus: error: the path "/var/snap/microk8s/1668/kube-prometheus/manifests/" does not exist

[Aaron-Ritter]

after upgrading microk8s prometheus somehow is not possible to disable anymore

$ microk8s disable prometheus
Disabling old Prometheus
Disabling Prometheus
error: the path "/var/snap/microk8s/1668/kube-prometheus/manifests/" does not exist
Failed to disable prometheus

[balchua]

Thanks for reporting this issue.
There's a bug in the disable prometheus command.
QQ. Do you previously have prometheus installed prior to upgrade? If yes, previous prometheus components must have been removed.

[Aaron-Ritter]

correct it was installed prior to upgrade and then got upgraded, automatically through snap, so there wouldn't have been a chance for me to do so manually.

[balchua]

Thanks for confirming, at this point your pre microk8s upgrade prometheus should no longer be there eventhough the "disable" failed.

[Aaron-Ritter]

Thanks, Thats what i thought (btw i fixed the version on to one major version as a first precaution, not sure what other precautions i could implement):

1. shouldnt the snap refresh somehow cover this scneario e.g. disable enable prometheus during the upgrade procedure?
2. can i somehow fix this in the current state without having to completely wipe the installation? e.g. somehow create the directory and content manually to then disable the service or force disable it?

In addition, is there anywhere a documentation referring to (you have to remove prometheus bevore the upgrade) or is microk8s simply going with the caution statement to always reinstall between major versions?

maybe it would be good to add this directly in the getting started section? not only in the upgrade section of the documentation:

Caution: Changing the channel to a different series will almost certainly cause problems...

[balchua]

1. shouldnt the snap refresh somehow cover this scneario e.g. disable enable prometheus during the upgrade procedure?

Nope. MicroK8s will not touch the enabled addons. Although we do upgrade the addons in each release. To upgrade the addons, most often one should disable then enable.

1. can i somehow fix this in the current state without having to completely wipe the installation? e.g. somehow create the directory and content manually to then disable the service or force disable it?

Can you check if there are prometheus pods running? If no, you can do microk8s enable prometheus. The next time you do disable, it shouldn't fail because the manifests will exist due to the enable command you did.

In addition, is there anywhere a documentation referring to (you have to remove prometheus bevore the upgrade) or is microk8s simply going with the caution statement to always reinstall between major versions?

The recommendation is if you are running a long lasting cluster is to stick to a particular channel, say 1.18/stable. Moving from one minor kubernetes version to another do introduce breaking changes in upstream kubernetes or MicroK8s. In a particular channel we do not usually upgrade addons, unless there are fixes.

maybe it would be good to add this directly in the getting started section?

That's a good idea.

Thanks for the feedback and using MicroK8s.

[Aaron-Ritter]

Thanks @balchua

the part about enabling when prometeus is not running did actually point me in the right direction.

I looked for all deployed prometheus ressources
$ microk8s kubectl get all --all-namespaces | grep promet

and deleted them accordingly, disabled as well the dashboard and metrics-server via microk8s.disable
$ microk8s kubectl delete --namespace monitoring

and activated everything accordingly

[balchua]

@Aaron-Ritter thanks. I need to get this one fixed. Apologies for the troubles.