Loki: Helm: Loki stuck in ContainerCreating stage due to PVC

[agolomoodysaada]

Describe the bug
When using helm upgrade using PVC, Loki stays stuck as ContainerCreating stage due to Unable to mount volumes for pod.

To Reproduce
Steps to reproduce the behavior:

1. Helm install Loki
2. Helm upgrade Loki with any change to config or resources
3. run kubectl get pods and never get results

loki-5f9d8554f-wqgmk   1/1     Running             0          1h
loki-d694cfc6-ccfvf    0/1     ContainerCreating   0          1h

Describing the stuck pod shows

  Warning  FailedMount  3s (x45 over 100m)  kubelet, gke-crawlers-pool-1-2005c642-8ks9  Unable to mount volumes for pod "loki-d694cfc6-ccfvf_loki(e44ce3e3-7326-11e9-a5cf-42010a8e0212)": timeout expired waiting for volumes to attach or mount for pod "loki"/"loki-d694cfc6-ccfvf". list of unmounted volumes=[storage]. list of unattached volumes=[config storage loki-token-dpkvn]

Should we use StatefulSet instead of Deployment types since they're better at mounting and unmounting and general management of PVCs?

Expected behavior

I expect the new release to just work

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

[agolomoodysaada]

As a workaround, I had to delete the old replicaset (not even the deployment).

[steven-sheehy]

Yes, it should probably be a StatefulSet. I can take a look at doing so. It will be a breaking change requiring manual intervention though. Meanwhile you can set loki.deploymentStrategy=Recreate.