Victoriametrics: Kubernetes: second storage unit stuck in Unready Status

[Serrvosky]

Hello everyone,

First of all, congrats for this amazing project. Until now, it was the most simplest and cheapest implementation that I found to implement and work with HA monitoring. Very well documented as well.

However I'm having a problem spining the second storage instance. I implemented the cluster version of VictoriaMetrics in Kubernetes, with one instance of each segment (storage, insert and select). Now I'm trying to spin up the second instance of storage to check if all works properly, but this second one it's stuck in unready status.

This is my log line from kubectl command:

Error from server (BadRequest): container "vitoria-storage" in pod "vitoria-storage-statefulset-1" is waiting to start: ContainerCreating

I'm using rancher as well to managing k8's clusters, and there, I can see this error:
Containers with unready status: [vitoria-storage]

This is my storage yaml file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vitoria-storage-statefulset
  labels:
    app: vitoria-storage
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vitoria-storage
  serviceName: vitoria-storage
  template:
    metadata:
      labels:
        app: vitoria-storage
        name: vitoria-storage
    spec:
      containers:
      - name: vitoria-storage
        image: victoriametrics/vmstorage:latest
        imagePullPolicy: IfNotPresent
        args:
        - -retentionPeriod=1
        - -storageDataPath=/data/metrics
        volumeMounts:
          - name: data
            mountPath: /data/metrics
        ports:
        - name: insert-port
          protocol: TCP
          containerPort: 8400
        - name: select-port
          protocol: TCP
          containerPort: 8401
        - name: http
          protocol: TCP
          containerPort: 8482
        readinessProbe:
          httpGet:
            path: /health
            port: http
          initialDelaySeconds: 5
          periodSeconds: 15
          timeoutSeconds: 5
        livenessProbe:
          tcpSocket:
            port: http
          initialDelaySeconds: 5
          periodSeconds: 15
          timeoutSeconds: 5
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

Any help with this problem? Thanks

[tenmozes]

Hello @Serrvosky could you run kubectl describe po vitoria-storage-statefulset-1 and check the status of the pod, and kubectl get pvc as well?

[Serrvosky]

Humm... I see. The problem it's with the volumes.

> kubectl --kubeconfig=/home/user/Downloads/do-moni-kubeconfig.yaml describe po vitoria-storage-statefulset-1
...
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vitoria-storage-statefulset-1
    ReadOnly:   false
  default-token-bnbkp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bnbkp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason              Age                    From                         Message
  ----     ------              ----                   ----                         -------
  Warning  FailedAttachVolume  4m31s (x489 over 16h)  attachdetach-controller      AttachVolume.Attach failed for volume "pvc-63d28cb0-a160-11e9-b5f5-5681ebed6a42" : rpc error: code = NotFound desc = volume "64c299f9-a160-11e9-8b70-0a58ac14c058" not found
  Warning  FailedMount         2m36s (x434 over 16h)  kubelet, k8s-pool-moni-orie  Unable to mount volumes for pod "vitoria-storage-statefulset-1_default(067d2982-a197-11e9-b5f5-5681ebed6a42)": timeout expired waiting for volumes to attach or mount for pod "default"/"vitoria-storage-statefulset-1". list of unmounted volumes=[data]. list of unattached volumes=[data default-token-bnbkp]

> kubectl --kubeconfig=/home/user/Downloads/do-moni-kubeconfig.yaml get pvc 

data-vitoria-storage-statefulset-0   Bound    pvc-0b721856-a160-11e9-b5f5-5681ebed6a42   10Gi       RWO            do-block-storage   22h
data-vitoria-storage-statefulset-1   Bound    pvc-63d28cb0-a160-11e9-b5f5-5681ebed6a42   10Gi       RWO            do-block-storage   22h
data-vitoria-storage-statefulset-2   Bound    pvc-70a48984-a160-11e9-b5f5-5681ebed6a42   10Gi       RWO            do-block-storage   22h

I deleted them in the cloud provider dashboard, I pod can't get access to it. Just had to delete them from pvc list and redeploy pod again.

Thanks for your help guys.

[tenmozes]

you're welcome, if you need any further help feel free to reopen ticket