Velero: Can`t create Rook PVC backup with Restic

[ivaravko]

What steps did you take and what happened:

The Pod with Rook PVC annotated with backup.velero.io

• kubectl -nmonitoring annotate pod/monitoring-grafana-588bf894f4-rvg4l backup.velero.io/backup-volumes=storage-volume

Executed the following commands

• velero backup create grafana-01 --include-namespaces=monitoring --snapshot-volumes=false --ttl 5h
• velero backup describe grafana-01 --details

As a result, an error

Name:         grafana-01
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  InProgress

Namespaces:
  Included:  monitoring
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  false

TTL:  5h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    <n/a>
Completed:  <n/a>

Expiration:  2019-05-08 15:33:57 +0200 CEST

Validation errors:  <none>

Persistent Volumes: <none included>

Restic Backups:
  Failed:
    monitoring/monitoring-grafana-588bf894f4-rvg4l: storage-volume

What did you expect to happen:
Expected to be successfully backup of PVC storage-volume.

The output of the following commands will help us better understand what's going on:

• kubectl logs deployment/velero -n velero
  Link to logs https://gist.github.com/ivaravko/7c262ac22cd8f2c88a1c6c4f07e49e12
• kubectl -nvelero logs restic-qfbhq

time="2019-05-08T08:34:04Z" level=info msg="Backup starting" backup=velero/grafana-01 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:171" name=grafana-01-p55bd namespace=velero
time="2019-05-08T08:34:07Z" level=error msg="Error running command=restic backup --repo=s3:https://s3.ap.cloud-object-storage.appdomain.cloud/rpg-02-seo01-backup/restic/monitoring --password-file=/tmp/velero-restic-credentials-monitoring569539951 --cache-dir=/scratch/.cache/restic . --tag=pod=monitoring-grafana-588bf894f4-rvg4l --tag=pod-uid=c0cfd4aa-70eb-11e9-a0ae-0cc47abafaba --tag=volume=storage-volume --tag=backup=grafana-01 --tag=backup-uid=05aff93c-716c-11e9-8721-0cc47a889cd2 --tag=ns=monitoring --host=velero, stdout=open repository\n, stderr=Fatal: unable to save snapshot: snapshot is empty\n" backup=velero/grafana-01 controller=pod-volume-backup error="exit status 1" error.file="pod_volume_backup_controller.go:232" error.function="(*podVolumeBackupController).processBackup" logSource="pkg/controller/pod_volume_backup_controller.go:232" name=grafana-01-p55bd namespace=velero

Environment:

• Velero version (use velero version): 0.11.0
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.12", GitCommit:"c757b93cf034d49af3a3b8ecee3b9639a7a11df7", GitTreeState:"clean", BuildDate:"2018-12-19T11:16:52Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: kubespray 2.6.0
• Cloud provider or hardware configuration: Bare Metal Server
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Rook version: 0.8.3

[ivaravko]

@skriss yes, the data is there

    Mounts:
      /etc/grafana from config-volume (rw)
      /var/lib/grafana from storage-volume (rw)
      /var/lib/grafana/dashboards from dashboard-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2rczg (ro)

  storage-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  monitoring-grafana
    ReadOnly:   false



md5-6571f35b4a3a82768be43f16a383a74b



$ df -h
Filesystem      Size  Used Avail Use% Mounted on
none            916G   37G  832G   5% /
tmpfs            16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/sda6       916G   37G  832G   5% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M   45M  865M   5% /var/lib/grafana
tmpfs            16G   12K   16G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            16G     0   16G   0% /sys/firmware



md5-6571f35b4a3a82768be43f16a383a74b



$ ls -la /var/lib/grafana
total 832
drwxrwsr-x  6     104 grafana   4096 Jan 31 16:18 .
drwxr-xr-x 11 root    root      4096 Aug 22  2018 ..
drwxrwsrwx  2 root    grafana   4096 May  8 12:39 dashboards
drwxrwsr-x  4 grafana grafana   4096 May  8 15:53 data
drwxrwS---  2 grafana grafana  16384 Jun 18  2018 lost+found
drwxrwsr-x  6 grafana grafana   4096 Apr 12 12:41 plugins

[skriss]

A few questions:

• Is mount propagation enabled in your cluster? (I believe it was a beta feature in 1.10, it's required for restic to work)
• Can you provide the YAML for both the PV and PVC?
• Can you find the restic daemonset pod that's running on the same node as monitoring-grafana-588bf894f4-rvg4l, exec into it, and go into /host_pods/<your-pod's-uid>/..., see if the storage-volume is in there, and if so, if you see data in it?

[ivaravko]

@skriss

• MountPropagation is enabled by default from 1.10 accordingly https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
• kubectl -nmonitoring get pvc monitoring-grafana -o yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"355e1514-64a2-11e8-86ba-2ab0dcbd0d87","leaseDurationSeconds":15,"acquireTime":"2018-06-18T14:04:03Z","renewTime":"2018-06-18T14:04:05Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rook.io/block
  creationTimestamp: 2018-06-18T14:04:02Z
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: monitoring-grafana
    chart: grafana-0.8.5
    component: grafana
    heritage: Tiller
    release: monitoring
  name: monitoring-grafana
  namespace: monitoring
  resourceVersion: "3284716"
  selfLink: /api/v1/namespaces/monitoring/persistentvolumeclaims/monitoring-grafana
  uid: 749fa353-7300-11e8-aa94-0cc47a8f861a
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-block
  volumeName: pvc-749fa353-7300-11e8-aa94-0cc47a8f861a
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

• kubectl get pv pvc-749fa353-7300-11e8-aa94-0cc47a8f861a -o yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rook.io/block
  creationTimestamp: 2018-06-18T14:04:03Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-749fa353-7300-11e8-aa94-0cc47a8f861a
  resourceVersion: "5612365"
  selfLink: /api/v1/persistentvolumes/pvc-749fa353-7300-11e8-aa94-0cc47a8f861a
  uid: 74f49d25-7300-11e8-82ca-0cc47abafaba
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: monitoring-grafana
    namespace: monitoring
    resourceVersion: "3284605"
    uid: 749fa353-7300-11e8-aa94-0cc47a8f861a
  flexVolume:
    driver: rook.io/rook-system
    options:
      image: pvc-749fa353-7300-11e8-aa94-0cc47a8f861a
      pool: replicapool
      storageClass: rook-block
  persistentVolumeReclaimPolicy: Retain
  storageClassName: rook-block
status:
  phase: Bound

• kubectl -nmonitoring get po monitoring-grafana-796f54bf74-tt9lh -o yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    backup.velero.io/backup-volumes: storage-volume
    checksum/config: 747985b0b682fea047242b96cfce57159513f0b608d2e2452d12cbb7f6af1f63
    checksum/dashboards-config: 618eac5fa709486c58049ac002824fabcb2b253b0003ca71580c9f77b10be923
  creationTimestamp: 2019-05-08T12:39:53Z
  generateName: monitoring-grafana-796f54bf74-
  labels:
    app: grafana
    component: grafana
    pod-template-hash: "3529106930"
    release: monitoring
  name: monitoring-grafana-796f54bf74-tt9lh
  namespace: monitoring
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: monitoring-grafana-796f54bf74
    uid: 63a6ba52-2579-11e9-9428-0cc47a8f861a
  resourceVersion: "71471739"
  selfLink: /api/v1/namespaces/monitoring/pods/monitoring-grafana-796f54bf74-tt9lh
  uid: 61031cfa-718e-11e9-a0ae-0cc47abafaba
spec:

• List data in restic pod

kubectl -nvelero exec -it restic-qfbhq sh
/ # cd host_pods/61031cfa-718e-11e9-a0ae-0cc47abafaba

/host_pods/61031cfa-718e-11e9-a0ae-0cc47abafaba # ls -l -R
.:
total 16
drwxr-x---    4 root     root          4096 May  8 12:40 containers
-rw-r--r--    1 root     root           234 May  8 12:40 etc-hosts
drwxr-x---    3 root     root          4096 May  8 12:39 plugins
drwxr-x---    6 root     root          4096 May  8 12:39 volumes

./containers:
total 8
drwxr-x---    2 root     root          4096 May  8 12:40 copy-configs
drwxr-x---    2 root     root          4096 May  8 12:40 grafana

./containers/copy-configs:
total 0
-rw-rw-rw-    1 root     root             0 May  8 12:40 cc7742f0

./containers/grafana:
total 0
-rw-rw-rw-    1 root     root             0 May  8 12:40 0dc139d3
-rw-rw-rw-    1 root     root             0 May  8 12:40 fec27543

./plugins:
total 4
drwxr-x---    7 root     root          4096 May  8 12:39 kubernetes.io~empty-dir

./plugins/kubernetes.io~empty-dir:
total 20
drwxr-x---    2 root     root          4096 May  8 12:39 config-volume
drwxr-x---    2 root     root          4096 May  8 12:39 dashboard-volume
drwxr-x---    2 root     root          4096 May  8 12:39 wrapped_config-volume-configmap
drwxr-x---    2 root     root          4096 May  8 12:39 wrapped_dashboard-volume-configmap
drwxr-x---    2 root     root          4096 May  8 12:39 wrapped_default-token-2rczg

./plugins/kubernetes.io~empty-dir/config-volume:
total 0
-rw-r--r--    1 root     root             0 May  8 12:39 ready

./plugins/kubernetes.io~empty-dir/dashboard-volume:
total 0
-rw-r--r--    1 root     root             0 May  8 12:39 ready

./plugins/kubernetes.io~empty-dir/wrapped_config-volume-configmap:
total 0
-rw-r--r--    1 root     root             0 May  8 12:39 ready

./plugins/kubernetes.io~empty-dir/wrapped_dashboard-volume-configmap:
total 0
-rw-r--r--    1 root     root             0 May  8 12:39 ready

./plugins/kubernetes.io~empty-dir/wrapped_default-token-2rczg:
total 0
-rw-r--r--    1 root     root             0 May  8 12:39 ready

./volumes:
total 16
drwxr-xr-x    4 root     root          4096 May  8 12:39 kubernetes.io~configmap
drwxr-xr-x    4 root     root          4096 May  8 12:39 kubernetes.io~empty-dir
drwxr-xr-x    3 root     root          4096 May  8 12:39 kubernetes.io~secret
drwxr-x---    3 root     root          4096 May  8 12:39 rook.io~rook-system

./volumes/kubernetes.io~configmap:
total 8
drwxrwsrwx    3 root     472           4096 May  8 12:39 config-volume-configmap
drwxrwsrwx    3 root     472           4096 May  8 12:39 dashboard-volume-configmap

./volumes/kubernetes.io~configmap/config-volume-configmap:
total 0
lrwxrwxrwx    1 root     root            30 May  8 12:39 grafana-install-plugins -> ..data/grafana-install-plugins
lrwxrwxrwx    1 root     root            18 May  8 12:39 grafana.ini -> ..data/grafana.ini
lrwxrwxrwx    1 root     root            38 May  8 12:39 grafanaAuthGoogleAllowedDomains -> ..data/grafanaAuthGoogleAllowedDomains
lrwxrwxrwx    1 root     root            30 May  8 12:39 grafanaAuthGoogleSignUp -> ..data/grafanaAuthGoogleSignUp

./volumes/kubernetes.io~configmap/dashboard-volume-configmap:
total 0

./volumes/kubernetes.io~empty-dir:
total 8
drwxrwsrwx    2 root     472           4096 May  8 12:40 config-volume
drwxrwsrwx    2 root     472           4096 May  8 12:39 dashboard-volume

./volumes/kubernetes.io~empty-dir/config-volume:
total 16
-rw-r--r--    1 472      472             23 May  8 12:40 grafana-install-plugins
-rw-r--r--    1 472      472            549 May  8 12:40 grafana.ini
-rw-r--r--    1 472      472             14 May  8 12:40 grafanaAuthGoogleAllowedDomains
-rw-r--r--    1 472      472              4 May  8 12:40 grafanaAuthGoogleSignUp

./volumes/kubernetes.io~empty-dir/dashboard-volume:
total 0

./volumes/kubernetes.io~secret:
total 4
drwxrwxrwx    2 root     root          4096 May  8 12:39 default-token-2rczg

./volumes/kubernetes.io~secret/default-token-2rczg:
total 0

./volumes/rook.io~rook-system:
total 4
drwxr-x---    2 root     root          4096 May  8 12:39 pvc-749fa353-7300-11e8-aa94-0cc47a8f861a

./volumes/rook.io~rook-system/pvc-749fa353-7300-11e8-aa94-0cc47a8f861a:
total 0

I have not found in ./volumes something like that kubernetes.io~rook-pvc with folder storage-volume. But there is present folders for other volumes from pod

volumes:
  - emptyDir: {}
    name: config-volume
  - emptyDir: {}
    name: dashboard-volume
  - configMap:
      defaultMode: 420
      name: monitoring-grafana-config
    name: config-volume-configmap
  - configMap:
      defaultMode: 420
      name: monitoring-grafana-dashs
    name: dashboard-volume-configmap
  - name: storage-volume
    persistentVolumeClaim:
      claimName: monitoring-grafana
  - name: default-token-2rczg
    secret:
      defaultMode: 420
      secretName: default-token-2rczg

[skriss]

Hmm, so:

./volumes/rook.io~rook-system/pvc-749fa353-7300-11e8-aa94-0cc47a8f861a:
total 0

That corresponds to the PVC in question, and I'd expect to see files in there.

Is it possible for you to SSH to that node, and look from there at what appears in /var/lib/kubelet/pods/61031cfa-718e-11e9-a0ae-0cc47abafaba/volumes/rook.io~rook-system/pvc-749fa353-7300-11e8-aa94-0cc47a8f861a?

[ivaravko]

@skriss I found root case - two PV for claim monitoring-grafana in the cluster

pvc-709632d1-6fbb-11e8-aa94-0cc47a8f861a   1Gi        RWO            Retain           Released   monitoring/monitoring-grafana                                 rook-block               328d
pvc-749fa353-7300-11e8-aa94-0cc47a8f861a   1Gi        RWO            Retain           Bound      monitoring/monitoring-grafana                                 rook-block               324d

Second have state Released and not used. After removing it, I was able to make an backup.

[skriss]

@ivaravko I'm closing this as resolved, feel free to reopen if needed!