Velero: Restic case study: Mongo diff snapshots take longer than init snapshot

[agolomoodysaada]

What steps did you take and what happened:
Hey Velero team,

I’m using the new v1.0.0-alpha.2 with Restic with GCS (bucket and VM in the same region) and I’m having trouble wrapping my head around the performance. Everytime I run incremental snapshots for MongoDB, they take longer than the initial snapshot.

Annotations on pod

kubectl annotate pod mongodb-mongodb-replicaset-2 \
    --overwrite \
    pre.hook.backup.velero.io/command='["/bin/sh", "-c", "mongo --eval \"rs.stepDown()\" ; mongo --eval \"db.fsyncLock()\" "]' \
    pre.hook.backup.velero.io/container=mongodb-replicaset \
    post.hook.backup.velero.io/command='["/bin/sh", "-c", "mongo --eval \"db.fsyncUnlock()\" "]' \
    post.hook.backup.velero.io/container=mongodb-replicaset \
    backup.velero.io/backup-volumes=configdir,datadir,workdir

velero backup create backup1 -l statefulset.kubernetes.io/pod-name=mongodb-mongodb-replicaset-2
velero backup create backup2 -l statefulset.kubernetes.io/pod-name=mongodb-mongodb-replicaset-2
...

# backup1
Started:    2019-04-24 19:16:32 -0400 EDT
Completed:  2019-04-24 19:52:10 -0400 EDT
Time taken: ~36mins
`gsutil du -h` output:  163.88G

# backup2 (diff)
Started:    2019-04-25 08:40:48 -0400 EDT
Completed:  2019-04-25 09:24:54 -0400 EDT
Time taken: ~44mins
`gsutil du -h` output: 170.48 GiB

There's only ~7G of added data in backup2. It took 45mins to transfer ~7G. I think it should take <5 mins for such a diff to be transferred based on GCS’s 60MB/s average

What did you expect to happen:

I expect the performance to be faster on the following snapshot runs since restic only transfers the diff between each snapshot. However, that’s not what I’m observing… it seems to be getting slower or remain the same.

The output of the following commands will help us better understand what's going on:

The second snapshot logs show the following which is not very useful as it doesn't show Restic specific actions.

time="2019-04-25T12:40:55Z" level=info msg="Executing takePVSnapshot"
time="2019-04-25T12:40:55Z" level=info msg="Skipping Persistent Volume snapshot because volume has already been backed up."
# post-backup hook
time="2019-04-25T13:24:52Z" level=info msg="running exec hook"
time="2019-04-25T13:24:52Z" level=info msg="stdout: ..........."

Metrics for Backup2

Bucket ingress traffic:


Seems like most of the time of the backup was spent compute diffs of modified files rather that uploading data to GCS.

CPU and Memory

We can see from the container memory metrics that the working set remains under 1G but the cache memory just eats up the OS memory without releasing it.

Exec-ing into a restic pod allowed me to confirm that snapshots are in fact incremental using:

velero restic repo get -o yaml | grep resticIdentifier
restic -r <repo-identifier> snapshots --json

I was also able to run a restic -r <repo-identifier> diff <snapshot-id-1> <snapshot-id-2> and got

Files:           4 new,     4 removed,    48 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  18151 new, 18703 removed
Tree Blobs:      2 new,     2 removed
  Added:   9.064 GiB
  Removed: 9.360 GiB

Anything else you would like to add:

Reference to findings can be found from conversations with the Velero team on slack @skriss and @nrb .

Thank you so much to the team for their help here.

Environment:

• Velero version (use velero version): v1.0.0-alpha.2
• Kubernetes version (use kubectl version): v1.11.7-gke.12
• Cloud provider or hardware configuration: beta.kubernetes.io/instance-type=n1-highmem-16
• OS (e.g. from /etc/os-release): cloud.google.com/gke-os-distribution=ubuntu

Suggested actions

• [ ] check if restic can be configured to release memory after completing a backup or limit its memory cache size
• [ ] velero install and helm chart should have sane defaults for restic daemonset pods as they eat up cache memory without releasing it. memory: 1Gi seems to be a sane default from above metrics for both requests and limits.
• [ ] document cases where restic might not be efficient at backup time such as this case study for MongoDB. Restic is slow at backup time, but still minimizes storage cost and restore performance.
• [ ] make it easy to find restic specific logs. Maybe expose a new velero restic logs command? Or make it easy to find out which restic pod, ran which backup, for which volume.

[skriss]

thanks so much for this detailed report @agolomoodysaada! really helpful.

[skriss]

@agolomoodysaada can you provide the YAML for the restic daemonset that you're using? thanks.

[agolomoodysaada]

Hey @skriss , I used velero install --use-restic ...

[skriss]

Can you provide a rough file count of this volume, and the sizes of the top ~10 files? Trying to understand if most of the data is in one large file.

[agolomoodysaada]

Restic diff

M    /WiredTiger.backup
M    /WiredTiger.turtle
M    /WiredTiger.wt
M    /collection-0--4937860834407901523.wt
M    /collection-14-4691893134753119794.wt
M    /collection-2--4937860834407901523.wt
M    /collection-29--4937860834407901523.wt
M    /collection-31--4937860834407901523.wt
M    /collection-33--4937860834407901523.wt
M    /collection-35--4937860834407901523.wt
M    /collection-4-4691893134753119794.wt
M    /collection-42--4937860834407901523.wt
M    /collection-50--4937860834407901523.wt
-    /diagnostic.data/metrics.2019-04-12T19-55-53Z-00000
M    /diagnostic.data/metrics.2019-04-24T13-47-20Z-00000
+    /diagnostic.data/metrics.2019-04-25T03-30-48Z-00000
M    /diagnostic.data/metrics.interim
M    /index-105--4937860834407901523.wt
M    /index-110--4937860834407901523.wt
M    /index-112--4937860834407901523.wt
M    /index-113--4937860834407901523.wt
M    /index-124--4937860834407901523.wt
M    /index-126--4937860834407901523.wt
M    /index-140--4937860834407901523.wt
M    /index-147--4937860834407901523.wt
M    /index-153--4937860834407901523.wt
M    /index-164--4937860834407901523.wt
M    /index-169--4937860834407901523.wt
M    /index-172--4937860834407901523.wt
M    /index-174--4937860834407901523.wt
M    /index-177--4937860834407901523.wt
M    /index-179--4937860834407901523.wt
M    /index-3--4937860834407901523.wt
M    /index-30--4937860834407901523.wt
M    /index-32--4937860834407901523.wt
M    /index-34--4937860834407901523.wt
M    /index-36--4937860834407901523.wt
M    /index-37--4937860834407901523.wt
M    /index-38--4937860834407901523.wt
M    /index-39--4937860834407901523.wt
M    /index-4--4937860834407901523.wt
M    /index-43--4937860834407901523.wt
M    /index-44--4937860834407901523.wt
M    /index-45--4937860834407901523.wt
M    /index-46--4937860834407901523.wt
M    /index-47--4937860834407901523.wt
M    /index-48--4937860834407901523.wt
M    /index-49--4937860834407901523.wt
M    /index-51--4937860834407901523.wt
-    /journal/WiredTigerLog.0000003833
-    /journal/WiredTigerLog.0000003834
+    /journal/WiredTigerLog.0000003861
+    /journal/WiredTigerLog.0000003862
-    /journal/WiredTigerPreplog.0000000070
+    /journal/WiredTigerPreplog.0000000097
M    /sizeStorer.wt

Files:           4 new,     4 removed,    48 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:  18703 new, 18151 removed
Tree Blobs:      2 new,     2 removed
  Added:   9.360 GiB
  Removed: 9.064 GiB

File sizes

total 161G
-rw-rw---- 1 mongodb mongodb 130G Apr 29 11:26 collection-31--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  17G Apr 29 11:26 collection-14-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  12G Apr 29 11:26 collection-33--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 465M Apr 29 11:26 index-124--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 440M Apr 29 11:26 index-153--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 434M Apr 29 11:26 index-169--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 422M Apr 29 11:26 index-179--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 421M Apr 29 11:26 index-174--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 246M Apr 29 11:26 index-177--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 192M Apr 29 11:26 index-172--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 179M Apr 29 11:26 index-32--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 154M Apr 29 11:26 index-126--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 149M Apr 29 11:26 index-164--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  64M Apr 29 11:26 index-112--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  59M Apr 29 11:26 index-140--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  57M Apr 29 11:26 index-147--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  47M Apr 29 11:26 collection-29--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  43M Apr 29 11:26 index-34--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  26M Apr 29 11:26 index-105--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  24M Apr 29 11:26 index-110--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  23M Apr 29 11:26 index-113--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  14M Apr 29 10:58 collection-50--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 7.6M Apr 29 10:04 collection-42--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 6.2M Apr 29 11:26 index-30--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 2.3M Apr 25 14:23 collection-52--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 1.5M Apr 29 02:34 collection-35--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 1.4M Apr 29 02:34 index-39--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 1.1M Apr 29 11:26 index-3--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 924K Apr 29 02:34 index-37--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 824K Apr 23 10:22 index-41--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 500K Apr 23 10:23 collection-40--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 500K Apr 29 02:34 index-36--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 416K Apr 29 11:26 WiredTiger.wt
-rw-rw---- 1 mongodb mongodb 312K Apr 29 11:26 collection-2--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 300K Apr 29 10:58 index-51--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 268K Apr 29 10:02 index-43--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 156K Apr 23 10:23 collection-54--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 124K Apr 25 14:23 index-53--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 116K Apr 29 02:34 index-38--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  84K Apr 27 11:53 collection-56--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  84K Apr 29 10:02 index-47--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  84K Apr 29 10:04 index-49--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  68K Apr 29 10:02 index-44--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  68K Apr 29 10:02 index-45--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  68K Apr 29 10:02 index-46--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  68K Apr 29 10:02 index-48--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  64K Apr 29 11:26 index-4--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  52K Apr 27 11:53 index-60--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  44K Apr 23 10:23 _mdb_catalog.wt
-rw-rw---- 1 mongodb mongodb  44K Apr 29 11:26 sizeStorer.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 29 11:26 collection-0--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 23 10:24 collection-0-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 23 10:23 collection-180--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 29 11:26 collection-4-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 24 23:16 collection-5--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 24 19:54 collection-69--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 24 19:56 collection-76--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 23 10:24 index-1-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 27 11:53 index-63--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  36K Apr 24 19:54 index-70--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  32K Apr 23 10:23 collection-15-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  32K Apr 23 10:23 index-16-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  32K Apr 23 10:22 index-181--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  32K Apr 23 10:23 index-182--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  24K Apr 23 10:22 index-55--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  20K Apr 23 10:24 index-66--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-17-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-2-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-6-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-71--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-73--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-78--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-8-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 collection-80--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 index-1--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr  8 20:24 index-18-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr  8 20:24 index-3-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr  8 20:24 index-5-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:24 index-57--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 26 00:01 index-59--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-6--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-68--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:23 index-7-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-72--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-74--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-75--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 11:13 index-77--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-79--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr 23 10:22 index-81--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb  16K Apr  8 20:24 index-9-4691893134753119794.wt
-rw------- 1 mongodb mongodb 4.0K Apr 23 10:23 WiredTigerLAS.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-101--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-103--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-183--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-19-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-192--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-197--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-200--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-203--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-82--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-84--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-86--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-88--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-90--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-92--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-94--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-96--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:23 collection-98--4937860834407901523.wt
drwxrwS--- 2 mongodb mongodb 4.0K Apr 29 11:26 diagnostic.data
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:52 index-100--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:52 index-102--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:44 index-104--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-184--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-185--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-186--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-187--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-188--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-189--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-190--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-191--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-193--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-194--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-195--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-196--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-198--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-199--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 24 22:08 index-20-4691893134753119794.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-201--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-202--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 04:02 index-204--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:39 index-205--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-58--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-61--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 12:01 index-62--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-64--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-65--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 24 22:12 index-67--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-83--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-85--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-87--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-89--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-91--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-93--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 23 10:22 index-95--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 12:39 index-97--4937860834407901523.wt
-rw-rw---- 1 mongodb mongodb 4.0K Apr 21 11:54 index-99--4937860834407901523.wt
-rw------- 1 mongodb mongodb 1.2K Apr 29 11:26 WiredTiger.turtle
-rw-rw---- 1 mongodb mongodb  114 Apr  8 20:24 storage.bson
drwxrwS--- 2 mongodb mongodb  110 Apr 29 11:03 journal
-rw-rw---- 1 mongodb mongodb   45 Apr  8 20:24 WiredTiger
-rw-rw---- 1 mongodb mongodb   21 Apr  8 20:24 WiredTiger.lock
-rw-rw---- 1 mongodb mongodb    2 Apr 23 10:23 mongod.lock

[skriss]

OK, so definitely a couple big files that dominate the size of the data dir (as expected). My sense from looking at the restic code is that restic is not particularly efficient when this is the case -- it's much more efficient when it has lots of files that it can process in parallel. I'll look into some of the open issues/PRs and see if there's any work in progress on this particular area.

[vitobotta]

Metrics for Backup2

Hi, I hope you don't mind this little off-topic. How do you visualize the metrics? I have installed Prometheus Operator and then Velero with the metrics enabled. Where/how do I see the graphs etc? Thanks

[agolomoodysaada]

@vitobotta , In my case, I use Grafana to view the metrics scraped from cAdvisor by Prometheus. I'm not showing any Velero metrics in this report. If you wanted to access Velero metrics, you should be able to go to the prometheus UI or Grafana and and type something like velero and autocomplete should kick in...

[vitobotta]

Thanks @agolomoodysaada

[eleanor-millman]

Thanks for the information!