Velero: Unable to backup persistentvolumeclaims to multiple storage locations

[a4nowgh]

What steps did you take and what happened:
An annotated persistent volume claim mount volume fails to backup to a secondary storage location after successfully backing up to a primary storage location. The same backup job to the primary storage location can be performed again successfully.

$ kubectl -n annotate po backup.velero.io/backup-volumes=
$ velero backup create backup-job-1 --include-namespace --storage-location s3-bucket-1 (success)
$ velero backup create backup-job-2 --include-namespace --storage-location s3-bucket-2 (PartiallyFailed (1 error))
$ velero backup describe backup create backup-job-2 --details | grep persistentvolumeclaims
time="2020-02-04T13:50:47Z" level=info msg="Skipping item because it's already been backup." backup=velero/backup-job-2 group=v1 logSource="pkg/backup/item_backupper.go:163" name= namespace= resource=persistentvolumeclaims

What did you expect to happen:
Backup persistentvolumeclaim to a second storage location after backing up to primary storage location.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version: v1.2.0
• Velero features:
• Kubernetes Client version: v1.15.3
• Kubernetes Server version: v1.14.6
• Cloud provider or hardware configuration: aws
• OS: Ubuntu 18.04.3 LTS

[jenting]

@a4nowgh, which case did you configure? Two backup-location and two snapshot-locations or two backup-locations and one snapshot-location?

In my usage, I could take 2 volume snapshots with two backup-locations (to different S3 bucket) and one snapshot-location on AWS. (Velero would create 2 snapshots in AWS EBS snapshots.)

[carlisia]

Hey @a4nowgh would you please give us the output of:

kubectl describe backupstoragelocations?

and

kubectl describe volumesnapshotlocations?

[a4nowgh]

Hello,
Below is my output:

$ kubectl describe backupstoragelocations default
Spec:  Config:    Region:    us-east-1    s3ForcePathStyle: true    s3Url:    https://backupserver2.cloud.com
  Object Storage:    Bucket: bucketname  Provider: aws

$ kubectl describe backupstoragelocations default
Spec:  Config:    Region:    minio    s3ForcePathStyle: true    s3Url:    https://backupserver2.cloud.com
  Object Storage:    Bucket: bucketname  Provider: aws
$ kubectl describe volumesnapshotlocation default
Spec:  Config:    Profile: default
    Region: us-east-1  Provider: velero.io/aws
$ kubectl describe volumesnapshotlocation minio
Spec:  Config:    Profile: minio
    Region: minio  Provider: velero.io/aws

On Wednesday, February 5, 2020, 4:45:04 PM EST, Carlisia Campos <[email protected]> wrote:

Hey @a4nowgh would you please give us the output of:

kubectl describe backupstoragelocations?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[skriss]

@a4nowgh couple of questions:

1. I see you're running on AWS - are you using EBS snapshots or restic for the volume backup?
2. Could you include the full logs of both backups? (you can put them in a Gist/pastebin/etc and include a link). If that's not possible for some reason, please at least send the error message(s) that you're getting on the second backup.

Wondering if this is a duplicate of #2192?

[a4nowgh]

Yes, this seems to be a duplicate of #2192.  I'm wondering if PR #2244(https://github.com/vmware-tanzu/velero/pull/2244) will address this issue?

On Tuesday, February 11, 2020, 3:41:30 PM EST, Steve Kriss [e-mail] wrote:

@a4nowgh couple of questions:

• I see you're running on AWS - are you using EBS snapshots or restic for the volume backup?
• Could you include the full logs of both backups? (you can put them in a Gist/pastebin/etc and include a link). If that's not possible for some reason, please at least send the error message(s) that you're getting on the second backup.

Wondering if this is a duplicate of #2192?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[a4nowgh]

I'm using restic for the volume backup.

On Wednesday, February 12, 2020, 5:17:30 AM EST, [email protected] <[email protected]> wrote:

Yes, this seems to be a duplicate of #2192.  I'm wondering if PR #2244(https://github.com/vmware-tanzu/velero/pull/2244) will address this issue?

On Tuesday, February 11, 2020, 3:41:30 PM EST, Steve Kriss [e-mail] wrote:

@a4nowgh couple of questions:

• I see you're running on AWS - are you using EBS snapshots or restic for the volume backup?
• Could you include the full logs of both backups? (you can put them in a Gist/pastebin/etc and include a link). If that's not possible for some reason, please at least send the error message(s) that you're getting on the second backup.

Wondering if this is a duplicate of #2192?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[skriss]

I would expect #2244 to fix it, yes. I'll link this issue up to that PR. Hoping to get that fix in for our upcoming release.

[skriss]

xref https://github.com/vmware-tanzu/velero/pull/2244#issuecomment-591446887

@jenting - any ideas on why @a4nowgh is still getting this error with v1.3.0-beta.2? I'm not sure off the top of my head. I'm going to try to reproduce as well.

[skriss]

@a4nowgh did you also update your restic daemonset to use the beta.2 image? kubectl -n velero set image daemonset/restic restic=velero/velero:v1.3.0-beta.2 would do that, if not.

FWIW, I was not able to reproduce this using beta.2.

[jenting]

From the log, my first idea is the container image is not v1.3.0-beta.2.

@a4nowgh, did you follow the v1.3.0-beta.2 release note on upgrading the velero and restic container image?

[a4nowgh]

Upgrading the restic container image, in additional to the velero container image, resolved my issue. Thanks very much for the quick response!