velero/restic use default region "us-east-1" when region=GRA (uppercase)

[konvergence]

What steps did you take and what happened:

• follow this doc: https://docs.ovh.com/gb/en/public-cloud/getting_started_with_the_swift_S3_API/
• run install

BUCKET_NAME=lab-k8s-velero-bucket
BUCKET_PUBLICURL="https://storage.gra.cloud.ovh.net"
BUCKET_S3URL="https://storage.gra.cloud.ovh.net"
BUCKET_REGION=GRA


velero install \
   --namespace velero-ovh \
   --use-restic=true \
   --provider aws \
   --plugins velero/velero-plugin-for-aws:v1.0.0 \
   --bucket ${BUCKET_NAME} \
   --backup-location-config region=${BUCKET_REGION},s3ForcePathStyle="true",s3Url="${BUCKET_S3URL}" \
   --snapshot-location-config region=${BUCKET_REGION} \
   --secret-file /tmp/credentials-velero-ovh

• test backup

rke@lab-k8s-admin:~$ velero -n velero-ovh  backup create backup-wordpress-mysql-flex  --include-namespaces wordpress-
Backup request "backup-wordpress-mysql-flex" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background
......
Backup completed with status: PartiallyFailed.  [....]

What did you expect to happen:

• with OVH, I need to put region in uppercase to be able to use S3 API (region=GRA)

The output of the following commands will help us better understand what's going on
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl -n velero-ovh logs deployment/velero

• see https://github.com/vmware-tanzu/velero/files/4142387/deployment_velero.log

• velero -n velero-ovh backup get

NAME                          STATUS                      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-wordpress-mysql-flex   PartiallyFailed (1 error)   2020-01-31 16:34:32 +0100 CET   29d       default           

• velero -n velero-ovh backup describe backup-wordpress-mysql-flex

Name:         backup-wordpress-mysql-flex
Namespace:    velero-ovh
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  PartiallyFailed (run `velero backup logs backup-wordpress-mysql-flex` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  wordpress-mysql-flex
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-01-31 16:34:32 +0100 CET
Completed:  2020-01-31 16:34:37 +0100 CET

Expiration:  2020-03-01 16:34:32 +0100 CET

Persistent Volumes: <none included>

• velero -n velero-ovh backup logs backup-wordpress-mysql-flex
  see https://github.com/vmware-tanzu/velero/files/4142386/backup-wordpress-mysql-flex.log

• velero -n velero-ovh backup logs backup-wordpress-mysql-flex | grep -i error

time="2020-01-31T15:34:33Z" level=info msg="1 errors encountered backup up item" backup=velero-ovh/backup-wordpress-mysql-flex group=v1 logSource="pkg/backup/resource_backupper.go:284" name=wordpress-mysql-84d76d4c9d-kjnkq namespace=wordpress-mysql-flex resource=pods
time="2020-01-31T15:34:33Z" level=error msg="Error backing up item" backup=velero-ovh/backup-wordpress-mysql-flex error="restic repository is not ready: error running command=restic init --repo=s3:https://storage.gra.cloud.ovh.net/lab-k8s-velero-bucket/restic/wordpress-mysql-flex --password-file=/tmp/velero-restic-credentials-wordpress-mysql-flex784278886 --cache-dir=/scratch/.cache/restic, stdout=, stderr=Fatal: create repository at s3:https://storage.gra.cloud.ovh.net/lab-k8s-velero-bucket/restic/wordpress-mysql-flex failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'GRA'\n\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/repository_ensurer.go:186" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*repositoryEnsurer).EnsureRepo" group=v1 logSource="pkg/backup/resource_backupper.go:288" name=wordpress-mysql-84d76d4c9d-kjnkq namespace=wordpress-mysql-flex resource=pods

Anything else you would like to add:

• If I define region=gra, the backup Failed with error : <Message>Error parsing the X-Amz-Credential parameter; the region 'gra' is wrong; expecting 'GRA'</Message>
• If I define region=GRA, the backup is Partial. Velero is able to save Kubernetes Objects on S3, but restic seem to not accept uppercase and return the error : <Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'GRA'</Message>

Environment:

• Velero version (use velero -n velero-ovh version):

Client:
        Version: v1.2.0
        Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
Server:
        Version: v1.2.0

• Velero features (use velero client config get features):

features: <NOT SET>

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: rke
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[konvergence]

backup-wordpress-mysql-flex.log
deployment_velero.log

[skriss]

@konvergence have you been able to try this with restic directly (no velero)? I'm unclear on whether this is a velero issue, or a restic issue.

[konvergence]

Well I can test restic, but I don't well how to do it.
what is the release of restic to test ?
Can you give me a sample of command ?

[skriss]

You can find restic install instructions here: https://restic.readthedocs.io/en/stable/020_installation.html

Velero 1.2 uses restic 0.9.5, but Velero's next release will use restic 0.9.6. So it'd be good to try out both, if possible. You may need to go directly to the GitHub releases to get each: https://github.com/restic/restic/releases

I did see a release note on 0.9.6 about adding support for the AWS_DEFAULT_REGION environment variable or the -o s3.region flag (see https://github.com/restic/restic/releases/tag/v0.9.6 and https://github.com/restic/restic/pull/2484). I'm guessing that using these may be needed for your use case.

I would first try to create a restic repository:

restic --repo=s3:https://storage.gra.cloud.ovh.net/<BUCKET>/<PREFIX> init  

And then if it does not work, try using the env var or -o s3.region flag per above on v0.9.6 and see if that works.

[skriss]

You could also try using velero 1.3.0-beta.1 which has the updated restic 0.9.6, since some other folks reported that this magically fixed their region issues for another S3-compatible API (see https://github.com/vmware-tanzu/velero/issues/1253)

[konvergence]

Thanks skriss, i will try and keep you inform of the results

[konvergence]

I shriss,
I made test with restic 0.9.5 and get the same error.

I think the reason is in the https://github.com/restic/restic/blob/master/CHANGELOG.md
Enh #2350: Add option to configure S3 region
The support of region has been added only since restic 0.9.6

I tested with restic 0.9.6 and it works !

[skriss]

good to know - so my next question is - does it work with velero v1.3.0-beta.1 (which uses restic v0.9.6)? Per https://github.com/vmware-tanzu/velero/issues/2237#issuecomment-584849395, some other folks had success just by upgrading.

It seems like there may need to be a velero code change to plumb this env var through, though.

[skriss]

@konvergence were you able to try out v1.3.0-beta.1?

[konvergence]

@skriss , I plan to test this week

[skriss]

@konvergence just checking in again to see if you were able to test. Also, v1.3.0 is out so you could try that, rather than the beta.

[konvergence]

Good news ! I have no errors with velero 1.3.0

[skriss]

fantastic! I'm not exactly sure what change fixed it (I think it was in restic itself), but glad to hear. Also, I'd recommend bumping to v1.3.1 since we just fixed a bug with backing up CRDs :)