Velero: Velero 1.5.1 error="backupstoragelocations.velero.io \"default\" not found"

Created on 15 Jan 2021 · 15Comments · Source: vmware-tanzu/velero

What steps did you take and what happened:

Installed Velero 1.5.1 in a baremetal cluster with S3 hosted on Rook-ceph object storage

velero install  --provider aws  --plugins velero/velero-plugin-for-aws:v1.0.0  --bucket velero-bucket   --secret-file ./credentials-velero-k8sstorage  --use-volume-snapshots=true  --backup-location-config region=default,s3ForcePathStyle="true",s3Url=http://rook-s3.10.131.228.112.nip.io  --image velero/velero:v1.5.1   --snapshot-location-config region="default"  --use-restic

Getting this error logs

time="2021-01-13T14:04:49Z" level=error msg="Error patching backup location's last-synced time" backupLocation=default controller=backup-sync error="backupstoragelocations.velero.io \"default\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2021-01-13T14:04:58Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2021-01-13T14:04:58Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
time="2021-01-13T14:05:20Z" level=error msg="Error patching backup location's last-synced time" backupLocation=default controller=backup-sync error="backupstoragelocations.velero.io \"default\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311

Try to take backup - It does not complete in time

 velero backup create nginx-test --include-namespaces test-nginx --wait
Backup request "nginx-test" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
............................................................................................................................................^C
````

velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE
default aws velero-bucket Unknown Unknown ReadWrite

In the same testbed , deleted Velero namespace and installed v1.4.0 version

velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.0.0 --bucket velero-bucket --secret-file ./credentials-velero-k8sstorage --use-volume-snapshots=true --backup-location-config region=default,s3ForcePathStyle="true",s3Url=http://rook-s3.10.131.228.112.nip.io --image velero/velero:v1.4.0 --snapshot-location-config region="default" --use-restic

Took backup, everything works; No change between these two related to anything - S3 URL, access keys etc; Only version change

velero backup create nginx-test --include-namespaces test-nginx --wait
Backup request "nginx-test" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.

Backup completed with status: Completed. You may check for more information using the commands `velero backup describe nginx-test` and `velero backup logs nginx-test`.

alex@N-20HEPF0ZU9PR:/mnt/c/Users/acp/Documents/certs3/ee$ velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE
default aws velero-bucket Unknown Unknown ReadWrite

velero get backup
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
nginx-test Completed 0 0 2021-01-15 15:36:17 +0530 DST 29d default

Backup is proper in S3
$ mc ls k8sstorage/velero-bucket/backups
[2021-01-15 15:46:08 DST] 0B nginx-test/
```
*Expected behaviour *

Velero installation with v1.5.1 should work

Needs info Reviewed Q2 2021

Source

alexcpn

👍2

Most helpful comment

Hey @zubron, yes we are using 1.5.2.

Reinkaos on 19 Jan 2021

👍2

All 15 comments

I am seeing the same on v1.5.2.

time="2021-01-18T11:34:19Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2021-01-18T11:34:19Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"

The same BSL configuration works fine on v1.4.0.

- apiVersion: velero.io/v1
  kind: BackupStorageLocation
  metadata:
    creationTimestamp: "2021-01-18T11:33:55Z"
    generation: 1
    labels:
      component: velero
    managedFields:
    - apiVersion: velero.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:component: {}
        f:spec:
          .: {}
          f:config:
            .: {}
            f:insecureSkipTLSVerify: {}
            f:region: {}
            f:s3ForcePathStyle: {}
            f:s3Url: {}
          f:objectStorage:
            .: {}
            f:bucket: {}
          f:provider: {}
      manager: OpenAPI-Generator
      operation: Update
      time: "2021-01-18T11:33:55Z"
    name: default
    namespace: velero
    resourceVersion: "142838293"
    selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
    uid: e5c05c5d-7e8d-4652-95e9-7f8d7c55eef3
  spec:
    config:
      insecureSkipTLSVerify: "true"
      region: minio
      s3ForcePathStyle: "true"
      s3Url: https://s3-noobaa.apps.example.domain.net
    objectStorage:
      bucket: first.bucket
    provider: aws

rflorenc on 18 Jan 2021

Hi! This looks like it is due to incorrect versions of the CRDs on your cluster. Given that you had no issues with 1.4.0, I'm going to assume that you either had a previous installation of 1.4 on the cluster, or are using a 1.4 version of the CLI. You can check this by running velero version.

In order to use 1.5, you will need to follow our instructions here: https://velero.io/docs/v1.5/upgrade-to-1.5/
particularly ensuring that you are using a 1.5 version of the CLI and then performing the instructions to upgrade the CRDs in the cluster.

zubron on 18 Jan 2021

Hey, we are experiencing the same issue. Downgrading to 1.4.3 fixes the problem. We have tried reinstalling the CRDs as described in the link but still no luck.

Reinkaos on 19 Jan 2021

👍1

Hi @Reinkaos - to confirm, were you using a 1.5 version of the CLI to reinstall the CRDs?

zubron on 19 Jan 2021

Hey @zubron, yes we are using 1.5.2.

Reinkaos on 19 Jan 2021

👍2

FYI, I'm also seing the following logs after upgrading to 1.5.2:

time="2021-01-19T07:06:25Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/0/1)" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:154"
time="2021-01-19T07:06:25Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
time="2021-01-19T07:06:25Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"

Using AWS S3 bucket for BackupStorageLocation with this config:
spec: backupSyncPeriod: 0s config: region: eu-west-1 objectStorage: bucket: mybucket prefix: staging provider: aws validationFrequency: 0s

I updated the CRD with velero 1.5.2 CLI.

supasteev0 on 19 Jan 2021

I have the same issue with 1.5.3 installed from scratch, although the backup seems to be running fine.
I'm using the aws provider with minio.

giordyb on 25 Jan 2021

@supasteev0 Your logs don't look like an error - those are expected log messages.

@giordyb Can you be more specific on which issue you're having? I think folks are reporting multiple issues here.

@alexcpn and @rflorenc - can you provide the output of kubectl get crd/backupstoragelocation.velero.io -o yaml - we should see a Status.Phase field in it. Otherwise, the CRD needs to be upgraded.

See step 2 in https://velero.io/docs/v1.5/upgrade-to-1.5/#docs

nrb on 25 Jan 2021

@nrb thank you for the quick reply.

I was following this issue because I was getting the same error as in the title of the issue.
After reinstalling velero from scratch that specific message went away but I kept seeing the same messages as @supasteev0 mentioned (Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available) every minute so I figured it might be related.

If you are saying they are expected then it's fine, sorry if I created some confusion...

giordyb on 26 Jan 2021

I have this same problem, installing from scratch with the helm, on version 1.5.3

DrissiReda on 5 Feb 2021

@DrissiReda Please start a new issue and include your configuration info and logs (it will ask you for them when you create the issue). Thanks!

dsu-igeek on 12 Mar 2021

Hi Team I am unable to create velero backup with my existing namespace in kubernetes cluster. Getting below error:
root@W107J2D3X2:~# velero get backups
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo-backup-1 Failed 0 0 2021-05-04 21:20:13 +0530 IST 29d default app=nginx
demobackup Deleting 0 0 2021-05-04 17:31:46 +0530 IST 29d default
firstbackup Deleting 0 0 2021-05-04 17:19:15 +0530 IST 29d default
newdemobackup Failed 0 0 2021-05-04 21:06:45 +0530 IST 29d default
nginx-backup Failed 0 0 2021-05-04 18:50:24 +0530 IST 29d default
root@W107J2D3X2:~# velero backup-location get

Velero logs:

time="2021-05-04T15:45:40Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://IPdetailscurrentlymasked:9000/kubedemo?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp IPdetailscurrentlymasked:9000: i/o timeout" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:361" error.function="main.(ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182"
time="2021-05-04T15:47:19Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
time="2021-05-04T15:48:11Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://IPdetailscurrentlymasked:9000/kubedemo?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp IPdetailscurrentlymasked:9000: i/o timeout" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:361" error.function="main.(ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182"

devivaraprasad901 on 4 May 2021

@devivaraprasad901
Please create a new issue or check the GitHub discussion.
From the log you provided, if not related to this issue topic.

And it's more like the Velero server can't connect to your S3 bucket.

Could you please check the network connectivity from your Velero server to S3 bucket?
Could you please check velero backup-location get to see if the backup location is Available?

jenting on 6 May 2021

Hi Team,

Please find below snapshot where I can view backup location from velero to
minion application.

But still backup is not getting completed status showing as failed only.

Regards,
Devivara prasad

On Thu, 6 May 2021, 07:39 JenTing Hsiao, @.*> wrote:

@devivaraprasad901 https://github.com/devivaraprasad901
Please create a new issue or check the GitHub discussion.
From the log you provided, if not related to this issue topic.

And it's more like the Velero server can't connect to your S3 bucket.

Could you please check the network connectivity from your Velero
server to S3 bucket?

Could you please check velero backup-location get to see if the
backup location is Available?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vmware-tanzu/velero/issues/3282#issuecomment-833171912,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANPZWTPXTFQL5HULEOP5QBTTMH25LANCNFSM4WD2TYUA
.

devivaraprasad901 on 6 May 2021

I was trying with v 1.4.2 on vSphere TKGI(PKS) cluster, backup stuck in InProgress state and was getting the velero pod logs as backupstoragelocations.velero.io \"default\" not found, replaced with 1.4.3 binary, it worked.