Velero: crash when backup 'invalid memory address or nil pointer dereference'

Created on 16 Nov 2020  路  8Comments  路  Source: vmware-tanzu/velero

What steps did you take and what happened:

  • create a backup: velero backup create mybackup7

What did you expect to happen:

  • created backup successfully without error

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

  • log
velero-plugin-for-aws stream closed
velero time="2020-11-16T14:02:31Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:191"
velero time="2020-11-16T14:02:31Z" level=info msg="Starting Velero server v1.5.2 (456eb19668f8da603756353d9179b59b5a7bfa04)" logSource="pkg/cmd/server/server.go:193"
velero time="2020-11-16T14:02:31Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:197"
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/init-restore-hook
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
velero time="2020-11-16T14:02:31Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
velero I1116 14:02:32.954107       1 request.go:621] Throttling request took 1.02624723s, request: GET:https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
velero time="2020-11-16T14:02:32Z" level=info msg="Checking existence of namespace." logSource="pkg/cmd/server/server.go:380" namespace=velero
velero time="2020-11-16T14:02:32Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:386" namespace=velero
velero time="2020-11-16T14:02:36Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:415"
velero time="2020-11-16T14:02:36Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:449"
velero time="2020-11-16T14:02:36Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:571"
velero time="2020-11-16T14:02:36Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:578"
velero time="2020-11-16T14:02:36Z" level=info msg="Backup sync period is 1m0s" logSource="pkg/controller/backup_sync_controller.go:76"
velero time="2020-11-16T14:02:36Z" level=info msg="Waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:836"
velero time="2020-11-16T14:02:39Z" level=info msg="Creating schedule velero-mybackup" controller=schedule logSource="pkg/controller/schedule_controller.go:105"
velero time="2020-11-16T14:02:39Z" level=info msg="Done waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:839"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.Backup" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.DownloadRequest" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.Schedule" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.ResticRepository" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.Restore" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.PodVolumeBackup" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.VolumeSnapshotLocation" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Informer cache synced" informer="*v1.DeleteBackupRequest" logSource="pkg/cmd/server/server.go:850"
velero time="2020-11-16T14:02:39Z" level=info msg="Server starting..." logSource="pkg/cmd/server/server.go:896"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=downloadrequest logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=gc-controller logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=restic-repository logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=schedule logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=backup-sync logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=backup-deletion logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:592"
velero time="2020-11-16T14:02:39Z" level=info msg="Done checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:620"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:39Z" level=info msg="Starting controller" controller=backup logSource="pkg/controller/generic_controller.go:76"
velero time="2020-11-16T14:02:40Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:40Z" level=error msg="Error patching backup location's last-synced time" backupLocation=default controller=backup-sync error="backupstoragelocations.velero.io \"default\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
velero time="2020-11-16T14:02:40Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:40Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:40Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:40Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:40Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:40Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:41Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:41Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:41Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:41Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:41Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:41Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:41Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:42Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
velero time="2020-11-16T14:02:42Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="backupstoragelocations.velero.io \"default\" not found" logSource="pkg/controller/backupstoragelocation_controller.go:115"
velero time="2020-11-16T14:02:42Z" level=info msg="Setting up backup log" backup=velero/mybackup7 controller=backup logSource="pkg/controller/backup_controller.go:512"
velero time="2020-11-16T14:02:42Z" level=info msg="Setting up backup temp file" backup=velero/mybackup7 logSource="pkg/controller/backup_controller.go:534"
velero time="2020-11-16T14:02:42Z" level=info msg="Setting up plugin manager" backup=velero/mybackup7 logSource="pkg/controller/backup_controller.go:541"
velero time="2020-11-16T14:02:42Z" level=info msg="Getting backup item actions" backup=velero/mybackup7 logSource="pkg/controller/backup_controller.go:545"
velero time="2020-11-16T14:02:42Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/mybackup7 logSource="pkg/controller/backup_controller.go:551"
velero time="2020-11-16T14:02:42Z" level=info msg="Writing backup version file" backup=velero/mybackup7 logSource="pkg/backup/backup.go:236"
velero time="2020-11-16T14:02:42Z" level=info msg="Including namespaces: *" backup=velero/mybackup7 logSource="pkg/backup/backup.go:242"
velero time="2020-11-16T14:02:42Z" level=info msg="Excluding namespaces: <none>" backup=velero/mybackup7 logSource="pkg/backup/backup.go:243"
velero time="2020-11-16T14:02:42Z" level=info msg="Including resources: *" backup=velero/mybackup7 logSource="pkg/backup/backup.go:246"
velero time="2020-11-16T14:02:42Z" level=info msg="Excluding resources: <none>" backup=velero/mybackup7 logSource="pkg/backup/backup.go:247"
velero E1116 14:02:42.968459       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
velero goroutine 867 [running]:
velero k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b91520, 0x3197c30)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
velero k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
velero panic(0x1b91520, 0x3197c30)
velero     /usr/local/go/src/runtime/panic.go:969 +0x166
velero github.com/vmware-tanzu/velero/pkg/backup.(*kubernetesBackupper).Backup(0xc000a8cba0, 0x22b72e0, 0xc0000a4770, 0xc00029e280, 0x22383a0, 0xc00000eba0, 0xc000cf6200, 0x4, 0x4, 0x7f73f8d80cc8, ...)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/backup/backup.go:248 +0x677
velero github.com/vmware-tanzu/velero/pkg/controller.(*backupController).runBackup(0xc000153000, 0xc00029e280, 0x0, 0x0)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:568 +0x8a5
velero github.com/vmware-tanzu/velero/pkg/controller.(*backupController).processBackup(0xc000153000, 0xc000493a80, 0x10, 0x0, 0x0)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:272 +0x753
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).processNextWorkItem(0xc000a8ce40, 0x203000)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:132 +0xef
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker(0xc000a8ce40)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:119 +0x2b
velero k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004fffa0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
velero k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000e25fa0, 0x2237500, 0xc0003130e0, 0xc0006ab301, 0xc0005189c0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xa3
velero k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004fffa0, 0x3b9aca00, 0x0, 0x1, 0xc0005189c0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
velero k8s.io/apimachinery/pkg/util/wait.Until(...)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run.func2(0xc000a8ce40, 0x2285340, 0xc0006ab380, 0xc0009b8010)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:92 +0x82
velero created by github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:91 +0x1f3
velero panic: runtime error: invalid memory address or nil pointer dereference [recovered]
velero     panic: runtime error: invalid memory address or nil pointer dereference
velero [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17a0c27]
velero goroutine 867 [running]:
velero k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
velero panic(0x1b91520, 0x3197c30)
velero     /usr/local/go/src/runtime/panic.go:969 +0x166
velero github.com/vmware-tanzu/velero/pkg/backup.(*kubernetesBackupper).Backup(0xc000a8cba0, 0x22b72e0, 0xc0000a4770, 0xc00029e280, 0x22383a0, 0xc00000eba0, 0xc000cf6200, 0x4, 0x4, 0x7f73f8d80cc8, ...)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/backup/backup.go:248 +0x677
velero github.com/vmware-tanzu/velero/pkg/controller.(*backupController).runBackup(0xc000153000, 0xc00029e280, 0x0, 0x0)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:568 +0x8a5
velero github.com/vmware-tanzu/velero/pkg/controller.(*backupController).processBackup(0xc000153000, 0xc000493a80, 0x10, 0x0, 0x0)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:272 +0x753
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).processNextWorkItem(0xc000a8ce40, 0x203000)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:132 +0xef
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).runWorker(0xc000a8ce40)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:119 +0x2b
velero k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004fffa0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
velero k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000e25fa0, 0x2237500, 0xc0003130e0, 0xc0006ab301, 0xc0005189c0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xa3
velero k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004fffa0, 0x3b9aca00, 0x0, 0x1, 0xc0005189c0)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
velero k8s.io/apimachinery/pkg/util/wait.Until(...)
velero     /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
velero github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run.func2(0xc000a8ce40, 0x2285340, 0xc0006ab380, 0xc0009b8010)
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:92 +0x82
velero created by github.com/vmware-tanzu/velero/pkg/controller.(*genericController).Run
velero     /go/src/github.com/vmware-tanzu/velero/pkg/controller/generic_controller.go:91 +0x1f3
velero stream closed
  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
Name:         mybackup7
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion: v1.18.10+k3s2
              velero.io/source-cluster-k8s-major-version: 1
              velero.io/source-cluster-k8s-minor-version: 18
API Version:  velero.io/v1
Kind:         Backup
Metadata:
  Creation Timestamp:  2020-11-16T14:02:42Z
  Generation:          2
  Managed Fields:
    API Version:  velero.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:hooks:
        f:includedNamespaces:
        f:ttl:
      f:status:
    Manager:      velero
    Operation:    Update
    Time:         2020-11-16T14:02:42Z
    API Version:  velero.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:velero.io/source-cluster-k8s-gitversion:
          f:velero.io/source-cluster-k8s-major-version:
          f:velero.io/source-cluster-k8s-minor-version:
        f:labels:
          .:
          f:velero.io/storage-location:
      f:spec:
        f:storageLocation:
      f:status:
        f:expiration:
        f:formatVersion:
        f:phase:
        f:startTimestamp:
        f:version:
    Manager:         velero-server
    Operation:       Update
    Time:            2020-11-16T14:02:42Z
  Resource Version:  791920
  Self Link:         /apis/velero.io/v1/namespaces/velero/backups/mybackup7
  UID:               7a20c4be-2155-4ff5-a5de-c7d3b0dbf713
Spec:
  Hooks:
  Included Namespaces:
    *
  Storage Location:  default
  Ttl:               720h0m0s
Status:
  Expiration:       2020-12-16T14:02:42Z
  Format Version:   1.1.0
  Phase:            InProgress
  Start Timestamp:  2020-11-16T14:02:42Z
  Version:          1
Events:             <none>

Environment:

  • Velero version (use velero version): 1.5.2
  • Kubernetes version (use kubectl version): v1.18.10+k3s2
  • Kubernetes installer & version: v1.18.10+k3s2

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • :+1: for "I would like to see this bug fixed as soon as possible"
  • :-1: for "There are more important bugs to focus on right now"
Needs info
Source
picture thanhtoan1196
👍3

Most helpful comment

Thanks for that pointer @fabriciojs. We will call that out in our upgrade instructions next time.
I am going to close this issue as resolved.

picture ashish-amarnath on 17 Nov 2020
👍2

All 8 comments

It seems something with the parsing of the new flag DefaultVolumesToRestic - https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/backup.go#L224 - trying to figure out in the middle of other changes in our deployment. Trying to narrow it down.

fabriciojs picture fabriciojs on 17 Nov 2020

@fabriciojs I think the cause is the DefaultVolumesToRestic flag, I got the error after upgrading to 1.5.2

thanhtoan1196 picture thanhtoan1196 on 17 Nov 2020

@fabriciojs I think the cause is the DefaultVolumesToRestic flag, I got the error after upgrading to 1.5.2

Yep. We are eager to use the flag 馃槃

We thought it could be mismatched versions of server/cli, but didn't work. We tried defining the DefaultVolumesToRestic both through the CLI flag as well as the Helm chart configuration value. No luck.

Looking at the history change on that file the only thing jumps out (without deeply understanding all structs in use) is this change: https://github.com/vmware-tanzu/velero/commit/63f7690f442b03ffb93490645c01c87ab271528b#diff-bf8e16a3294f23b13ff103c0de59707a9e50eaad4e564a50c76adea06da1351dL245-R245

The removed conditional did expect the value could be nil at that point, it was removed but kept the below log line which expected the pointer to be not nil.

Hope some maintainer/developer familiar with the flow can quickly spot what is going on - it can be either bad configuration on our end using the flag, but certainly the code is not validating properly the pointer state for that log line.

fabriciojs picture fabriciojs on 17 Nov 2020

@thanhtoan1196 and @fabriciojs
You probably skipped a critical step in the upgrade instructions, installing the CRDs. This is step 2 of the upgrade instruction.
The upgrade instructions can be found at https://velero.io/docs/v1.5/upgrade-to-1.5/
image

Also, you will need to use the v1.5.X Velero CLI binary for this.

ashish-amarnath picture ashish-amarnath on 17 Nov 2020
👍1

@ashish-amarnath thanks a lot, it solved for us!

Although we saw the notes, we naively expected that since we were using the Helm chart - which has the CRDs already - that step would be taken care of for us.

Thanks!

fabriciojs picture fabriciojs on 17 Nov 2020
👍1

Looking here https://github.com/vmware-tanzu/helm-charts/pull/163/files i see that the CRDs were updated here, and as we have installCRDs: true we thought it would upgrade it properly.

Maybe helm chart is out of sync or is there a different behavior when using helm?

dbpolito picture dbpolito on 17 Nov 2020

Just for whoever sees this, @dbpolito and I finally understood why the step is indeed necessary and not managed by Helm alone.

Discussion: https://github.com/helm/helm/issues/6581#issuecomment-538515612

fabriciojs picture fabriciojs on 17 Nov 2020

Thanks for that pointer @fabriciojs. We will call that out in our upgrade instructions next time.
I am going to close this issue as resolved.

ashish-amarnath picture ashish-amarnath on 17 Nov 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

doronmak picture doronmak  路  3Comments
concaf picture concaf  路  3Comments
vitobotta picture vitobotta  路  3Comments
akgunjal picture akgunjal  路  3Comments
totemcaf picture totemcaf  路  4Comments