Velero: Error: unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64

[guyguy333]

What steps did you take and what happened:

Run a full backup of cluster using Restic

What did you expect to happen:

No fail during backup

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero

Found 2 pods, using pod/restic-sgkfd
time="2020-03-04T17:44:27Z" level=info msg="Setting log-level to INFO"
time="2020-03-04T17:44:27Z" level=info msg="Starting Velero restic server v1.3.0 (8fec8ed7fb8b4776b191753497eafeb47f2f9136)" logSource="pkg/cmd/cli/restic/server.go:62"
time="2020-03-04T17:44:28Z" level=info msg="Starting controllers" logSource="pkg/cmd/cli/restic/server.go:156"
time="2020-03-04T17:44:28Z" level=info msg="Controllers started successfully" logSource="pkg/cmd/cli/restic/server.go:199"
time="2020-03-04T17:44:28Z" level=info msg="Starting controller" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:76"
time="2020-03-04T17:44:28Z" level=info msg="Waiting for caches to sync" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:79"
time="2020-03-04T17:44:28Z" level=info msg="Starting controller" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:76"
time="2020-03-04T17:44:28Z" level=info msg="Waiting for caches to sync" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:79"
time="2020-03-04T17:44:28Z" level=info msg="Caches are synced" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:83"
time="2020-03-04T17:44:28Z" level=info msg="Caches are synced" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:83"
time="2020-03-04T17:45:04Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-f6rzx namespace=velero
time="2020-03-04T17:45:05Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-f6rzx namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-04T17:45:05Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-f6rzx namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-04T17:45:05Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-f6rzx namespace=velero
time="2020-03-04T17:45:12Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-f6rzx namespace=velero
time="2020-03-05T11:04:12Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-gc2nt namespace=velero
time="2020-03-05T11:04:16Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-gc2nt namespace=velero
time="2020-03-05T11:04:16Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-d5272 namespace=velero
time="2020-03-05T11:04:16Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-d5272 namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:04:16Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-d5272 namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:04:16Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-d5272 namespace=velero
time="2020-03-05T11:04:20Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-d5272 namespace=velero
time="2020-03-05T11:04:20Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-ct92z namespace=velero
time="2020-03-05T11:04:20Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-ct92z namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:04:20Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-ct92z namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:04:20Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-ct92z namespace=velero
time="2020-03-05T11:04:36Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-ct92z namespace=velero
time="2020-03-05T11:04:37Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-5swgr namespace=velero
time="2020-03-05T11:04:37Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-5swgr namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:04:37Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-5swgr namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:04:37Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-5swgr namespace=velero
time="2020-03-05T11:04:40Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-5swgr namespace=velero
time="2020-03-05T11:14:52Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-544n5 namespace=velero
time="2020-03-05T11:14:52Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-544n5 namespace=velero pvcUID=ca3ef62f-3ecf-4527-a565-32e779cd5920
time="2020-03-05T11:14:52Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-544n5 namespace=velero pvcUID=ca3ef62f-3ecf-4527-a565-32e779cd5920
time="2020-03-05T11:14:52Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-544n5 namespace=velero
time="2020-03-05T11:17:12Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-544n5 namespace=velero
time="2020-03-05T11:17:12Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-c9c5p namespace=velero
time="2020-03-05T11:17:12Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-c9c5p namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:17:12Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-c9c5p namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:17:12Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-c9c5p namespace=velero
time="2020-03-05T11:17:16Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-c9c5p namespace=velero
time="2020-03-05T11:17:16Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-psrrr namespace=velero
time="2020-03-05T11:17:16Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-psrrr namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:17:16Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-psrrr namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:17:16Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-psrrr namespace=velero
time="2020-03-05T11:17:25Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-psrrr namespace=velero
time="2020-03-05T11:17:25Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-zwdt8 namespace=velero
time="2020-03-05T11:17:25Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-zwdt8 namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:17:25Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-zwdt8 namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:17:25Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-zwdt8 namespace=velero
time="2020-03-05T11:17:28Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-zwdt8 namespace=velero
time="2020-03-05T11:27:23Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-nn7h8 namespace=velero
time="2020-03-05T11:27:23Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-nn7h8 namespace=velero pvcUID=ca3ef62f-3ecf-4527-a565-32e779cd5920
time="2020-03-05T11:27:23Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-nn7h8 namespace=velero pvcUID=ca3ef62f-3ecf-4527-a565-32e779cd5920
time="2020-03-05T11:27:23Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-nn7h8 namespace=velero
time="2020-03-05T11:28:20Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-nn7h8 namespace=velero
time="2020-03-05T11:28:21Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-p7nn7 namespace=velero
time="2020-03-05T11:28:21Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-p7nn7 namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:28:21Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-p7nn7 namespace=velero pvcUID=34c79b4b-3f7f-41e4-97a4-636945cf9815
time="2020-03-05T11:28:21Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-p7nn7 namespace=velero
time="2020-03-05T11:28:25Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-p7nn7 namespace=velero
time="2020-03-05T11:28:25Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-sb8bh namespace=velero
time="2020-03-05T11:28:25Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-sb8bh namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:28:25Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-sb8bh namespace=velero pvcUID=369483b4-92d3-41a6-9bb5-50b03d05eaf7
time="2020-03-05T11:28:25Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-sb8bh namespace=velero
time="2020-03-05T11:28:38Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-sb8bh namespace=velero
time="2020-03-05T11:28:38Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-5llwt namespace=velero
time="2020-03-05T11:28:38Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-5llwt namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:28:38Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-5llwt namespace=velero pvcUID=627f2cdc-df81-4d4d-b3a8-6dea4501dbb9
time="2020-03-05T11:28:38Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-5llwt namespace=velero
time="2020-03-05T11:28:42Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-5llwt namespace=velero
time="2020-03-05T11:28:45Z" level=info msg="Backup starting" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=test-backup-926tt namespace=velero
time="2020-03-05T11:28:45Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=test-backup-926tt namespace=velero pvcUID=5e733400-e696-4f2a-85f4-4ee1458bb664
time="2020-03-05T11:28:45Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=test-backup-926tt namespace=velero pvcUID=5e733400-e696-4f2a-85f4-4ee1458bb664
time="2020-03-05T11:28:45Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=test-backup-926tt namespace=velero
time="2020-03-05T11:30:48Z" level=info msg="Backup completed" backup=velero/test-backup controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=test-backup-926tt namespace=velero

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

Name:         test-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  PartiallyFailed (run `velero backup logs test-backup` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-03-05 12:27:08 +0100 CET
Completed:  2020-03-05 12:30:54 +0100 CET

Expiration:  2020-04-04 13:27:08 +0200 CEST

Persistent Volumes: <none included>

Restic Backups (specify --details for more information):
  Completed:  5

• velero backup logs <backupname>

[...]
resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=resticrepositories.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=resticrepositories.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=restores.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=restores.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=schedules.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=schedules.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=serverstatusrequests.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=serverstatusrequests.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=volumesnapshotclasses.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=volumesnapshotclasses.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="1 errors encountered backup up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:283" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions
time="2020-03-05T11:30:53Z" level=error msg="Error backing up item" backup=velero/test-backup error="error executing custom action (groupResource=customresourcedefinitions.apiextensions.k8s.io, namespace=, name=volumesnapshotcontents.snapshot.storage.k8s.io): rpc error: code = Unknown desc = unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_backupper.go:334" error.function="github.com/vmware-tanzu/velero/pkg/backup.(*defaultItemBackupper).executeActions" group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:287" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=volumesnapshotlocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=volumesnapshotlocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=volumesnapshots.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing custom action" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_backupper.go:330" name=volumesnapshots.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/test-backup cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:50" pluginName=velero
time="2020-03-05T11:30:53Z" level=info msg="Backing up group" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-03-05T11:30:53Z" level=info msg="Backing up resource" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=priorityclasses
time="2020-03-05T11:30:53Z" level=info msg="Listing items" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace= resource=priorityclasses
time="2020-03-05T11:30:53Z" level=info msg="Retrieved 2 items" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace= resource=priorityclasses
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=system-cluster-critical namespace= resource=priorityclasses.scheduling.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Backing up item" backup=velero/test-backup group=scheduling.k8s.io/v1 logSource="pkg/backup/item_backupper.go:169" name=system-node-critical namespace= resource=priorityclasses.scheduling.k8s.io
time="2020-03-05T11:30:53Z" level=info msg="Backing up group" backup=velero/test-backup group=coordination.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-03-05T11:30:53Z" level=info msg="Backing up resource" backup=velero/test-backup group=coordination.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=leases
time="2020-03-05T11:30:53Z" level=info msg="Listing items" backup=velero/test-backup group=coordination.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace= resource=leases
time="2020-03-05T11:30:53Z" level=info msg="Retrieved 5 items" backup=velero/test-backup group=coordination.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace= resource=leases
[...]

• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

time="2020-03-05T11:30:53Z" level=info msg="1 errors encountered backup up item" backup=velero/test-backup group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:283" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions
time="2020-03-05T11:30:53Z" level=error msg="Error backing up item" backup=velero/test-backup error="error executing custom action (groupResource=customresourcedefinitions.apiextensions.k8s.io, namespace=, name=volumesnapshotcontents.snapshot.storage.k8s.io): rpc error: code = Unknown desc = unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_backupper.go:334" error.function="github.com/vmware-tanzu/velero/pkg/backup.(*defaultItemBackupper).executeActions" group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:287" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions

Environment:

• Velero version (use velero version): v1.3.0
• Velero features (use velero client config get features): features:
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-23T14:21:36Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
• Cloud provider or hardware configuration: Scaleway
• OS (e.g. from /etc/os-release):

[NissesSenap]

Was just going to create the same issue.
I get it even if i don't use restic.

[NissesSenap]

@nrb can this be the same as https://github.com/vmware-tanzu/velero/pull/1937?

[guyguy333]

@nrb can this be the same as #1937?

Indeed, looks related. It has been merged January 30, so it's included in release v1.3.0 I'm using. Maybe related, but not totally solved ?

[skriss]

Looks like it may be another instance of https://github.com/vmware-tanzu/velero/pull/1937#pullrequestreview-344047216

[nrb]

Yeah, looks like the conversion issue was reintroduced in another section of the code, this time the backup plugin that was meant to fix it.

Sorry about that. I'll work on a fix for that today.

[nrb]

@guyguy333 and @NissesSenap Would you be able to share the CRDs in the backup with me? I should only need the definitions, not any specific instances.

You can them by using velero backup download <backup name> and you'll get a tarball named <backup name>-data.tar.gz. Extract that, and you'll have a directory called resources. I'm interested in resources/cluster/customresourcedefinitions/ so I can have a broader sample to test against.

It looks like, in this specific instance, that we got hit with VolumeSnapshotContents though.

[NissesSenap]

cluster-backup-20200305134838-data.tar.gz
@nrb this is my failed backup...

An example on the crd:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
creationTimestamp: "2020-02-17T13:17:05Z"
generation: 1
name: cephfilesystems.ceph.rook.io
resourceVersion: "4484106"
selfLink: /apis/apiextensions.k8s.io/v1/customresourcedefinitions/cephfilesystems.ceph.rook.io
uid: c17df2ee-d467-4dec-b7c4-b6901f043f6c
spec:
conversion:
strategy: None
group: ceph.rook.io
names:
kind: CephFilesystem
listKind: CephFilesystemList
plural: cephfilesystems
singular: cephfilesystem
preserveUnknownFields: true
scope: Namespaced
versions:

• additionalPrinterColumns:
  
  
  
  • description: Number of desired active MDS daemons
    
    jsonPath: .spec.metadataServer.activeCount
    
    name: ActiveMDS
    
    type: string
  
  
  • jsonPath: .metadata.creationTimestamp
    
    name: Age
    
    type: date
    
    name: v1
    
    schema:
    
    openAPIV3Schema:
    
    properties:
    
    spec:
    
    properties:
    
    dataPools:
    
    items:
    
    properties:
    
    erasureCoded:
    
    properties:
    
    codingChunks:
    
    type: integer
    
    dataChunks:
    
    type: integer
    
    failureDomain:
    
    type: string
    
    replicated:
    
    properties:
    
    size:
    
    maximum: 10
    
    minimum: 1
    
    type: integer
    
    type: array
    
    metadataPool:
    
    properties:
    
    erasureCoded:
    
    properties:
    
    codingChunks:
    
    type: integer
    
    dataChunks:
    
    type: integer
    
    failureDomain:
    
    type: string
    
    replicated:
    
    properties:
    
    size:
    
    maximum: 10
    
    minimum: 1
    
    type: integer
    
    metadataServer:
    
    properties:
    
    activeCount:
    
    maximum: 10
    
    minimum: 1
    
    type: integer
    
    activeStandby:
    
    type: boolean
    
    annotations: {}
    
    placement: {}
    
    resources: {}
    
    served: true
    
    storage: true
    
    status:
    
    acceptedNames:
    
    kind: CephFilesystem
    
    listKind: CephFilesystemList
    
    plural: cephfilesystems
    
    singular: cephfilesystem
    
    conditions:
  
  
• lastTransitionTime: "2020-02-17T13:17:05Z"
  message: '[spec.validation.openAPIV3Schema.properties[spec].properties[dataPools].items.properties[erasureCoded].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[dataPools].items.properties[replicated].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[dataPools].items.type:
  Required value: must not be empty for specified array items, spec.validation.openAPIV3Schema.properties[spec].properties[metadataPool].properties[erasureCoded].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataPool].properties[replicated].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataPool].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataServer].properties[annotations].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataServer].properties[placement].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataServer].properties[resources].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[metadataServer].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].type:
  Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.type:
  Required value: must not be empty at the root]'
  reason: Violations
  status: "True"
  type: NonStructuralSchema
• lastTransitionTime: "2020-02-17T13:17:05Z"
  message: no conflicts found
  reason: NoConflicts
  status: "True"
  type: NamesAccepted
• lastTransitionTime: "2020-02-17T13:17:05Z"
  message: the initial names have been accepted
  reason: InitialNamesAccepted
  status: "True"
  type: Established
  storedVersions:
• v1

Some of the errors from the log:
➜ velero backup logs cluster-backup-20200305134838 --insecure-skip-tls-verify|grep error
time="2020-03-05T13:49:29Z" level=info msg="1 errors encountered backup up item" backup=velero/cluster-backup-20200305134838 group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:283" name=cephclusters.ceph.rook.io namespace= resource=customresourcedefinitions
time="2020-03-05T13:49:29Z" level=error msg="Error backing up item" backup=velero/cluster-backup-20200305134838 error="error executing custom action (groupResource=customresourcedefinitions.apiextensions.k8s.io, namespace=, name=cephclusters.ceph.rook.io): rpc error: code = Unknown desc = unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_backupper.go:334" error.function="github.com/vmware-tanzu/velero/pkg/backup.(defaultItemBackupper).executeActions" group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:287" name=cephclusters.ceph.rook.io namespace= resource=customresourcedefinitions
time="2020-03-05T13:49:29Z" level=info msg="1 errors encountered backup up item" backup=velero/cluster-backup-20200305134838 group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:283" name=cephfilesystems.ceph.rook.io namespace= resource=customresourcedefinitions
time="2020-03-05T13:49:29Z" level=error msg="Error backing up item" backup=velero/cluster-backup-20200305134838 error="error executing custom action (groupResource=customresourcedefinitions.apiextensions.k8s.io, namespace=, name=cephfilesystems.ceph.rook.io): rpc error: code = Unknown desc = unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_backupper.go:334" error.function="github.com/vmware-tanzu/velero/pkg/backup.(defaultItemBackupper).executeActions" group=apiextensions.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:287" name=cephfilesystems.ceph.rook.io namespace= resource=customresourcedefinitions

[guyguy333]

@nrb Sure, this is my customresourcedefinitions.apiextensions.k8s.io directory. Thanks for your investigation.

customresourcedefinitions.apiextensions.k8s.io.zip

[nrb]

I've got a failing test case at #2322 that reproduces this reliably for the plugin.

I'm currently debating between two implementations. I may need to do some more testing to see if I can get any characteristics to help me choose.

[turkenh]

I am still getting this error with Velero v1.3.1 and k8s v1.17.3:

Errors:
  Velero:     <none>
  Cluster:  error preparing customresourcedefinitions.apiextensions.k8s.io/bucketclasses.storage.gcp.crossplane.io: rpc error: code = Unknown desc = unable to convert unstructured item to custom resource definition: cannot convert int64 to float64
            error preparing customresourcedefinitions.apiextensions.k8s.io/buckets.storage.gcp.crossplane.io: rpc error: code = Unknown desc = unable to convert unstructured item to custom resource definition: cannot convert int64 to float64
  Namespaces: <none>

Here are the types causing this issue:

• https://github.com/crossplane/provider-gcp/blob/master/config/crd/storage.gcp.crossplane.io_bucketclasses.yaml
• https://github.com/crossplane/provider-gcp/blob/master/config/crd/storage.gcp.crossplane.io_buckets.yaml

No problem with the other types here: https://github.com/crossplane/provider-gcp/tree/master/config/crd

[nrb]

@turkenh Are you using Velero v1.3.1 on the cluster and the client? Can you verify with the output of velero version?

[turkenh]

@nrb yes, I reproduced again. Please see below:

Velero controller logs:

time="2020-03-25T20:41:50Z" level=info msg="Starting Velero server v1.3.1 (0665b05321eefeb7b7fdd6984750745b7429774f)" logSource="pkg/cmd/server/server.go:173"

Versions:

$ velero version
Client:
    Version: v1.3.1
    Git commit: 0665b05321eefeb7b7fdd6984750745b7429774f
Server:
    Version: v1.3.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Repro Steps:

$ kubectl create -f https://raw.githubusercontent.com/crossplane/provider-gcp/master/config/crd/storage.gcp.crossplane.io_buckets.yaml
customresourcedefinition.apiextensions.k8s.io/buckets.storage.gcp.crossplane.io created

$ velero backup create repro --snapshot-volumes=false --include-cluster-resources=true --include-resources=customresourcedefinition.apiextensions.k8s.io
Backup request "repro" submitted successfully.
Run `velero backup describe repro` or `velero backup logs repro` for more details.

$ velero backup get repro
NAME    STATUS       CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
repro   Completed   2020-03-25 20:47:36 +0000 UTC   29d       default            <none>

$ kubectl delete -f https://raw.githubusercontent.com/crossplane/provider-gcp/master/config/crd/storage.gcp.crossplane.io_buckets.yaml
customresourcedefinition.apiextensions.k8s.io "buckets.storage.gcp.crossplane.io" deleted

$ velero restore create --from-backup repro
Restore request "repro-20200325204952" submitted successfully.
Run `velero restore describe repro-20200325204952` or `velero restore logs repro-20200325204952` for more details.

$ velero restore get
NAME                   BACKUP   STATUS            WARNINGS   ERRORS   CREATED                         SELECTOR
repro-20200325204952   repro    PartiallyFailed   34         1        2020-03-25 20:49:52 +0000 UTC   <none>

$ velero restore describe repro-20200325204952
Name:         repro-20200325204952
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs repro-20200325204952' for more information)

Warnings:
  Velero:     <none>
  Cluster:  could not restore, customresourcedefinitions.apiextensions.k8s.io "applicationconfigurations.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "backups.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "backupstoragelocations.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "buckets.storage.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "clusterstackinstalls.stacks.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "components.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "containerizedworkloads.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "crossplaneclusters.crossplanecluster.cloud.upbound.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "deletebackuprequests.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "downloadrequests.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "kubernetesapplicationresources.workload.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "kubernetesapplications.workload.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "kubernetesclusters.compute.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "kubernetestargets.workload.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "machineinstances.compute.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "manualscalertraits.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "mysqlinstances.database.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "nosqlinstances.database.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "podvolumebackups.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "podvolumerestores.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "postgresqlinstances.database.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "providers.kubernetes.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "redisclusters.cache.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "resticrepositories.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "restores.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "schedules.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "scopedefinitions.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "serverstatusrequests.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "stackdefinitions.stacks.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "stackinstalls.stacks.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "stacks.stacks.crossplane.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "traitdefinitions.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "volumesnapshotlocations.velero.io" already exists. Warning: the in-cluster version is different than the backed-up version.
            could not restore, customresourcedefinitions.apiextensions.k8s.io "workloaddefinitions.core.oam.dev" already exists. Warning: the in-cluster version is different than the backed-up version.
  Namespaces: <none>

Errors:
  Velero:     <none>
  Cluster:  error preparing customresourcedefinitions.apiextensions.k8s.io/buckets.storage.gcp.crossplane.io: rpc error: code = Unknown desc = unable to convert unstructured item to custom resource definition: cannot convert int64 to float64
  Namespaces: <none>

Backup:  repro

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

[turkenh]

Any update on this?

Can we reopen this bug, or shall I create a new one?

[skriss]

@nrb sounds like we may need to use this approach that we used in the backu plugin, for the restore plugin as well (specifically this code) - does that sound right to you?

[nrb]

@skriss yeah. Will re-open and re-add to v1.4 milestone.

[skriss]

I think we said that the fix for #2383 may fix this as well by replacing/removing the relevant code, so we can hold off on directly addressing for now.

If it turns out that the #2383 fix doesn't cover this, then we can update the restore plugin per https://github.com/vmware-tanzu/velero/issues/2319#issuecomment-617379431.

[nrb]

@skriss Yeah - it's possible that we may end up just removing the plugins, which would remove the call to FromUnstructured that you linked to. If not, then we can revisit the marshalling.