Velero: Known Issue: CRDs cannot be restored due to fields being identical values

[TomaszKlosinski]

Edit from @nrb -> Please find a summary of the causes HERE. THIS IS A KNOWN ISSUE WITH A FIX TARGETTED FOR VERSION 1.4

What steps did you take and what happened:
I've made a backup of my namespaces with elastic-operator, elasticsearch, kibana and filebeat.

Then I've deleted them with following script:

$ kubectl delete ns elastic-system elk-stack filebeat

Then I've restored it. The restored Elasticsearch+Filebeat+Kibana seems to work, but velero showed errors.

What did you expect to happen:
Restore without errors.

The output of the following commands will help us better understand what's going on:

$ velero restore logs elk-backup-20200401140210 | grep error
time="2020-04-01T12:02:11Z" level=info msg="error restoring elasticsearches.elasticsearch.k8s.elastic.co: CustomResourceDefinition.apiextensions.k8s.io \"elasticsearches.elasticsearch.k8s.elastic.co\" is invalid: [spec.versions[0].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[4].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[4].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[4].JSONPath: Required value, spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]" logSource="pkg/restore/restore.go:1199" restore=velero/elk-backup-20200401140210
time="2020-04-01T12:02:12Z" level=info msg="error restoring kibanas.kibana.k8s.elastic.co: CustomResourceDefinition.apiextensions.k8s.io \"kibanas.kibana.k8s.elastic.co\" is invalid: [spec.versions[0].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[3].JSONPath: Required value, spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]" logSource="pkg/restore/restore.go:1199" restore=velero/elk-backup-20200401140210

It shows erros when restoring CRDs but afterwards they're restored:

$ kubectl get crd -A | grep elastic
apmservers.apm.k8s.elastic.co                  2020-01-20T16:36:56Z
elasticsearches.elasticsearch.k8s.elastic.co   2020-01-20T16:36:56Z
kibanas.kibana.k8s.elastic.co                  2020-01-20T16:36:56Z

• kubectl logs deployment/velero -n velero
  https://termbin.com/uvbx

• velero restore describe elk-backup-20200401140210
  https://termbin.com/k87h

• velero restore logs elk-backup-20200401140210
  https://termbin.com/u23p

Environment:

• Velero version (use velero version):

Client:
    Version: v1.3.1
    Git commit: -
Server:
    Version: v1.3.1

• Velero features (use velero client config get features):

features: <NOT SET>

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
  Manually installed with kubeadm. Version 1.16.

• Cloud provider or hardware configuration:
  Physical hardware + KVM VMs.

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[nrb]

Looks like this is very similar to what was happening in #2370, but different fields.

[nrb]

@TomaszKlosinski Could you provide a link to the ElasticSearch and Kibana YAML you were using to install, so I can reproduce this with the exact same files?

[TomaszKlosinski]

Hello, I've used ECK operator to install it:
https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html

And then I've used this YAML to create Elasticsearch:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: ${ELK_STACK_NAMESPACE}
spec:
  version: 7.5.2
  nodeSets:

  - name: master-dc1
    count: 2
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc1
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearch
          tier: master
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc1
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1Gi
              cpu: "0.2"
            limits:
              memory: 5Gi
              cpu: "2"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: master-dc2
    count: 2
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc2
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearch
          tier: master
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc2
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1Gi
              cpu: "0.2"
            limits:
              memory: 5Gi
              cpu: "2"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: data-dc1
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc1
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearhc
          tier: data
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc1
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            requests:
              memory: ${ELK_DATA_XMX_SIZE}Gi
              cpu: "0.5"
            limits:
              memory: 16Gi
              cpu: "3"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms${ELK_DATA_XMX_SIZE}g -Xmx${ELK_DATA_XMX_SIZE}g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: data-dc2
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc2
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearhc
          tier: data
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc2
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            requests:
              memory: ${ELK_DATA_XMX_SIZE}Gi
              cpu: "0.5"
            limits:
              memory: 16Gi
              cpu: "3"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms${ELK_DATA_XMX_SIZE}g -Xmx${ELK_DATA_XMX_SIZE}g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  http:
    tls:
      certificate:
        # provide our own certificate
        secretName: ${ELK_STACK_NAMESPACE}.${EXTERNAL_DOMAIN}-tls

PS. I use envsubst to resolve the shell variables.

[nrb]

Great, thanks!

[hasheddan]

Hey @nrb! Thanks for looking into this :) It is due to the validation here: https://github.com/kubernetes/apiextensions-apiserver/blob/089b5f38f499f0d173cb1d7f424ade7214d4423e/pkg/apis/apiextensions/validation/validation.go#L238. I am guessing velero may be adding suberesources to both the top-level and per-version stanzas. I have a really simple example here that I noticed it on, which might be helpful. I also am willing to work on a fix here if that would be helpful to the team :+1:

Error:

error restoring customresourcedefinitions.apiextensions.k8s.io/gcpsamples.gcp.stacks.crossplane.io: CustomResourceDefinition.apiextensions.k8s.io "gcpsamples.gcp.stacks.crossplane.io" is inval
id: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(*apiextensions.CustomRes
ourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00c176930), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version subresources may 
not all be set to identical values (top-level subresources should be used instead)       

CRD:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: null
  name: gcpsamples.gcp.stacks.crossplane.io
spec:
  preserveUnknownFields: true
  group: gcp.stacks.crossplane.io
  names:
    kind: GCPSample
    listKind: GCPSampleList
    plural: gcpsamples
    singular: gcpsample
  scope: Cluster
  subresources:
    status: {}
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

Versions:

Velero:
        Version: v1.3.2
        Git commit: 55a9914a3e4719fb1578529c45430a8c11c28145
Kubernetes:
        version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

[nrb]

@hasheddan Thanks for digging into this! Velero _shouldn't_ be adding anything extra to the schemas, but I could be mistaken.

The CRD manipulations we do are in these plugins, if you'd like to double check my work which would be appreciated :) These were introduced with v1.3.0

Backup:

• https://github.com/vmware-tanzu/velero/blob/master/pkg/backup/remap_crd_version_action.go

Restore:

• https://github.com/vmware-tanzu/velero/blob/master/pkg/restore/crd_v1_preserve_unknown_fields_action.go

Thanks for that small CRD, too...I was wondering if it was something about the definitions themselves, but given your data, clearly something else is going on.

[nrb]

I did some tests with the GCP Crossplane Stack CRD that @hasheddan posted and confirmed the error he saw with Velero master. I did a Velero backup then got the JSON files using velero backup download and just tested them out using kubectl apply.

First, with the original JSON (the only difference being I ran it through jq '.' -r to get prettier JSON for eventual diffing):

% k apply -f gcpsamples.gcp.stacks.crossplane.io.json
The CustomResourceDefinition "gcpsamples.gcp.stacks.crossplane.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(0xc009318e30), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead)

Next, I copied the contents to test.json and removed the only subresource, and it applied cleanly:

% k apply -f test.json
customresourcedefinition.apiextensions.k8s.io/gcpsamples.gcp.stacks.crossplane.io created

The diff:

% diff gcpsamples.gcp.stacks.crossplane.io.json test.json
32,35c32
<         "storage": true,
<         "subresources": {
<           "status": {}
<         }
---
>         "storage": true

[nrb]

For comparison, I applied https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/crds/all-crds.yaml to a 1.17.2 KinD cluster, then backed it up and received the same errors as @TomaszKlosinski was originally reporting.

I notice that in the ElasticSearches CRD as defined in the YAML, the additionalPrinterColumns are defined at the top level Spec. However, after being retrieved from the Kubernetes API server by Velero and stored into the backup as JSON, that field is no longer at the top level. It's instead copied into each version with the original values.

See here, here, and here.

I don't think Velero's doing this, as when I retrieve the CRD via kubectl, I see 3 lines for that field, too:

% kubectl get customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co -o yaml | grep additionalPrinter
  - additionalPrinterColumns:
  - additionalPrinterColumns:
  - additionalPrinterColumns:

I need to do some more research, because there's some API server behavior here that I'm not understanding.

[nrb]

Ok, I think I see what's happening now.

Using kubectl get --raw /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/elasticsearches.elasticsearch.k8s.elastic.co, I see that I get the correct object; that is, there's only 1 top-level additionalPrinterColumns entry, not one per version.

I got a hint from this commit, which said that v1 CRDs don't support the top-level field anymore, and also remembered that our plugin just sets the version string, while keeping the object that was fetched from the preferred version endpoint (which on Kubernetes v1.16+ would have been a v1 CRD)

Instead, I think the correct path will be to get a proper v1beta1 client and retrieve the CRD from the correct endpoint on backup.

[hasheddan]

@nrb apologies for just following up here, your digging appears to be in accordance with my understanding though :) An alternative could be to convert both v1 and v1beta to internal schema then render them from there. Here is an example of this strategy: https://github.com/crdsdev/doc/blob/3a12003326ce6e2fee55c388369debf15047799a/pkg/crd/crd.go#L102

[nrb]

@hasheddan No worries! I was capturing notes as I found things. I'll take a look at that approach.

We also have #2373 which will get both versions, too.

[nrb]

@hasheddan Storing the internal representation does seem like a good idea, especially since https://github.com/kubernetes/apiextensions-apiserver/blob/c47e10e6d5a3d95d427c5bee109d934602f0861e/pkg/apis/apiextensions/v1/zz_generated.conversion.go#L306 exists.

I think I'll have to revisit the CRD backup/restore logic entirely. Thank you for your pointers! Do you mind if I tag you on reviews for these changes?

[hasheddan]

@nrb sure thing! Thanks for the great work you do on this project!

[nrb]

A summary so far:

Right now, Velero only gets the server's preferred version of a resource.

In Kubernetes v1.16+, that's v1 of CustomResourceDefinitions.

With v1.3.0, Velero applies a heuristic to see if it had a CRD that was v1 but was _really_ a v1beta1 CRD retrieved through the v1 endpoint. If so, then the backup plugin switched the version string from v1 to v1beta1, and that's it.

As documented in this issue, that's the wrong approach, because the overall structure returned from the v1 endpoint is invalid for v1 CRDs. Our sample size of CRDs was too small to catch this, which will be remedied in #2447.

I believe the heuristics on backup are working - they're catching the v1beta1 CRDs and applying the string swap, which is what then causes this problem.

We have 3 options for fixing the current behavior, captured from the discussion at the community meeting on April 21, 2020:

1. Keep the heuristic, and if matched, use a v1beta1 CRD client to fetch the correct version of the object. This stays in line with Velero's current approach of dropping k8s cluster objects directly into the tarball.
2. Explore adjusting the Velero backup format to store CRDs as apiextension.CustomResourceDefinition, which is the __unversioned__, internal format that the Kubernetes API server uses. This would very likely trigger a major version bump for the backup file format version, and thus a major version bump for Velero. That's because the current expectation is that the files inside the Velero backup tarball are assumed to be directly usable via kubectl without any conversion. Using an internal format breaks that assumption. It would require conversion at back up and restore time.
3. Building on #2373, remove the current CRD backup/restore plugins. Instead, try to restore v1 of a CRD first. If it fails, try to restore v1beta1, since it's already captured in the backup tarball.

Of these options, I think 1 & 3 are viable for the v1.4 timeframe. 3 builds on existing work. 1 duplicates some of 3's logic.

Long term, I think 2 may be the most correct thing to do, but I haven't done the proper research to know what it fully entails.

All of these versions special case the CustomResourceDefinition API group, which I think is unavoidable.

The upstream CRD code states that v1beta1 is planned for removal in v1.19. However, Velero will need to be able to take v1beta1 backups and restore them going forward longer than this window.

@vmware-tanzu/velero-owners Please let me know what your thoughts are! I'd like to dig into this in earnest this week.

@hasheddan Do you have anything to add? I see you've worked on the upstream CRD code - have I gotten anything wrong in this summary?

[skriss]

Thanks for the writeup @nrb. I agree that options 1 and 3 are most feasible now. Option 2 would be a pretty big change, and I'm not not even sure at this point whether it's desirable or not.

Of 1 and 3, I think 3 is the more failsafe option, since it doesn't rely on us correctly identifying whether a CRD was originally created as v1beta1 or not - we just naively back up both representations and try to restore both representations in turn. One of them should work :) It does seem like there are potentially other forms of CRDs that could have been created as v1beta1, that aren't v1-compatible, that won't be caught by our heuristic, though I don't have a specific example off the top of my head.

The upstream CRD code states that v1beta1 is planned for removal in v1.19. However, Velero will need to be able to take v1beta1 backups and restore them going forward longer than this window.

This isn't something Velero has ever supported (restoring into a cluster that doesn't have the API version that the backup uses) -- if we want to support this, we'll have to embed our own conversion logic to upconvert v1beta1 CRDs to v1 prior to creating them via the API.

[nrb]

I've got a branch working with the elasticsearch CRDs now, but I need to update the unit tests to account for the changes. PR should be up tomorrow.

[ashish-amarnath]

@nrb thanks for the awesome description. And sorry for not taking at look at this earlier.
I agree with the plan of making option 2 a long term plan and choosing option 1 in the interim.