velero v1.2.0 --dry-run -o yaml inserts "nullable" in generated kube manifest files

[cmcconnell1]

What steps did you take and what happened:

generate kube manifest files from the velero binary and then attempt to apply to cluster via kubectl apply

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:${VELERO_AWS_PLUGIN_VERSION} \
    --bucket ${VELERO_BUCKET_NAME} \
    --backup-location-config region=$VELERO_AWS_REGION \
    --snapshot-location-config region=$VELERO_AWS_REGION \
    --secret-file ./velero-credentials.secret \
    --dry-run -o yaml \
> ./manifests/velero/velero.yaml

works fine but cannot apply them to cluster as shown below. 

**What did you expect to happen:**
Expected generated kube manifest files to be usable per the documentaion and be able to be applied to kubernetes `kubectl apply` without errors. 

**Errors during kubectl apply**:
```console
kubectl apply --recursive -f $MANIFEST_PATH -n velero
error: error validating "manifests/velero/velero.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.excludedNamespaces): unknown field "nullable"

Note: there are ~15 more errors like above for every 'nullable' reference autogenerated in the velero manifest files...

Anything else you would like to add:

Environment:

velero version
Client:
  Version: v1.2.0
  Git commit: -
Server:
Version: (cannot apply manifests therefore cannot provide server info as it errors-out)

• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):

 k version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.12-eks-c500e1", GitCommit:"c500e11584c323151d6ab17526d1ed7461e45b0c", GitTreeState:"clean", BuildDate:"2019-10-22T03:11:52Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
  terraform-eks-module

• OS (e.g. from /etc/os-release):

sw_vers
ProductName:  Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G1012

system_profiler SPSoftwareDataType| egrep "(System|Kernel)"
    System Software Overview:
      System Version: macOS 10.14.6 (18G1012)
      Kernel Version: Darwin 18.7.0
      System Integrity Protection: Enabled

• Note that modifying my velero build/install to use helm template works as expected to generate requisite kube manifest files and there are no errors when applying to cluster. The caveat to this method being that the helm velero chart currently installs an older v1.1.0 version of velero.

Thank you.

[kzap]

nullable: true is a CRD feature in 1.14, I also have this error on 1.13 and it causes other issues around not being able to specify null because the new CRDs wont allow it.

Maybe for k8s < 1.14 we could get the older CRDs installed?

[cmcconnell1]

@kzap Thanks for the response and info. These particular clusters are EKS, so we are usually a few kube versions back due to AWS EKS release cycles. We are currently using helm templated install/configuration (from stable helm charts), but the caveat to that method is that it currently deploys an older version (1.1.0) of velero: velero stable helm chart

install/config via velero helm stable chart:

velero version
Client:
    Version: v1.2.0
    Git commit: -
Server:
    Version: v1.1.0

[skriss]

cc @prydonius

[cmcconnell1]

Hey All, just wanting to validate as @kzap noted above. We've upgraded the kube cluster to 1.4 and now we can use the documented velero binary template functionality to generate the large monolithic manifest file and apply to kube within our IAC code/scripts without errors.

script excerpt:

# generate the large monolithic kube manifest file from the velero install command with options
# ref: https://github.com/vmware-tanzu/velero-plugin-for-aws
velero install \
    --provider aws \
    --bucket ${VELERO_BUCKET_NAME} \
    --secret-file ./velero-credentials.secret \
    --plugins velero/velero-plugin-for-aws:${VELERO_AWS_PLUGIN_VERSION} \
    --backup-location-config region=$VELERO_AWS_REGION \
    --snapshot-location-config region=$VELERO_AWS_REGION \
    --prefix ClusterBackups \
    --dry-run -o yaml \
    --namespace velero \
> ./manifests/velero/velero.yaml

the generated manifest file is later applied with kubectl apply without errors with the nullable in the manifest file

egrep -ir 'nullable' ./manifests/velero/velero.yaml | wc -l
      50

Hope this helps/validates, etc.
Thanks

[prydonius]

@cmcconnell1 thanks for the info. I've tried to repro this in GKE on a 1.13 cluster and I didn't run into any issues (https://github.com/vmware-tanzu/velero/issues/2060#issuecomment-554160145). I'll try to reproduce this on AWS.

Also, just FYI, this PR should land soon and you'll be able to use the Helm chart to deploy Velero v1.2.0: https://github.com/helm/charts/pull/18492.

[cmcconnell1]

Interesting. Thanks for following-up and working on this @prydonius

[prydonius]

@cmcconnell1 I wasn't able to reproduce this on a v1.13.12 cluster on AWS using kops. Looks like you're using EKS, so I'll give that a try too.

$ velero install \
    --provider aws \
    --bucket $BUCKET \
    --secret-file _test/creds-aws \
    --backup-location-config region=$REGION \
    --snapshot-location-config region=$REGION --plugins velero/velero-plugin-for-aws:v1.0.0
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

$ k get po -n velero
NAME                      READY   STATUS    RESTARTS   AGE
velero-566d8f8bb5-phplb   1/1     Running   0          22s
$ velero version
Client:
    Version: v1.2.0
    Git commit: -
Server:
    Version: v1.2.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"a8b52209ee172232b6db7a6e0ce2adc77458829f", GitTreeState:"clean", BuildDate:"2019-10-15T12:04:30Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

[prydonius]

Ah, I just realised I've been letting the Velero CLI directly install everything. If I use dry-run and pipe the output to kubectl apply -f - I can reproduce the validation issue.

It seems like velero install ignores validation warnings from the API server and happily installs them. You can get the same effect with kubect apply -f - --validate=false, though this probably isn't ideal.

Let me take a look at where velero install is ignoring these warnings, and if it's possible for a mistake in our Velero resource definitions (https://github.com/vmware-tanzu/velero/tree/master/pkg/install) to incorrectly get applied without us noticing.

I'm not too concerned about this particular validation error, since the CRDs do seem to work in v1.13 once applied with --validate=false, and the CRDs are generated by kubebuilder/controller-tools.

[prydonius]

@cmcconnell1 any reason you are piping to kubectl apply instead of allowing the velero CLI to perform the install?

[prydonius]

Looks like velero install still catches more meaningful errors (e.g. an invalid pullPolicy in the Deployment), as this results in the API server returning an error. On the other hand, it looks like kubectl validates the input client-side against a schema before sending it to the API server (though it appears to use different schemas for different Kubernetes API versions).

Given the API server does a full validation anyway (https://github.com/kubernetes/kubernetes/issues/64830), I don't think velero install really needs to do this validation step, although it is confusing that the behaviour is different from kubectl apply. That said, given this won't affect 1.14+, I think we can simply document that --validate=false should be used in this case.

[cmcconnell1]

@cmcconnell1 any reason you are piping to kubectl apply instead of allowing the velero CLI to perform the install?

Our IAC patterns for everything possible is essentially:

1. create manifest files (dynamically)
2. kubectl apply
3. check newly generated kube manifest files into respective git repos.

I try to keep the anticipated code/model the same (i.e. avoid one-offs that might deviate from our model) to prevent confusion, continue to keep us as efficient as possible, etc.

I was unaware that this might have unexpected/undesired side effects with velero specifically. If it does, please let me know and I will definitely refactor if required. Thanks!

[prydonius]

@cmcconnell1 Understood, I figured it might be something like that. Is it possible in your pipeline for you to set --validate=false just for the Velero manifest? Also, is this a non-issue for you now given you've upgraded to 1.14, or are you still trying to deploy Velero on pre-1.13 clusters?

Also, the Helm chart now supports v1.2.0, so you could move to that to solve this as well.

[cmcconnell1]

@cmcconnell1 Understood, I figured it might be something like that. Is it possible in your pipeline for you to set --validate=false just for the Velero manifest?

We likely could, however, please see below.

Also, is this a non-issue for you now given you've upgraded to 1.14, or are you still trying to deploy Velero on pre-1.13 clusters?

Yes, we are unblocked having upgraded our kube clusters to 1.4.x

Also, the Helm chart now supports v1.2.0, so you could move to that to solve this as well.

We've moved forward with our IAC-based deployment scripts, based on the code excerpt noted in previous responses. Thanks again.

[prydonius]

@cmcconnell1 thanks for the info, glad this is resolved for you.

@kzap does the above workaround work for you, or is this still an issue for you?

[carlisia]

I see there are workaround for this. @prydonius do you think there's an action item for Velero in terms of either documenting or making a change to make this work with k8s < 1.14?

[prydonius]

@carlisia I'm not sure there's a great place to document this right now, but the workaround is if you're trying to do a velero install --dry-run -o yaml | kubect apply -f - you need to use the --validate=false for kubectl.

[skriss]

We could probably mention it in this section: https://github.com/vmware-tanzu/velero/blob/master/site/docs/master/customize-installation.md#generate-yaml-only

[prydonius]

Ah of course, yeah that would certainly be the best place to put that. Thanks @skriss!

[skriss]

moved this over to the release backlog as a nice-to-have documentation item.