Kubeapps: Review being able to upgrade apprepository CRDs

[migmartri]

Currently, if we update the appRepo CRD template in the chart but the resource already exists globally in the cluster, it will not get updated.

We should find a way to apply chart updates to the crd spec even if it's installed. This might mean that we need to do it as a k8s job during install or upgrade.

[prydonius]

see also #803

[andresmgot]

The good news is that it's fairly simple to upgrade versions in CRDs and support deprecated versions without implying breaking changes. Actually the documentation is quite good for that:

https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/

The only thing we would need to do is add a versions: field in the CRD template adding both versions and setting v1 as "default". Existing apprepositories created with the version v1alpha1 will keep working as usual. Apart from that we would need to change the version in the controller to listen for the version v1 instead v1alpha1.

The bad news is that the versions field of a CRD is only available in Kubernetes +1.11. If we want to change the version to v1, users running Kubernetes 1.10 would need to do similar steps than when upgrading to Kubeapps v1.0.0:

# Backup resources
kubectl  get apprepository -n kubeapps <name> > <name>.yaml
# Change the version in the backup
sed -i '' 's/kubeapps.com\/v1alpha1/kubeapps.com\/v1/g' <name>.yaml
# Delete Kubeapps
helm del --purge kubeapps
# Delete the old CRD version
kubectl delete crd apprepositories.kubeapps.com
# Install the new version (with the new CRD)
helm install --name kubeapps  --namespace kubeapps bitnami/kubeapps
# Restore the resources
kubectl apply -f <name>.yaml

Those steps are a bit disrupting and users won't get any benefit from it so in my opinion we should wait until we drop support to k8s 1.10, whenever we introduce changes in the apprepository definition or the next time we introduce breaking changes in the chart.

NOTE: If we update the apprepository fields but we don't change the version we could avoid breaking changes as far as the changes are backward compatible (like adding a new optional field) since that is handled by the controller, it's not included in the CRD spec.

I am going to send a PR for the changes updating the version (the ones I used for running tests).

[prydonius]

Thanks for doing the research @andresmgot, that's pretty good to know! I agree that we should wait until we drop 1.10 support to update this.

Can you explain a bit more about how Helm handles this upgrade? From what I understand, the crd-install hook won't update the CRD. Does the operator need to manually apply the new CRD?

[andresmgot]

Can you explain a bit more about how Helm handles this upgrade? From what I understand, the crd-install hook won't update the CRD. Does the operator need to manually apply the new CRD?

Yes, it doesn't handle it. Doing an apply would only work if it's performed before doing the upgrade. Note that the condition that we have right now in the CRD template would change to if not (.Capabilities.APIVersions.Has "kubeapps.com/v1") so if someone tries to upgrade while the previous version is installed it will try to reinstall the CRD and it will fail:

Error: UPGRADE FAILED: [unable to recognize "": no matches for kind "AppRepository" in version "kubeapps.com/v1", unable to recognize "": no matches for kind "AppRepository" in version "kubeapps.com/v1", unable to recognize "": no matches for kind "AppRepository" in version "kubeapps.com/v1", unable to recognize "": no matches for kind "AppRepository" in version "kubeapps.com/v1"]

So it would be needed to either reinstall Kubeapps or apply the new version of the CRD before the upgrade.

[prydonius]

Thanks, this sounds like a pretty big breaking change then. We may want not want to change anything until v2.0.

[migmartri]

Thanks @andresmgot for the detailed analysis, it's very useful :)

[absoludity]

We no longer support k8s 1.10, I'm not sure this is still relevant, nor has it ever been required yet (but may be soon if there are changes related to #2078.