Kubevirt: Upgrade from v0.30.4 to 0.33.0-rc.0 fails

[nunnatsa]

/kind bug

What happened:
In HCO, Bumping KubeVirt to version v0.33.0-rc.0 was failed. The PR is here: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/774

The failure was in the upgrade-prev tests, that upgrades HCO from previous version to the current one. HCO failed to complete the upgrade process because kubevirt reports it's degraded. The full error message in virt-operator's Degraded condition is:

KubeVirt is degraded: An error occurred during deployment: unable to
      create crd &CustomResourceDefinition{ObjectMeta:{virtualmachineinstances.kubevirt.io      0
      0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:kubevirt
      app.kubernetes.io/managed-by:virt-operator kubevirt.io:] map[kubevirt.io/install-strategy-identifier:33adf21bf38aeba200a1636c6cfd592a2d3262b3
      kubevirt.io/install-strategy-registry:docker.io/kubevirt kubevirt.io/install-strategy-version:v0.33.0-rc.0]
      [] []  []},Spec:CustomResourceDefinitionSpec{Group:kubevirt.io,Version:v1alpha3,Names:CustomResourceDefinitionNames{Plural:virtualmachineinstances,Singular:virtualmachineinstance,ShortNames:[vmi
      vmis],Kind:VirtualMachineInstance,ListKind:,Categories:[all],},Scope:Namespaced,Validation:nil,Subresources:nil,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1alpha3,Served:true,Storage:true,Schema:nil,Subresources:nil,AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},},},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{CustomResourceColumnDefinition{Name:Age,Type:date,Format:,Description:,Priority:0,JSONPath:.metadata.creationTimestamp,},CustomResourceColumnDefinition{Name:Phase,Type:string,Format:,Description:,Priority:0,JSONPath:.status.phase,},CustomResourceColumnDefinition{Name:IP,Type:string,Format:,Description:,Priority:0,JSONPath:.status.interfaces[0].ipAddress,},CustomResourceColumnDefinition{Name:NodeName,Type:string,Format:,Description:,Priority:0,JSONPath:.status.nodeName,},CustomResourceColumnDefinition{Name:Live-Migratable,Type:string,Format:,Description:,Priority:1,JSONPath:.status.conditions[?(@.type==''LiveMigratable'')].status,},CustomResourceColumnDefinition{Name:Paused,Type:string,Format:,Description:,Priority:1,JSONPath:.status.conditions[?(@.type==''Paused'')].status,},},Conversion:nil,PreserveUnknownFields:nil,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}:
      customresourcedefinitions.apiextensions.k8s.io "virtualmachineinstances.kubevirt.io"
      already exists'

What you expected to happen:
Kubevirt v0.33.0-rc.0 will report Degraded=False and will complete the upgrade from v0.30.4

Anything else we need to know?:

Environment:

• KubeVirt version (use virtctl version):
• Kubernetes version (use kubectl version):
• VM or VMI specifications:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[davidvossel]

/release-blocker release-0.33

[davidvossel]

yeah... so we broke this pretty bad. The label we key off of for our informers somehow changed between versions.

previously our informers keyed off of app.kubernetes.io/managed-by: kubevirt-operator

now they key off of app.kubernetes.io/managed-by: virt-operator

I'm tracking this down still to gain a better understanding of how this went unnoticed and what versions it impacts.

[davidvossel]

to fix this, our informers need to filter on both app.kubernetes.io/managed-by: kubevirt-operator and app.kubernetes.io/managed-by: virt-operator.

It looks like label selectors are capable of expressing this logic. For example.

kubectl get crds -l 'app.kubernetes.io/managed-by in (virt-operator, kubevirt-operator)'

I'm working on a fix now.