Kube-prometheus: CRDs missing `x-kubernetes-int-or-string` makes CRDs unapplyable on existing setups on 1.15+ clusters

[ari-becker]

What happened?

Cannot idempotently apply the CRDs on an existing cluster with an existing setup, I get the following error (in the case of alertmanager):

The CustomResourceDefinition "alertmanagers.monitoring.coreos.com" is invalid: 

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[postStart].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[postStart].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[postStart].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[postStart].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[preStop].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[preStop].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[preStop].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[lifecycle].properties[preStop].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[livenessProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[livenessProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[livenessProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[livenessProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[readinessProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[readinessProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[readinessProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[readinessProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[startupProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[startupProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[startupProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[startupProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[postStart].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[postStart].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[postStart].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[postStart].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[preStop].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[preStop].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[preStop].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[lifecycle].properties[preStop].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[livenessProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[livenessProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[livenessProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[livenessProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[readinessProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[readinessProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[readinessProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[readinessProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[startupProbe].properties[httpGet].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[startupProbe].properties[httpGet].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[startupProbe].properties[tcpSocket].properties[port].anyOf[0].type: Forbidden: must be empty to be structural

* spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[startupProbe].properties[tcpSocket].properties[port].anyOf[1].type: Forbidden: must be empty to be structural

Did you expect to see some different?
No error.

How to reproduce it (as minimally and precisely as possible):

Apply the CRDs on a cluster running Kubernetes 1.15 or later.
This is a known bug, see: https://github.com/kubernetes-sigs/controller-tools/issues/304
the x-kubernetes-int-or-string: true field needs to be added where appropriate.

Environment

• Kubernetes version information:

1.15.6

• Manifests:

https://github.com/coreos/kube-prometheus/blob/1d1ce4967f65f4f2612ff00ec000cb6b40d76720/manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
^ this is currently what master points to

[brancz]

cc @pgier

[ari-becker]

I think I found my fix, which is not to try to apply the CRDs myself but to allow the prometheus-operator to create the CRDs for me. Reading through the issues, this now seems like the expected behavior (we were trying to upgrade from 0.32.0).

Before, our application script looked like:

1. Apply CRDs
2. Apply Prometheus Operator yamls
3. Apply Prometheus objects, ServiceMonitor objects, etc.

Now our application script looks like:

1. Apply Prometheus Operator yamls
2. Wait for CRDs to be available
3. Apply Prometheus objects, ServiceMonitor objects, etc.

I'll close this issue after I confirm that the upgraded operator updates our CRDs in production.

[ari-becker]

No, the upgraded operator isn't updating the CRD in-place, so I'm leaving this issue open.

I'm wondering if this is related / causing a separate issue we're currently having, whereby trying to kubectl edit prometheus/kube-prometheus and change spec.podMetadata results in the Kubernetes API server claiming that the edit successfully configured the object, but immediately running kubectl get prometheus/kube-prometheus -oyaml shows podMetadata: {} in the YAML (i.e. that the changes are discarded).

[brancz]

@pgier the later thing sounds like it might be the the 1.17 bug we’re experiencing.

[a8j8i8t8]

I'm also facing this issue on K8S with below version.

❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-23T14:21:54Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"1bea6c00a7055edef03f1d4bb58b773fa8917f11", GitTreeState:"clean", BuildDate:"2020-02-11T20:05:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[a8j8i8t8]

Is there any work-around for this at the moment?

[a8j8i8t8]

Just an update, I used kube-prometheus commit eee5e10 and podMetadata is picked up by both Prometheus as well as AlertManager.

[pacoxu]

@a8j8i8t8 set preserveUnknownFields: true

Default value of preserveUnknownFields is false.

[ari-becker]

Stale, the current CRDs do have x-kubernetes-int-or-string so this isn't an issue anymore