Kube-prometheus: CustomResourceDefinition "prometheusrules.monitoring.coreos.com" is invalid

[yogeek]

What happened?

Sometimes, the following error appears when creating the CRDs :

kubectl apply -f kube-prometheus/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
The CustomResourceDefinition "prometheusrules.monitoring.coreos.com" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified

Did you expect to see some different?

customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created

How to reproduce it (as minimally and precisely as possible):
It does not seem to be the case each time but I bumped into the error after several attempts of this command (sometimes the CRDS already existed, sometimes not) :

kubectl apply -f kube-prometheus/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml

Environment

• Prometheus Operator version:
  v0.34.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

K8S cluster installed on AWS with kubeadm in HA mode

[yogeek]

_Maybe a hint_ :

In this doc it is noted that :

1. if metadata is specified, then only restrictions on metadata.name and metadata.generateName are allowed

In the CRD file, the metadata section contains neither generateName nor name by type :

validation:
    openAPIV3Schema:
      description: PrometheusRule defines alerting rules for a Prometheus instance
      properties:
        apiVersion:
          description: [...]
          type: string
        kind:
          description: [...]
          type: string
        metadata:
          type: object

[brancz]

cc @pgier

[elisiano]

I'm also seeing this

[pgier]

Looks like this is caused by the prometheus-operator deployment set to v0.34.0, and the CRD yaml files using the latest from prometheus-operator master. The CRD yaml files were updated to prometheus-operator master in https://github.com/coreos/kube-prometheus/commit/0be63d47fcf5f7f306a1f35292e691334ecfcdb8
but prometheus-operator also automatically creates CRDs. So the two versions are conflicting with each other.

You can reproduce the issue consistently by first deploying prometheus-operator and giving it a few seconds to create the CRDs.

kubectl apply -f manifests/setup/0namespace-namespace.yaml 
kubectl apply -f manifests/setup/prometheus-operator-clusterRole.yaml
kubectl apply -f manifests/setup/prometheus-operator-clusterRoleBinding.yaml
kubectl apply -f k apply -f manifests/setup/prometheus-operator-serviceAccount.yaml
kubectl apply -f manifests/setup/prometheus-operator-deployment.yaml

Note, that the CRDs will appear shortly.

kubectl get crd

Then apply the prometheusrules CRD to get the error.

kubectl apply -f kube-prometheus/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml

So why does the error only affect prometheusrules and not the other CRDs? Because prometheusrules is the only one that had openapi generation turned on for standard metadata. The other CRDs explicitly turned this off.
Compare
https://github.com/coreos/prometheus-operator/blob/8d44e0990230144177f97cf62ae4f43b1c4e3168/pkg/apis/monitoring/v1/types.go#L611
and
https://github.com/coreos/prometheus-operator/blob/8d44e0990230144177f97cf62ae4f43b1c4e3168/pkg/apis/monitoring/v1/types.go#L802

I'm thinking that we should just remove the CRD yaml files from the manifests/setup directory since these are created by prometheus-operator. Then they will be automatically updated when we update the version in the prometheus-operator deployment config. @brancz WDYT?

[elisiano]

@pgier that kinda solves the problem reported here but there is something else going on (which I don't know if it's related or not).
I have a prometheus cr which defines podMetadata (specifically annotations).
That yaml can be applied without errors but the annotations do not get applied, they are actually empty.

It's worth mentioning that before updating to latest master the annotations were propagated correctly.

[pgier]

@elisiano Would you mind creating an issue in prometheus-operator? If you can include your prometheus cr in that issue it would be helpful also.

[elisiano]

I think I found a workaround which I have no idea why it works.

• apply all manifests
• delete the operator pod
• apply again prometheus-prometheus.yaml

[brancz]

I think the best thing to do here is to actually pin the jsonnet dependency of the prometheus operator to the version we intend to use. The generated CRDs should be consistent with that. The reason the CRDs are part of the setup is so that people can quickly apply the prometheus/alertmanager/servicemonitor objects without having the race with the prometheus operator creating the CRDs.

[yogeek]

/reopen

[yogeek]

@brancz I just add this problem again on release-0.4 when trying to re-apply my prometheus operator deployment on a cluster in which it was already deployed.
It is odd because I did a deployment yesterday and all was ok. Is it possible that the latest commits introduced a regression ?

[yogeek]

@brancz FYI : a workaround was to delete the CRDS

kubectl delete crd prometheusrules.monitoring.coreos.com
customresourcedefinition.apiextensions.k8s.io "prometheusrules.monitoring.coreos.com" deleted

and to execute the deployment again...
It confirms that it is the update of the CRD that causes the issue

[bgdnlp]

Same here. Updated cluster to Kubernetes 1.17 (AWS EKS, but probably not relevant), pulled tag 0.4, applied setup manifests, got the invalid spec error. Deleting CRD fixes it.