Cert-manager: Cannot deploy CRDs to Kubernetes 1.11

[munnerz]

Describe the bug:

When deploying CRDs to Kubernetes 1.11, they fail to apply with validation errors: https://prow.build-infra.jetstack.net/view/gcs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/2199/pull-cert-manager-e2e-v1-11/1182598410243411968

The relevant error: must only have "properties", "required" or "description" at the root if the status subresource is enabled

This is because we specify type: object at the root of our OpenAPI schema: https://github.com/jetstack/cert-manager/blob/4ec682db20d2f030c5bbaac67c4f2f5301a7b081/deploy/manifests/00-crds.yaml#L1366

You can see in Kubernetes 1.11, this field is in fact not allowed: https://github.com/kubernetes/kubernetes/blob/9016740a6ffe91bb29824f80c34087b993903bd6/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation.go#L303-L306

We added it is a requirement to make our OpenAPI schema structural - discussing on #sig-api-machinery, it may be possible to relax the requirement to have this field in order for a schema to be structural, however it will require a bugfix/patch which will take some time to propogate through to cloud providers/managed service providers.

Expected behaviour:

We should either explicitly declare 1.11 as unsupported, or otherwise remove this field so that users can deploy to kubernetes 1.11.

For what it's worth, 1.11 is now EOL, and EKS for example is upgrading all 1.11 users to 1.12 in November: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

ref #2064

Steps to reproduce the bug:

Attempt to apply the deploy/manifests/00-crds.yaml file against a Kubernetes 1.11 cluster.

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.11
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): N/A
• cert-manager version (e.g. v0.4.0): v0.11.0, v0.10.x
• Install method (e.g. helm or static manifests): N/A

/kind bug
/area deploy

[JulienBreux]

Not good, but the temporary workaround is to add this argument --validate=false.

[munnerz]

This argument won't actually help in specifically Kubernetes 1.11's case - because it refuses to accept our CRD manifest as it sets the type: object field on our CRDs schema (as highlighted in the first post).

It's a real PITA, and after discussion with #sig-api-machinery I think it may be one that we can't work around without removing this field altogether, which will mean our CRD schemas are no longer 'structural' (which is a requirement for CRDv1).

Whilst we could remove this field and it will work for 1.11, I'm not sure if it's really worthwhile (and it means we're reverting progress forward). That said, I'm not too comfortable with the way we've discovered this, and the 'surprise' drop of support...

[bpetit]

Hi,

I had exactly this behavior on an EKS cluster:

Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.10-eks-7f15cc", GitCommit:"7f15ccb4e58f112866f7ddcfebf563f199558488", GitTreeState:"clean", BuildDate:"2019-08-19T17:46:02Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

and validate=false didn't help.

Is there a workaround for kubernetes 1.11 yet ? I'll migrate this cluster to a newer version, but it may take some time so I'd be more comfortable to fix my cert-manager deployment first.

Or did someone identify a version of the crds configuration file that is working at some point ?

Thanks

[yehaifeng]

I also this:
must only have "properties", "required" or "description" at the root if the status subresource is enabled

And i comment this:
`

subresources:

#status: {}
`
That is OK.

[depohmel]

I have the same error.

[aude]

This bit me as well.

It might be an unfortunate situation, but as far as I can see, cert-manager 0.11 is not compatible with Kubernetes 1.11.

Would it make sense to update this documentation? https://docs.cert-manager.io/en/release-0.11/getting-started/install/kubernetes.html

[grzechukol]

Hi, just saying that dropping support for Kubernetes 1.11 means that you also drop support for OpenShift v3.11 which is as for now used by many enterprises. Consider to revert this breaking change for the switchover period when enterprises will start using OpenShift 4.X which is Kubernetes 1.14 based.

[grzechukol]

@yehaifeng Did you checked impact of commenting subresources: status: {}?

This breaking change was introduced by "Move status to a CRD Subresource (#2097, @JoshVanL)"
@JoshVanL can you evaluate the impact of this modification?

[munnerz]

@grzechukol thanks for the explanation there - I'd not seen your message until just now!

This is unfortunate as it means we can't make use of 'structural schemas', which is what allows things like kubectl explain to work, as well as various other new CRD related features.

What is OpenShift's deprecation policy for versions, and how are they maintaining Kubernetes 1.11? Presumably they are back-porting fixes all the way back there... 😬

Disabling the status subresource will allow the components to deploy, but the controller itself will no longer work so you should not do this. Instead, removing this field from all CRDs (i.e. type: object at the root of the schema): https://github.com/jetstack/cert-manager/blob/e6bf2c0ace6cf2e674149147a292a22108722af1/deploy/manifests/00-crds.yaml#L42

As well as setting preserveUnknownFields: true (it is currently explicitly false) should make it possible to run on OpenShift 3.x. We could potentially look at removing these lines from the -openshift variants of our manifests automatically in order to maintain support for 3.x until it is deprecated.

[munnerz]

Hm, it seems that 3.11 will be deprecated in June 2022 (for 'full support'). This seems like a really long time for us to hinder the UX of users on 4.x and beyond... If we also want to continue to support 3.x, we'll probably need to ship multiple different OpenShift variants because of this issue 🤦‍♂ https://access.redhat.com/support/policy/updates/openshift

[jkroepke]

Works...

If you want a no brainer, just grab:

#!/bin/bash

# https://github.com/jetstack/cert-manager/issues/2200#L42
# ./install-crd.sh | oc create --validate=false -f -

curl -sL https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager-openshift.yaml | \
  sed -e 's/preserveUnknownFields: false/preserveUnknownFields: true/g' | \
  sed -e 's/type: object//g'

[jkroepke]

@munnerz

Since crds are inside the helm chart, its possible to add some conditions inside the crds? Like a hybrid chart for kubernetes and openshift?

[xrow]

I think @jkroepke made a wonderfull suggestion here. I also need to stick with OKD 3.11 for the next 1,5 years.

[tmerten]

It might only affect some German users but I'd like to throw this one in: I just tried to install cert manager to Open Telekom Cloud (OTC) which is also stuck on K8S K8s v1.11.7-r1-CCE2.0.23.B001. I already asked when they consider to upgrade but until then I will also need to apply a workaround to make this run.

When I run kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.11.1/cert-manager.yaml no challenges, orders, certificaterequests, certificates, clusterissuers, or issuers are created.

(Edit with the change from @jkroepke I was able to install cert manager 0.11.1 as well as 0.12 (using the k8s version like so curl -sL https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml | sed -e 's/preserveUnknownFields: false/preserveUnknownFields: true/g' | sed -e 's/type: object//g' > cert-manager-0.12.0-patched.yaml). Will report whether I can configure it to grab ACME certs on 0.12 and if that does not work whether it does with 0.11.1)

Edit2: I was able to install and use version 0.12 on the above mentioned OTC K8s cluster version v1.11.7-something with the above mentioned changes. I just received and deployed ACME certs successfully by pretty much following the standard tutorial, except that I used a cluster-issuer instead of an issuer.

[tamalsaha]

We recently came across this same issue in one of our projects. I think one solution is to use Capabilities in helm charts to drop the unknown fields.

Here is an example: https://github.com/kubevault/installer/blob/master/charts/csi-vault/templates/appbinding-crd.yaml#L33-L35

Edit: The files in the crds folder can't use templating. So, I think the best option is to create the crds from inside operator IMHO. We use gen-bindata to add the files in the operators and then create from inside. Then you can also easily handle the 1.11 restrictions. Here is the code we use: https://github.com/kmodules/client-go/blob/master/apiextensions/v1beta1/kubernetes.go#L20-L67

[Carl4]

I found that commenting out the following section broke things later in the deployment.

subresources: 
  status: {} 

Instead, I commented out the (somewhat redundant): type: object line inside each opernAPIV3Schema in the bundled cert-manager-openshift.yaml file before running oc apply -f cert-manager-openshift.yaml. This worked like a champ.

[jkroepke]

@munnerz

Openshift 3.11 receives some patches last weeks, e.g.: https://github.com/openshift/origin/pull/24540 + the patch is released for all enterprise users.

Maybe you can drop all the workarounds created in 0.14 (#2609) for the old openshift 3.11 and just support one CRD.

[munnerz]

Sadly that still is not possible - the full list of fields we have to 'transform' to create the 3.11 compatible CRDs can be found here: https://github.com/jetstack/cert-manager/blob/f3df90305c03c82c0ccf113d87cd6025de7eca3e/hack/filter-crd/main.go#L174-L186

Notably, we have to remove the new API version (v1alpha3) altogether, as well as a few other additional OpenAPI schema options (such as x-kubernetes-preserve-unknown-fields) to avoid users having to use --validate=false on all their kubectl calls (which is often not possible with GitOps tools).

I think we are stuck with supporting two variants of the CRDs for the foreseeable future - the earliest we could remove it is once Kubernetes v1.14 is no longer a supported platform 😞