Cert-manager: Server Key Usage by Default Breaks GRPC Certs

[dmizelle]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
After the merge of #2351, certificates generated with default settings break GRPC client<->server communication. This is specifically noticed with KIAM, which uses certificates to authenticate/encrypt communication via the client/server.

WARNING: 2019/11/27 18:29:07 grpc: Server.Serve failed to complete security handshake from "127.0.0.1:51648": tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage

Expected behaviour:
The same certificates used by previous versions of KIAM continue to work after being upgraded.

Since the apiGroup is changed from cert-manager.k8s.io -> cert-manager.io, the certificates are regenerated with incorrect Extended Key Usages for this situation.

Steps to reproduce the bug:

1. Deploy quay.io/jetstack/cert-manager-controller:v0.10.1
2. Create the following objects in your Kubernetes cluster:

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-ca
  namespace: kiam
spec:
  commonName: kiam-ca
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: selfsigning
  secretName: kiam-ca
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-server
  namespace: kiam
spec:
  dnsNames:
  - localhost
  - kiam-server
  ipAddresses:
  - 127.0.0.1
  issuerRef:
    name: kiam-issuer
  secretName: kiam-server
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-agent
  namespace: kiam
spec:
  commonName: agent
  issuerRef:
    name: kiam-issuer
  secretName: kiam-agent

1. Upgrade to the latest cert-manager, which requires changing API groups.
2. Tail the logs of the kiam-server pod and notice that communication is now broken.

Anything else we need to know?:

Environment details::

• Kubernetes version 1.14.7
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS
• cert-manager version (e.g. v0.4.0): v0.10.1 / latest
• Install method (e.g. helm or static manifests): static manifests

/kind bug

[munnerz]

The solution here would be to add:

spec:
  ...
  keyUsages:
  - digital signature
  - key encipherment
  - server auth
  - client auth

explicitly to your Certificate resources.

Setting server auth by default is required for Safari/Mac OSX Catalina to properly validate a certificate for serving (else it'll be marked untrusted), in your case, client auth is the exception here and you should explicitly denote it as a keyUsage when requesting your Certificate 😄

[stefansedich]

Just bit by this one myself! I guess we should look at updating the cert-manager docs for new players so they don't chase their tail or hadbutt any walls, I will go do that once I sit down and understand the issue a little more.

[jsharpe]

The solution here would be to add:

spec:
  ...
  keyUsages:
  - digital signature
  - key encipherment
  - server auth
  - client auth

explicitly to your Certificate resources.

Setting server auth by default is required for Safari/Mac OSX Catalina to properly validate a certificate for serving (else it'll be marked untrusted), in your case, client auth is the exception here and you should explicitly denote it as a keyUsage when requesting your Certificate 😄

For anyone using this the keyUsages: key should be usages:

[jsharpe]

I finally resolved my issues with this and the fix was to set the key usages for the CA key as it was picking up the wrong values. Seems like a logic error somewhere

[jsharpe]

@munnerz has this been fixed in cert-manager? CA certs should work out the box without needing to specify usages for keys

[munnerz]

I am not sure if it’s normal behaviour to set clientAuth by default on certificates. Most things I’ve seen default to enable serverAuth. The breaking change here came about because if neither is specified, both are often assumed to be usable by TLS clients.

The CA used to sign the certificate should not require the clientAuth usage in order to create client certificates.

Requiring users to explicitly request clientAuth is intended behaviour and I don’t think we should change this.

[jsharpe]

The issue I was seeing though is that requesting a CA certificate actually generated a cert without the signing usage but with clientAuth. It was as if the isCA flag was being ignored.

I don't think requiring uses to request cert sign and crl sign on a CA is an intended behaviour for users as this is what I had to do to fix this issue using with kiam. @dmizelle @stefansedich did you come across the same problem?

[munnerz]

Would you be able to put together a simple reproduction to show/explain
what you’re seeing? Setting isCA as you note should result in a certificate
with ‘cert sign’ and ‘crl sign’, as well as any other usages the user has
explicitly specified (or if none were explicitly specified, then the
defaults of digital signature, key encipherment and serverAuth.

On Thu, 20 Feb 2020 at 14:21, James Sharpe [e-mail] wrote:

The issue I was seeing though is that requesting a CA certificate actually
generated a cert without the signing usage but with clientAuth. It was as
if the isCA flag was being ignored.

I don't think requiring uses to request cert sign and crl sign on a CA is
an intended behaviour for users as this is what I had to do to fix this
issue using with kiam. @dmizelle https://github.com/dmizelle
@stefansedich https://github.com/stefansedich did you come across the
same problem?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/2407?email_source=notifications&email_token=AABRWP75ZSYQI446J2O6VTTRD2GXRA5CNFSM4JSK7FVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMOJFWQ#issuecomment-589075162,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABRWP2KL7HOUFUBHE5FAZ3RD2GXRANCNFSM4JSK7FVA
.

[jsharpe]

It was literally the config in the original post that I was seeing this with. The CA certificate was being generated with the wrong usages - I don't recall exactly what it was ending up with but I remember it had at least the serverAuth usage.

[munnerz]

Creating these resources:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ca
spec:
  commonName: root-ca
  isCA: true
  issuerRef:
    name: selfsigned
  secretName: root-ca

Creates a certificate with:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE

If I explicitly add:

usages:
- key encipherment
- digital signature
- client auth

to the certificate then it gets:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE

which all seems like it is behaving as expected to me, but perhaps I'm misunderstanding still 😄

[jsharpe]

I think the issue here was that GRPC doesn't see this as a valid CA certificate for securing the connection.
Its not really best practice to have any extensions other than crl sign and cert sign for a CA certificate - surely cert-manager should default to a minimal set of extensions (as it previously did) and then require users to opt in via usages for any other uses?

[johannwagner]

We ran into the same problem with in context with WAMP protocol.
So, I think, everything, which does TLS Client Authentication with go is affected.

[jessedearing]

Another gRPC user here.

What has triggered this behavior for those of us using mTLS with frameworks like gRPC, is specified in the Internet X.509 Public Key Infrastructure Certificate spec: RFC 5280.

In RFC 5280 under the section "Extended Key Usage" it states:

If the extension is present, then the certificate MUST only be used for one of the purposes indicated.

This indicates that if no extended key usage is present then it can be used for client auth or server auth. By specifying server auth as a default in #2351 we now have an extended key usage that says the certs created can only be used for the purpose of server auth and not client auth.

This prevented an upgrade from 0.11 to 0.13 for us and it would be really inconvenient to have to now specify usages on all our certificates.

Since #2351 was a fix for temporary certificates, can we add the usages to the temporary certificates instead of changing the default?

[agusti-t]

Got bitten by this as well.

[jsharpe]

@munnerz @JoshVanL Can this be reopened as this clearly needs addressing...

[egoon]

I also have this problem with gRPC... Pleas reopen and fix this issue!

[wallyqs]

The workaround that we adopted in NATS on K8S for when cert-manager version is > v0.11.0 is the following: https://github.com/nats-io/nats-operator/pull/255/files

[dhermes]

I was digging into the "why" on this today and was surprised to see how (A) the Go standard library validates the non-leaf parts of the chain and (B) how the gRPC Go implementation sets expected key usages for the non-leaf parts of the chain:

• https://github.com/golang/go/blob/go1.15.6/src/crypto/x509/verify.go#L815-L819
• https://github.com/grpc/grpc-go/blob/v1.34.0/security/advancedtls/advancedtls.go#L480-L489