Cert-manager: Allowing alternative Secret output formats (e.g. single .pem file priv/cert output)

[julien-hedoux]

/kind feature

When we describe the generated secret, we can see tls.key and tls.crt. But if we want to get the pem output, we have to concat both to obtain it.

The problem : If a container need a secret field (PEM file) as a mounted volume, we can't mount it from the secret api.
Could be great if the CertManager generate the PEM and store it as a key to the secret.

Now we have this :
yaml volumes: - name: volume-site-tls secret: secretName: site-secret-tls items: - key: tls.key path: certificate.key - key: tls.crt path: certificate.crt

Could be great to do this :
yaml volumes: - name: volume-site-tls secret: secretName: site-secret-tls items: - key: certiticate.pem path: certificate.pem

[munnerz]

Thanks for the request.

There have been similar requests for additional fields or alternative field names to be provided in the resulting secret resource.

My main gripe here, is that this creates extra complexity, and thus confusion for users, in how we should behave if a certificate resource is 'out-of-sync', i.e. if the secret is missing a certificate.pem key (because a user has migrated from an older version, or are supplying their own certificates during a migration).

There's numerous edge-cases to handle, and I'd like to know that you problem can't be otherwise solved before we introduce the complexity 😄.

Could you describe your use-case a bit further? For something the likes of Apache2, which does not (as far as I know) support having separate key and cert files, I see the following:

• You want to easily configure Apache2, which requires a concated pem file
• You want certificates to be rotated automatically when they change on disk/when they are renewed

In order to trigger Apache2 to actually reload the cert pem, you will need to provide some glue that sends a SIGHUP when the file changes.

If you have to create/configure this glue already, an additional piece that observes either the key or pem changing on disk and automatically calls cat tls.key tls.crt > certificate.pem is fairly trivial to add.

This ultimately achieves what you are looking for, and will still allowing inotify events etc to be fired, meaning you can easily trigger reloads in reaction to these files changing on disk 😄.

Perhaps your use case cannot be solved with this kind of solution? It'd be great to hear more 😄

[julien-hedoux]

Hi.

First, thank you for your reply.
Actually, it's not a piece of cake. I understand why you ask it before.

• From Apache/Nginx or other, you have to use key and cert. So you are right and we have all data we need.
• But for other, for a TLS proxy like Hitch, we need to get the PEM file (https://github.com/varnish/hitch/blob/master/docs/certificates.md) in the config.

Here, I would like to fetch pem from the Let's encrypt Issuer, but I didn't found it and how to get it.
So, as I usually do, and you told it, I concat key et cert to obtain the pem file. Without the CA cert, be I should do if I will have.

As I can't store it into the container into the dockerfile,

• I have to pass key et cert files as mounted volumes. Then
• When the container is starting, I concat them into a PEM file in a emptyDir volume
• I use it as PEM file in my Hitch configuration.

Here my actual config :
The container part
````

• name: wp-julienhedoux-fr-hitch
  image: gcr.io/poc-docker-87/hitch:1.4.6-15
  imagePullPolicy: Always
  volumeMounts:
  - mountPath: /ssl
  name: wp-julienhedoux-fr-pem
  - mountPath: /secrets
  name: wp-julienhedoux-fr-tls
  ports:
  - containerPort: 443
  `````

The volume part :

````

volumes:
- name: wp-julienhedoux-fr-pem
emptyDir: {}
- name: wp-julienhedoux-fr-tls
secret:
secretName: wp-julienhedoux-fr-tls
items:
- key: tls.key
path: tls.key
- key: tls.crt
path: tls.crt
Then In the docker entrypoint script, I do this :
cat /secrets/tls.key /secrets/tls.crt > /ssl/certificate.pem
````

And use finally the /ssl/certificate.pem as PEM into my hitch configuration.
I the cert manager was added the pem content into the k8s secret, as tls.pem for exemple. A simple mount will share the PEM from secret to the container. Instead of regenerated it ;)

An other way I tried, was to patch the secret by adding the pem into the secret. Like this :
kubectl patch secret wp-julienhedoux-fr-tls --patch "$(cat pem.yaml)"
With a yaml like this
data: tls.pem: _PEM_STRING_
but here, I got an error about base 64 encode string too long

Do you have a better way?

[ksemaev]

+1 I have to do exactly the same for my Haproxy, which is starting to be default ingress in Kuber, and require .pem format as well

[alexanderilyin]

Without it, it's a huge pain in the ass to make HAProxy (and all other software which needs a single pem file as a certificate) work with certificates generated by cert-manager.

[white-hat]

+1 it's a blocker for me too

[claudiocleberson]

Guys that would a great addition.
I have a Ejabbed cluster that uses a .pem key. I will probably have to implement a container to server the cert.pem. But I'd rather prefer to user the Cert-manager which is already serving a cert to my Traefik proxy backend.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[kbruner]

Maybe this could instead be implemented by allowing a "post-processing" step before the secret is created, if that's somehow simpler than specifying alternative output formats?

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[moritz31]

/remove-lifecycle rotten

We have a similar need for getting .pem files as output. Our use case is, that we are running opendistro for elasticsearch in a kubernetes cluster and do not want to build the pki ourself.
Since opendistro requires .pem files, we need to build a big chain on top of cert-manager, to convert the certificates to .pem format. That could be more easy if there is an option to output a .pem file instead of an .crt/.key file.

[munnerz]

This is something we'd like to support in cert-manager, but we're uncertain on how it should best be done.

If anyone has any insight on what we could do here, a few things at the front-of-mind:

• How does this play into alternate certificate formats? i.e. JWKs, DER encoded certs
• This ticket is actually requesting support for concatenating the two files we already produce, which is also slightly different once again.

Should these fields be related? How can we represent this in our API?

/lifecycle frozen

[bhagyas]

This is an issue when installing applications with built-in certificate management (eg. elasticsearch and xpack security).

Is there any known workaround to continue using cert-manager in such instances?

[jroper]

How does this play into alternate certificate formats? i.e. JWKs, DER encoded certs

There are multiple axes of encodings standards, there's PEM and DER, and there's PKCS. So you can have PEM encoded PKCS1, DER encoded PKCS8, PEM encoded PKCS8 etc. PKCS is more commonly referred to as the format (in some places its called the syntax), while PEM and DER are the ascii and binary encodings of whichever format you've chosen, and so are more commonly referred to as the encoding. That said, openssl just calls them all formats. It's really a mess.

If it were me, I this is what I would do:

• Rename keyEncoding to keyFormat.
• Create a new config, keyEncoding, to specify if the key should be PEM or DER encoded.
• For backwards compatibility, if keyFormat is not set, support PKCS8 or PKCS1 in the keyEncoding config, and treat that as the key format.

JVM apps, with the out of the box JDK crypto libraries, can only read keys in either PKCS8 DER encoding, or PKCS12 key stores (there's also JKS but let's just pretend that doesn't exist). PKCS12 key stores is in some cases more convenient, a PKCS12 file contains both the key and the cert chain, and you can make many of the standard network APIs in Java use your key store by setting a single system property. However, there are many cases where a PKCS8 DER encoded private key is necessary, for example, the postgres jdbc driver requires a PKCS8 DER encoded private key. So, I think it would be useful to add PKCS12 support. Note that if PKCS12 were used, you would probably call the output file keystore.p12, and have no other files, apart from perhaps the CA certificate.

[bergerx]

Along with #586, we keep getting hit by this. We are not comfortable about creating a PR since our knowledge about these is limited and we keep getting lost in the terminology used by openssl and java tools, i can see some of the reasons in @jroper's above post. I guess many others are in a similar status, so for so long we've not been getting a PR this :(

[kaceo]

Adding my support to the PEM format here. I am using HAProxy in docker (edge server outside cluster) and all the PEMs are in one directory. If I use Cert-manager I get a list of individual secrets, which I would need to copy individually, reload the HAproxy config, every time there is certificate events. I am too new to Kubernetes to know how to do this, perhaps if someone has sample code to show the strategy.

[fiberoptic89]

I am also requesting .PEM format support. I am using dovecot inside a docker image that runs inside a k8 pod.

[meyskens]

now we have certificate.spec.keystores we can start thinking about adding these under there

[meyskens]

We're taking on this for the v1.1 release, pkcs#12 and jks support already solves many of the usecases described above. Similar to this it would be nice to add "single PEM" support in something like:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: crt
spec:
  secretName: crt-secret
  dnsNames:
  - foo.example.com
  - bar.example.com
  issuerRef:
    name: letsencrypt-prod
  keystores:
    pkcs12:
      create: true
    cobinedPEM:
      create: true

using the logic of the already present alternate keyStores.

[GrandChaman]

Would it be possible to add support for DER encoded certificates and keys, to support postgres jbdc for mutual TLS ?

[meyskens]

@GrandChaman hey can you file a separate issue for that? that will make it easy to track and assign this!