Librdkafka: Provide options for SSL CA as string

[gaffneyc]

Description

We're using Heroku's hosted Kafka service which provides client cert, client private key, and the certificate authority that signed those certs as environment variables. With the release of 1.1.0 we can pass the client certificates as strings but the CA still has to be a file (which forces us to pass all certificates as files).

There was a discussion on the PR about this with no one having a good reason to add it. We would like to see it added as it lets us avoid writing the certificates to files.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka version (release number or git tag): 1.1.0
• [x] Apache Kafka version: 2.1.0
• [x] librdkafka client configuration: ssl_ca=ENV["KAFKA_TRUSTED_CERT"] ssl
• [x] Operating system: Gentoo Linux
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

You can set an in-memory CA cert with:
rd_kafka_conf_set_ssl_cert(conf, RD_KAFKA_CERT_CA, RD_KAFKA_CERT_ENC_PEM, your_ca_pem, strlen(your_ca_pem), errstr, sizeof(errstr))

[gaffneyc]

I'm using (or at least experimenting) with the ruby bindings so I don't have access to rd_kafka_conf_set_ssl_cert directly.

If you think it's better handled in the bindings rather than librdkafka we can close this issue and I'll look at adding it there.

[dstelljes]

Adding a PEM configuration option (ssl.ca.pem) would be beneficial for us as well. We’re using dynamically provisioned certificates and need to pass the issuing CA to verify the entire chain.

[jgada-asapp]

If coming from C++,

auto conf_result = rd_kafka_conf_set_ssl_cert(conf->c_ptr_global(), RD_KAFKA_CERT_CA, RD_KAFKA_CERT_ENC_PEM, ssl_ca_str.c_str(), ssl_ca_str.size(), errstr, sizeof(errstr))

[mxk1235]

@edenhill can CA PEM be provided in ssl_ca property?

[edenhill]

@mxk1235 Yes. We should document it properly, but here's the underlying code:
https://github.com/edenhill/librdkafka/blob/df6bcbf11e1bd9ed53423514de30d59dbe5fac07/src/rdkafka_cert.c#L160

[justinfx]

@edenhill , since the confluent_kafka python client is a binding on librdkafka, can you confirm that the "ssl.ca.location" can be set to string data instead of a file? It seems to result in a KafkaError invalid arg, where it is trying to open it as a file.

Edit: After dumping the properties from kafkacat I realised that "ssl_ca" was still an undocumented property, and I learned about "ssl.certificate.pem" and "ssl.key.pem". Unfortunately it looks like at least through the python bindings or kafkacat, "ssl_ca" errors out with "Internal property 'ssl_ca' not settable"

[alex-damian-negru]

@justinfx Indeed, I've the same issue on my ruby binding through WaterDrop.

Rdkafka::Config::ConfigError: Internal property "ssl_ca" not settable

So the only way we can specify this setting is through a file? Bummer.

[sunnybenari]

@edenhill any plan to add a PEM configuration option (ssl.ca.pem) as @dstelljes suggested?

[igor-chernikov]

Adding separate config for CA PEM would be very useful for us as well