Confluent-kafka-dotnet: SchemaRegistry with SSL and certificate

[alfhv]

Description

I have a kafka+schemaregistry both with strong SSL authentication (certificate)
Kafka SSL setup is well documented and works but when trying to communicate with schema registry URL my consumer and producer are failing.

is this possible ?
there is any doc or sample available ?

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [ 1.2.0] Confluent.Kafka nuget version.
• [ ] Apache Kafka version.
• [ ] Client configuration.
• [ RedHat] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[mhowlett]

this is currently not supported by the .net client. it's a fairly common request - we do want to do it. i had a quick look into it, and it seems pretty doable, but i'm not familiar with the SSL APIs enough to do it quickly. pull requests definitely accepted on this one!

[rlanhellas]

Guys, I'm working on this Issue, creating the possibility to connect in Schema Registry using SSL args. When everything is done and tested I will create a Pull Request.

[alfhv]

I have a solution working already, but it is not possible for me to push the code as it is inside my client's private network. I could reproduce it but I needs a proper kafka+schemaregistry setup with SSL and certs which could take some more time to me, sorry...

I explain the idea I have used:

• Right now we are setting the path to cert and private key for librdkafka as two separate files. But for dotnet class X509Certificate2, the only way I found so far to deal with cert+privkey is using a PFX file: cert and key merged in one single file using openssl.
• Then I have added an extra key to config with path to PFX file.
• In code, I have a new constructor for RestService as:

public RestService(string schemaRegistryUrl, int timeoutMs, string certfificatePath)
{         
    var webRH = new WebRequestHandler();
    webRH.ClientCertificates.Add(new X509Certificate2(certfificatePath, "", X509KeyStorageFlags.DefaultKeySet)); // second param is the priv key password, to add if need

    this.clients = schemaRegistryUrl
        .Split(',')
        .Select(SanitizeUri)// need https here
        .Select(uri =>
        {
            var client = new HttpClient(webRH) { BaseAddress = new Uri(uri, UriKind.Absolute), Timeout = TimeSpan.FromMilliseconds(timeoutMs) };
             return client;
        })
        .ToList();
}

Then add an extra code on constructor of CachedSchemaRegistryClient() to read the PFX path from config and call the appropiate constructor of RestService.

I'm trying to build the PFX file on the fly in code from cert and privkey. I found an interesting code here that could do the job:
https://github.com/StefH/OpenSSL-X509Certificate2-Provider/blob/master/src/OpenSSL-X509Certificate2-Provider/CertificateFromFileProvider.cs
didnt try yet.

waiting to see the solution from rlanhellas...

[rlanhellas]

@alfhv thanks for help, will be very useful. I will keep you up-to-date.

[dinegri]

I have been working on a ssl implementation for schema registry, as soon as possible I will contribute, possible in january

[mhowlett]

thanks for the input everyone, this is promising!

[acoca2]

Hi All,

Checking in to see what progress has been made on this.

We have a situation in our organization where we have a subset of applications using the confluent-kafka-dotnet API so we're unable to implement https/client auth with Schema Registry at the moment. Have thought about moving to BASIC AUTH in the meantime, but curious what the potential time line is looking on this support.

Thanks!
Anthony

[dinegri]

Hi All,

I have just opened a PR

Let's see how it goes

[mhowlett]

thanks @dinegri - we'll target the release after v1.4, which will also include a bunch of other changes related to schema registry. after a quick glance, looks good!

[sarwarbhuiyan]

Can this be closed now that it has been merged?

[mhowlett]

yep