Schema-registry: Producer with Schema Registry connection not using ssl settings

Created on 27 Nov 2018 · 11Comments · Source: confluentinc/schema-registry

I've got a spring boot app that produces json messages without using a schema registry, this works fine.

Now I am attempting to add a new kafka producer that will produce Avro messages using a confluent schema registry. I set up the producer properties the same way for both producers:

properties.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, securityProtocol);
properties.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, keystorePath);
properties.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, keystorePassword);
properties.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, truststorePath);
properties.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, truststorePassword);
properties.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, keystorePassword);

However the avro kafka producer doesn't seem to be respect the ssl setting with regard to connecting to the schema registry. Doing a kafka send with the avro producer gives me this stack trace:

org.apache.kafka.common.errors.SerializationException: Error serializing Avro message

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:142)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:187)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:238)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:230)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:225)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:59)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:91)
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:72)
    at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:54)
    at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:65)
    at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:55)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:783)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:768)
    at com.mycompany.service.KafkaPocIntegrationSpec.test - send avro message(KafkaPocIntegrationSpec.groovy:55)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
    ... 26 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 32 more

Following some advice from this page (near the bottom they are experiencing the same issue I'm reporting here):
https://groups.google.com/forum/#!topic/confluent-platform/i2N22ZYpplA

I was able to set system properties (note that these are system properties not producer properties):

System.setProperty("javax.net.ssl.trustStore", truststorePath);
System.setProperty("javax.net.ssl.trustStorePassword", truststorePassword);
System.setProperty("javax.net.ssl.keyStore", keystorePath);
System.setProperty("javax.net.ssl.keyStorePassword", keystorePassword);

After setting these system properties the producer works just fine. However his seems like a bug to me, I shouldn't have to do this (it negatively impacts other aspects of my app). The producer is not using the kafka ssl config to talk to the schema registry, instead it's using the jvm default ssl settings.

Source

madcap

👍6

Most helpful comment

Bumping this for some answers of how to resolve this as I have the same issues, I have Kafka on SSL and Schema registry using HTTPS + Avro, it works when schema is on http but not https.

Any updates on this?

sqilz on 4 Dec 2019

👍6

All 11 comments

I can recreate the same behavior from kafka-avro-console-producer

$ cat ~/ssl.config 
security.protocol=SSL
ssl.truststore.location=/path/to/my/truststore/here
ssl.truststore.password=password
ssl.keystore.location=/path/to/my/keystore/here
ssl.keystore.password=password
ssl.key.password=password

kafka-avro-console-producer --broker-list my.brokers.here --topic my.topic.here --producer.config ~/ssl.config --property value.schema='{"type" : "record", "name" : "Example", "namespace" : "com.example", "fields" : [ {"name" : "uniqueId", "type" : [ "null", "string" ]} ]}' --property my.schema.registry

Enter value: {"uniqueId" : {"string":"1234567"}}

StackTrace:

{"uniqueId" : {"string":"1234567"}}
[2018-11-27 10:11:41,415] ERROR Failed to send HTTP request to endpoint: my.schema.registry.here (io.confluent.kafka.schemaregistry.client.rest.RestService:176)
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:172)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:229)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:320)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:312)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:307)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:115)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:154)
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79)
    at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:181)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:59)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
    ... 23 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 29 more
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:172)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:229)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:320)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:312)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:307)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:115)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:154)
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79)
    at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:181)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:59)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
    ... 23 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 29 more

Now try with system properties set:

$ SCHEMA_REGISTRY_OPTS="-Djavax.net.ssl.keyStore=/path/to/my/keystore/here -Djavax.net.ssl.trustStore=/path/to/my/truststore/here -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStorePassword=password" kafka-avro-console-producer --broker-list my.broker --topic my.topic --producer.config ~/ssl.config --property value.schema='{"type" : "record", "name" : "Example", "namespace" : "com.example", "fields" : [ {"name" : "uniqueId", "type" : [ "null", "string" ]} ]}' --property schema.registry.url=my.schema.registry
{"uniqueId" : {"string":"1234567"}}
^C
$

No issues when setting the vm system properties.

madcap on 27 Nov 2018

I can confirm this problem: when using kafka-rest with schema-registry on SSL, we need to set the javax.net.ssl.* properties at the JVM level to be able to authenticate on schema-registry.

On some applications, this can be problematic as we may want the JVM to use one keystore to serve request but use another to call schema-registry

Having some specific properties for SSL for schema-registry client would be nice, or at least use the SslConfigs.SSL_* ones.

zigarn on 22 Jan 2019

We can confirm this problem too.

The AVRO Serializer connects to the schema registry using SSL/TLS and the JVM keystores / truststores.
This is different from what the rest of the Kafka clients (consumer, producer, admin, …) do: They use the keystore and truststore set by the kafka parameters (ssl.keystore… / ssl.truststore… ).

The AVRO serializer should have its own properties for specifying a key- and truststore or it should just use the standard client parameters for that.

sergeiwaigant on 29 Jan 2019

@zigarn @sergeiwaigant As discussed the SR client current relies on the JVM level properties to set the SSL configuration. There is an outstanding PR that should address this https://github.com/confluentinc/schema-registry/pull/957. Let me know if you have any thoughts on the PR.

mageshn on 5 Feb 2019

👍2

Is this making any progress? As I am stuck on this issue.

unixunion on 11 Oct 2019

👍3

Now that #957 is merged is there any chance this will make it into 5.4-beta or is that closed to new features now?? We are also impacted by this issue now. We do have a workaround by running our client app in a new jvm but its extra overhead as far as code/deployment/etc goes.

cah-nathanzender on 21 Oct 2019

Bumping this for some answers of how to resolve this as I have the same issues, I have Kafka on SSL and Schema registry using HTTPS + Avro, it works when schema is on http but not https.

Any updates on this?

sqilz on 4 Dec 2019

👍6

Hi All,

Tried to configure Schema registry with SSL and basic auth. When attempted to use with AVRO Producer, it failed with error "No SAN Name found". I had configured to ignore the host name checking. but still didnt work. Similar configuration had worked on KAFKA REST, CONNECT and KSQL.

ALso tried to run query from KSQL which uses schema registr. Failed with same error that No SAN Name found.

Below were configuration on Schema Registry for HTTPS;

listeners=https://0.0.0.0:8081
ssl.keystore.location=/confluent-5.5.0/cert/kafka.server.keystore.jks
ssl.keystore.password=password
ssl.key.password=password

On avro producer and KSQL i had configured truststore and specific ssl.endpoint.identification.algorithm=

Can you please help confirm if this is the same open issue we have here?