Kafka-connect-jdbc: JDBC Source connector does not work with externalization of secrets

[satyaelite]

I tried to externalize the connection.user and connection.password properties in the JDBC source connector. I used the Kafka's FileConfigProvider to test this approach and found out that the connector properties are not getting replaced when the initial attempt to establish a JDBC connection is done. This leads to failure in connection. I could replace other properties in the connection property file using the FileConfigProvider though.

Please note, the replacement of externalized properties are working AFTER the connection is established, if I use it for non connection related properties.

[mihbor]

I believe this has been addressed in KAFKA-7225

[satyaelite]

The issue is how the kafka jdbc connector uses the configuration properties. Per what I observed, jdbc connector is trying to make a connection even before the properties have been parsed using the external provider. I am specifically pointing to JdbcSourceConnectorConfig.TableRecommender class.

[adityajalkhare]

Hi @satyaelite I'm also facing the same issue. Did you find any solution yet?

[szicari-farmobile]

I've been trying for a couple days to make this work. The sink connector works w/ externalized secrets - the source connector does not.

I get an error connecting to the database: password authentication failed.

[szicari-farmobile]

Sorry to bug on this issue, but I would love to know if someone at Confluent can address this. I tried cloning the repo and getting a dev environment set up to attempt to fix this myself, but that brought me into the deep end. The build procedures for Kafka and Connect failed at various stages, and I do not have time to work through them at the moment.

[Yosafat1997]

Sorry to bug on this issue, but I would love to know if someone at Confluent can address this. I tried cloning the repo and getting a dev environment set up to attempt to fix this myself, but that brought me into the deep end. The build procedures for Kafka and Connect failed at various stages, and I do not have time to work through them at the moment.

This is happen for me too #911 . I found there's a problem about depedency issue on kafka since i found the forked kafka from orginal Apache Kafka are unsynchronized. I also ran out of time since i need to develop this connector to provide array data for postgres. Somebody have made it but unfortunately it wasn't merged

[C0urante]

Hi all--this sounds like KAFKA-7225, which caused connector config validation to fail in some cases when external config providers were used on the Connect worker. It's been fixed in Kafka versions 2.0.1, 2.1.0, and all major/minor releases from that point on.

If anyone is still experiencing this issue, please try to reproduce on a version of the Connect framework that should have the fix for KAFKA-7225. If you are able to reproduce on one of those versions, please provide logs (preferably at DEBUG or TRACE level) for your Connect worker(s) and let us know which version of the Connect framework you're running, and I'll be happy to take a look.

I'll leave this open for a week or so to give everyone a chance to try to reproduce; if I don't hear anything after then I'll close the issue to signal to the community that this issue appears to be solved.

[szicari-farmobile]

@C0urante This is awesome, thanks for the reply! I'm unclear on Kafka / Confluent version schemes and have been tripped up by it before. We are using confluentinc/cp-kafka-connect:5.5.1 and the JDBC connector bundled with that. Is that correct? If not, what version should I switch to?

[C0urante]

@szicari-farmobile CP 5.5.x corresponds to Kafka 2.5.x; you can see our docs site for a full listing of the CP/Kafka versions 🙂

It looks like the fix for KAFKA-7225 is present in CP 5.5.1: https://github.com/confluentinc/kafka/blob/v5.5.1/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L308

If you're still able to reproduce, can you provide logs for your Connect worker(s) during a reproduction of the issue? It'd also be great if you could give us your worker config and connector config (feel free to censor any sensitive properties such as passwords).

[szicari-farmobile]

@C0urante Thank you - bookmarking that docs link.

I was able to reproduce the issue. Here is my Connector configuration:

{
  "name": "db_data",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "config.action.reload": "restart",
    "config.providers" : "file",
    "config.providers.file.class": "org.apache.kafka.common.config.provider.FileConfigProvider",
    "connection.url": "jdbc:postgresql://my-database/db",
    "connection.user": "my_user",
    "connection.password": "${file:/opt/connect-secrets.properties:kafka-connect-my_connector-my_user}",
    "mode": "timestamp",
    "timestamp.column.name": "last_modified",
    "query": "SELECT * FROM my_view",
    "poll.interval.ms": "300000",
    "topic.prefix": "db_data"
  }
}

I did not have time to get the logs today but will post them as soon as possible (_should_ be tomorrow).

[C0urante]

Hi @szicari-farmobile! Any chance to reproduce with logs yet?

[szicari-farmobile]

No, I apologize. I will get it done as soon as I can.

[szicari-farmobile]

@C0urante Finally got some logs together.

connector-externalized-secret-trace.log

The logs start basically at when I uploaded the configuration but continues on for 30 or more seconds after validation fails.

[szicari-farmobile]

Forgot to mention I enabled TRACE level logging on the root logger because I was having trouble targeting the right logger otherwise. I hope that ensures you're not missing anything.

[C0urante]

Thanks @szicari-farmobile. I've taken a look at your logs, and although they've unearthed a small issue with our error logging (that I've opened https://github.com/confluentinc/kafka-connect-jdbc/pull/924 to address), they don't shed a ton of light on the issue outlined here.

As next steps, could I ask you for the following?

• Provide logs again, this time from the very beginning of a worker restart all the way up to and through the point of the failed connector create request
• In addition to the connection.password property, use the config provider mechanism for the connection.url and query properties in your connector configuration
• Provide your complete worker configuration (masking any sensitive fields as necessary)
• Provide any files you are using the FileConfigProvider with (masking any sensitive fields as necessary)

I know this is a bit much; if you can only give a few pieces at a time it'd still be a great help. Thank you for helping us tackle this issue!

[szicari-farmobile]

@C0urante Just acknowledging I got your message and will get this information together as soon as I can.

[szicari-farmobile]

I will not be able to provide this information anytime soon, unfortunately. You can feel free to close the ticket, and maybe I can reopen it when I get to this request.