Kafka-connect-jdbc: JDBC Sink Connector fails -upserting into multiple tables from multiples topics using kafka-connect

Created on 12 Sep 2017  路  2Comments  路  Source: confluentinc/kafka-connect-jdbc

I am trying to read 2 kafka topics using JDBC sink connector and upsert into 2 Oracle tables which I manually created it. Each table has 1 primary key I want to use it in upsert mode . connector works fine if I use only for 1 topic and only 1 field in pk.fields but if I enter multiple columns in pk.fields one from each table it fails to recognize the schema. Am I missing any thing please suggest.

name=oracle_sink_prod
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=KAFKA1011,JAFKA1011
connection.url=URL
connection.user=UID
connection.password=PASSWD
auto.create=false
table.name.format=KAFKA1011,JAFKA1011
pk.mode=record_value
pk.fields= ID,COMPANY // ID is pk of kafka1011 table and COMPANY is of other
auto.evolve=true
insert.mode=upsert

All 2 comments

I am trying to read 2 kafka topics using JDBC sink connector and upsert into 2 Oracle tables which I manually created it. Each table has 1 primary key I want to use it in upsert mode . connector works fine if I use only for 1 topic and only 1 field in pk.fields but if I enter multiple columns in pk.fields one from each table it fails to recognize the schema. Am I missing any thing please suggest.

name=oracle_sink_prod
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=KAFKA1011,JAFKA1011
connection.url=URL
connection.user=UID
connection.password=PASSWD
auto.create=false
table.name.format=KAFKA1011,JAFKA1011
pk.mode=record_value
pk.fields= ID,COMPANY // ID is pk of kafka1011 table and COMPANY is of other
auto.evolve=true
insert.mode=upsert

removing "pk.fields= ID,COMPANY". the connect will detect the pk field if you use schema registry

Thanks @dreamcatchernick

Was this page helpful?
0 / 5 - 0 ratings