Kafka-connect-jdbc: Delete Support for sink

[Kaiserchen]

Hello,

In kafka and especially kafka streams a delete is expressed as a null value being published for the specific key that needs to be deleted. I do understand that the current source can't catch deletes as the rows are then gone and wont come up as query results. But the Sink side should probably be able to handle deletes correctly. Especially if you want to store the results of a streams application in a relational database.

Best

[shikhar]

On the sink side, I agree it would be good to support DELETEs. One quirk to handle is that null is a valid record value and only means delete in the context of a compacted topic. So it may be desirable to allow for delete only behind a config option and when pk.mode=record_key.

I haven't dug into whether we need any framework-level enhancement for null records - not sure whether they are even propagated to sink connectors currently. cc @ewencp for thoughts

[Kaiserchen]

Fully agree pk.mode = record_key is the only mode where this is relevant.

[miguno]

+1 to this feature request, from a user who contacted me directly. Here's the gist of the feedback:

We have a few people running Amazon DynamoDB instances containing data that we need in production. So we publish Dynamo changes to Kafka and use Connect to ship the data to places. Unfortunately we had to implement our own JDBC sink because https://github.com/confluentinc/kafka-connect-jdbc/issues/127 blocked us using the default one, which I would have preferred to use out of the box.

[Kaiserchen]

Thanks @miguno for pushing. Its the same person :D

[ignaciolarranaga]

+1

[vasilyulianko]

+1

[tminglei]

+1

[andybryant]

Added PR #282 for delete support.

[repos-jmp]

+1
This is a necessary evil. Any update on this ?

[vishalpaalakurthi]

What property need to be added for delete support in sink properties?

[MarcoCucchi]

Hi guys, any news about delete support? thanks

[tolgamalachi]

so there are TWO (!) PRs and they still don't get reviewed? guys this feature is very important for some people and it is requested for over two years now.....
why do i have to use forks?

@rhauch @tlberglund

[Kaiserchen]

We are quite happy with our own sink we build. The framework is very well thought through and one can pull of writing an own one in a few hours. Not ideal though but if you really need it.

[RabihHallage]

Guys any updates on this?
Such feature is really important, hope we can get an input from confluent people @maxzheng

[rhauch]

FYI, neither of the two PRs is mergeable because of more recent code changes, and I simply haven't had time to rebuild a new PR that works with the master branch. Any help would be appreciated.

[MarcoCucchi]

sorry guys, but this is unacceptable.. so long time for a feature that so many people asking for..

[tejaram15]

Please merge this feature. Its necessary for us.

[mruggio]

hello everyone, do you have any news about it?
Thanks in advance

[RabihHallage]

ping!

[sanattalwar1994]

Hey Please Can anyone do that.
It would be highly appreciable

[andybryant]

Added a new PR #641 branched off the latest code in master using the new DatabaseDialect interface.

[Gk8143197049]

Hi,
I was using jdbc sink using record value for Microsoft sql server and it fails to create 3 Pk key fields . let me know,if there is example configuaration for microsoft sql server with pk fields.

Thanks

[guyromb]

This is a great feature. Is there anything I can help to speed up the release?

[Gk8143197049]

@guyromb I am sinking to microsoft sql server using the pk.mode=record_key. it is working for insert and updates but not deletes. Do I need to enable to anything on sql server for the JDBC sink for deletes to work.

[Gk8143197049]

This is a great feature. Is there anything I can help to speed up the release?

Does it work with microsoft sql server ??

[ibrba]

Is the delete support available in the last release? It's not working for me.

[andybryant]

Thanks for merging @rhauch 👍
Would be good to update the docs too to reference the new feature, but not sure if they're in a public repo any more.
I updated the docs in the old PR when they were co-located if that's useful...
https://github.com/confluentinc/kafka-connect-jdbc/pull/282/files#diff-08673bc3b350635924c293ab1187c542R60

[sanattalwar1994]

Hey,

confluent-hub install confluentinc/kafka-connect-jdbc:5.2.2 is returning
component not found.

Can someone push to confluent-hub also.

On Wed, May 22, 2019 at 7:15 AM Andy Bryant [e-mail]
wrote:

Thanks for merging @rhauch https://github.com/rhauch 👍
Would be good to update the docs too to reference the new feature, but not
sure if they're in a public repo any more.
I updated the docs in the old PR when they were co-located if that's
useful...

https://github.com/confluentinc/kafka-connect-jdbc/pull/282/files#diff-08673bc3b350635924c293ab1187c542R60

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/kafka-connect-jdbc/issues/127?email_source=notifications&email_token=ADQQ2LDLMAWM2BUKZ7NEJC3PWSQR5A5CNFSM4CPOKNE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5U25I#issuecomment-494620021,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQQ2LDJNBWQ6IWIYGVO77TPWSQR5ANCNFSM4CPOKNEQ
.

[helpermethod]

Hi, any idea when this feature will be released? Will this allow the Sink Connector to process Debezium deletes?

[sanattalwar1994]

Hey,
I have used and checked its working just take the codebase and create jar
and use that.

On Wed, Jun 5, 2019 at 5:45 PM Oliver Weiler [e-mail]
wrote:

Hi, any idea when this feature will be released? Will this allow the Sink
Connector to process Debezium deletes?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/kafka-connect-jdbc/issues/127?email_source=notifications&email_token=ADQQ2LAP23IMW6WFBBT3ZMDPY6U6ZA5CNFSM4CPOKNE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW7QL6Y#issuecomment-499058171,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQQ2LHQMAJYANPUWA5MYBTPY6U6ZANCNFSM4CPOKNEQ
.

[limadelrey]

Hey, I have used and checked its working just take the codebase and create jar and use that.
…
On Wed, Jun 5, 2019 at 5:45 PM Oliver Weiler @.*> wrote: Hi, any idea when this feature will be released? Will this allow the Sink Connector to process Debezium deletes? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#127?email_source=notifications&email_token=ADQQ2LAP23IMW6WFBBT3ZMDPY6U6ZA5CNFSM4CPOKNE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW7QL6Y#issuecomment-499058171>, or mute the thread https://github.com/notifications/unsubscribe-auth/ADQQ2LHQMAJYANPUWA5MYBTPY6U6ZANCNFSM4CPOKNEQ .

Can you detail more @sanattalwar1994?