Masstransit: Kafka Transport

[NooNameR]

Kafka is really good framework for Event Streaming, it seems like the API is common with MT, and would be nice to adapt new Confluent Kafka client to MT transporter.

• [x] Adapt Confluent Client to MT
• [x] Cover transport with tests
• [x] Open a pull request

[pvanroos]

@NooNameR Have you had any experience with Azure Event Hubs?

[NooNameR]

yes, I used to use it before, but now Microsoft provides compatibility with Kafka for EventHub

[pvanroos]

I need to take a look at both feature sets again. I've been looking for a good event streaming solution. I was hoping to use only Az Event Hubs but I heard great things about Kafka.

[NooNameR]

https://azure.microsoft.com/en-us/blog/announcing-the-general-availability-of-azure-event-hubs-for-apache-kafka/

[aallfredo]

Just adding a link to the google group chat on this topic https://groups.google.com/forum/#!topic/masstransit-discuss/SGShxmeEHoU

[phatboyg]

Spent a few hours trying to get the Confluent Kafka client working on OS X via Rider, and no luck thus far. Using the latest prerelease version, and still having issues getting it to load librdkafka...

[NooNameR]

Hey Chris, it works good with Docker :) but there is one problem with Memory exception sometimes...

[phatboyg]

I'm going to close this, as it isn't a priority atm.

[jstafford5380]

Any chance this is ready to be re-opened? We're about to take on Kafka, and it would be great to have a management layer for it with out me having to develop it myself :/

[phatboyg]

This is the approach I have taken at this point, the semantics are too different for a true transport. Experience with using the mediator with Kafka might produce some helpful additions at some point, though. If you give it a shot, let me know what's missing:

https://masstransit-project.com/2020/02/23/mediator/#what-about-kafka

[jstafford5380]

This is the approach I have taken at this point, the semantics are too different for a true transport. Experience with using the mediator with Kafka might produce some helpful additions at some point, though. If you give it a shot, let me know what's missing:

https://masstransit-project.com/2020/02/23/mediator/#what-about-kafka

I'm actually not familiar enough with Mass Transit at the moment to jump right in with implementing a transport. Most of my experience is in NServiceBus, and I've implemented my own watered down version of it, and I don't have much experience with Kafka (I got it running and go some messages in and out), but I have experience with EventStore -- so at least I'm not coming from nothing. Is there any documentation that could assist with implementing a transport? It could be helpful to have a thread to help collab with it, track the challenges etc. I'd participate in something like that.

[phatboyg]

The help would be appreciated. There aren't really any _guides_ on creating transports. The only examples are existing transports, with the RabbitMQ transport probably being the most extensively groomed in terms of structure and context. It's also the oldest and most widely used, which is why it gets so much attention. I can open this issue, to have a place to track it.

[NooNameR]

@phatboyg could you create PR for sharing codebase?

[phatboyg]

I could create a branch, and developers could submit their PR's to that branch?

[jstafford5380]

Copy that. I'm going to finish watching your series that you recently recorded to get oriented then I'll start poking

[NooNameR]

@phatboyg yeah, it should work, thx :)

[NooNameR]

@phatboyg do you have any thoughts/vision on this?
I'm planning to start branch and do some work

[phatboyg]

I don't think it's the same experience as MassTransit, it's a different thing. Like, I would not want anyone doing request/response via Kafka. So it feels like something different, but I don't know the exact way to approach it yet.

I don't think it's a transport, at least not a full transport.

[NooNameR]

I have the same feeling, it definitely seem like not a IBus

[phatboyg]

This is where I left it: https://masstransit-project.com/2020/02/23/mediator/#what-about-kafka

[jstafford5380]

Maybe if you just call it what it is, a "stream" (e.g. IEventStream). From there you could
potentially build it along side similar concepts. For example, Event Store,
Event Hubs, Kafka, etc. That said, I'm constantly reading that people are
using Kafka in the exact same way we would use traditional brokers. It
always felt kind of weird to me, but if that's how people are using it then
that's how people are using it. Would it help to call/list out the semantic
differences in context of MT? Like, what feels like it's cutting against
the grain?

On Sat, Apr 18, 2020 at 8:19 AM Chris Patterson [e-mail]
wrote:

This is where I left it:
https://masstransit-project.com/2020/02/23/mediator/#what-about-kafka

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MassTransit/MassTransit/issues/1309#issuecomment-615888466,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALI4REDPAPNASESDNEOPHVDRNHAI7ANCNFSM4GSMHUVQ
.

[phatboyg]

My notes on adding this type of support:

Support for Kafka and Event Hub may introduce new terminology to MassTransit. It isn’t about publish or subscribe, send or receive. It’s about produce and consume, write and read. There are producers and consumers, writers and readers. What makes MassTransit great must remain a first class citizen, while supporting the concept of connecting receivers to the bus.

[jstafford5380]

Just thinking out loud:

Topology-wise I think you would just shift implementations from queue/topic to solely topic but you could still keep the send, publish, and receive semantics the same. Using that approach, you would just create dedicated receive topics as the 'endpoint' through which the service would receive messages. The risk there is that since you can't "consume" the message, you have to ensure messages with a command semantic aren't replayed.

I could see request-response being tricky, too. Specifically, Kafka has no concept of competing consumers. Instead, they have partitions where each consumer is dedicated to processing a specific partition. At first glance this seems like it could be exploited to determine exactly what a reply-to address would be (i.e. a specific partition of the service's receive topic), but when scaling out, I think it creates additional workers for each partition by default rather than adding new partitions and redistributing the messages. So that wouldn't necessarily work. Likewise, you can't just create an ephemeral topic to reply to simulate the auto-deleting queues. Or could you...?

[phatboyg]

See, that's the point. The semantics don't align, and I definitely don't want to encourage the use of Kafka as a transport. Or Event Hub. They're designed for different things. Brokers are about dispatch, log streams are about logs.

So I think it is more about streaming messages from Kafka into a bus, that is using another transport (even if it is in-memory, or mediator). And providing the ability to produce messages that are written to Kafka.

It's also why I don't want to just try to map Kafka as a transport, it's too different. And expectations aren't the same.

[jstafford5380]

Yeah I'm with you.

So I think it is more about streaming messages from Kafka into a bus, that is using another transport (even if it is in-memory, or mediator). And providing the ability to produce messages that are written to Kafka.

Interesting! I like that.

[jstafford5380]

So maybe something along these lines

[phatboyg]

I don't think there is a need to go from Kafka -> Queue, it would make more sense to go directly to an 'endpoint' on the bus that is without a transport (the dispatch is the consumer group/consumer agent from the Kafka client). I think the Kafka dispatch/threading could be handled by the consumer group code, but I lack a depth of understanding of how it's done under the hood for the client SDKs.

[phatboyg]

So this is well along, as an attachment.