Nservicebus: Kafka transport

[pablocastilla]

Hi!

How about a Kafka transport? I see it has got a lot of momentum in the big data world and if you use it in the project it could be interesting to use it as a transport and not having several transport technologies.

[danielmarbach]

@pablocastilla thanks for reaching out. We had a conversation running internally started by @synhershko. We decided to not do it for v6 because there is simply not enough demand for such a transport and we wanted to focus on the async changes first. Here are a few snippets:

@synhershko commented on Sep 8, 2014

This is just a suggestion / point for discussion.
Kafka - a persistent messaging system built by LinkedIn as a commit log - is a gaining a lot of popularity lately. It's mostly being used as a relaxed publish / subscribe mechanism, and it features high throughput and very good durability. It also let's you replay events / messages as messages are not "delivered" per se but read from the log using an offset.

While its a bit of a pain to set up, definitely on Windows, its a very good solution for high messages throughput and IMO should definitely be at least an option for an NSB transport.

More about the story and ideas behind Kafka: http://sites.computer.org/debull/A12june/pipeline.pdf

@dannycohen commented on Sep 9, 2014

Questions:
1. Do you see Kafka having any distinct advantage ("killer feature") over existing transports ?
- AFAIK, Kafka is overwhelmingly a non-Windows community darling. This also aligns with your estimation on lack of ease-of-installation on Windows. How do you see this work out ?
- (we are aiming, practically invariably, at Windows community, and a primary goal is to provide an easy installation / maintenance)

@synhershko commented on Sep 9, 2014

1. Yes, mainly throughput, and also the "scrolling" feature where messages are not delivered but read by clients, allowing you to replay them.
2. So is RabbitMQ, yet it has matured and people started using it too. Kafka requires a ZooKeeper instance (or several), this is mainly why I was saying its hard to use. This requirement may go away, but honestly I can definitely see people choosing Kafka for its features and running it on Linux boxes in their clusters. This is because you will not install Kafka side-by-side on the server with other tech, like you shouldn't be doing with RMQ (which is highly memory volatile).
   That being said, I do see Kafka being used more and more by Windows people within a year or so.

@ SzymonPobiega commented on Sep 9, 2014

Regarding no 1, how Kafka relates to GetEventStore? GES seems to have
similar advantages (scrolling and throughput) without being hard to set up on Windows

[pablocastilla]

Hi!

Thanks for sharing that info.

I understand you must be very very busy with the v6. But if you realize the arguments has gained more weight in these two years. I definitely see that transport in the roadmap.

[padenj]

I just wanted to cast a vote for a Kafka transport at some point. My company is a mix of Java and .Net applications (heavier on the Java stack). We have previously been using RabbitMQ throughout the enterprise, but Kafka is quickly becoming the standard that all projects are encourage to utilize. We are interested in using NServiceBus for an upcoming .Net project and using Kafka as the transport would certainly help sell it to the project stakeholders!

[pablocastilla]

How about a kafka persistence? I see kafka as a transport and as a log. Maybe it would be useful, for instance we could store sagadata as events, but I am not sure about it.

[asednev]

Another vote for Kafka transport. Although not very Windows friendly, Kafka is very popular in Java community and will become more relevant with .NET Core adoption.

[JoeShook]

Growing interest in Kafka. +1

[SimonCropp]

@pablocastilla since you have started a Kafka transport, should we close this ?

https://docs.particular.net/nservicebus/kafka/ https://github.com/pablocastilla/NServiceBus.Kafka

[pablocastilla]

Yes! :)

It is almost finished. I would say it is in beta until we battle test it

[SimonCropp]

Any chance of you submitting a runnable sample as a PR to our docs site ?

[pablocastilla]

Sure

[pablocastilla]

Simon, one question.

Do you want it here?
https://github.com/Particular/docs.particular.net/tree/master/samples/kafka/Kafka_1

I see that's somehow a runnable example, what do you want me to change?

Thanks

[SimonCropp]

@pablocastilla so sorry. i forgot about that one. is there anything u want added to the sample md? https://github.com/Particular/docs.particular.net/blob/master/samples/kafka/sample.md

[pablocastilla]

@SimonCropp No problem!

I don't want to add anything at this moment. I will consider it finished and start with the apache phoenix persistence.

I suppose bugs and optimizations will have to be done when we start to use it.

[SimonCropp]

@pablocastilla awesome. and thanks for the contribution

[madhavabhyankar]

@SimonCropp , @pablocastilla:
I suppose the documentation for Kafka transport has moved. Can you please point me to the new location?

[SimonCropp]

@madhavabhyankar the kafka transport never made it to a 1.0 release and hasnt been updated for 2 years. i suspect it was removed from doco since it is no longer being maintained. you could ask @pablocastilla to transfer ownership of it to you, if you want to start maintaining it.

[pablocastilla]

Hi! I still think a Kafka transport is a good idea but it didn't get any momentum, if you want to get the ownership I can give it to you without problem.