Pulsar: Provide a netcore client

[fglaeser]

It will be good to have a pulsar netcore client.

Thanks.

[sijie]

Currently there are two dotnet clients:

https://github.com/fsharplang-ru/pulsar-client-dotnet
https://github.com/danske-commodities/dotpulsar

[mausch]

@sijie I guess that means Pulsar will not officially support .net ? Can you confirm?

[sijie]

@mausch Since there are already two active groups developing .net clients, we don't have a plan to develop a third one. We will work with these two groups on developing the .net clients. If any of them is willing to contribute the client back to the ASF, PMC will figure out a plan for it.

[rcollina]

Without an official, supported .NET Core client it becomes hard to suggest Pulsar versus e.g. Kafka.
Would you please reconsider your stance on this matter?

[mickdelaney]

@rcollina this might also be viable: https://github.com/apache/pulsar/issues/271

[rcollina]

@mickdelaney thanks for the link

[sijie]

@rcollina

Sure. I hear your concerns. Just to understand it more, if I understand correctly, Kafka doesn't provide an _official_ dotnet client in the apache repo. The dotnet client was maintained by confluent outside of the ASF. What is your thought about this? Is this model okay to you?

The model that Pulsar is using is very similar as Kafka. The Pulsar core repo maintains two core implementations, one is Java implementation and the other one is c++ implementation (with a c library exposed). So the community can develop other language wrappers based on the c/c++ library. In this way, Pulsar core team can focus on the core features and ensure it implemented correctly and reliably, and encourage more people in the community helping with developing different language clients (or wrappers). Although I guess people doesn't like implementing wrappers over a c/c++ library, people want native clients, which ends up with implementing the protocols in different languages (go, dotnet, rust and etc). If the requirement is to have a language-native client for every different language, it is really hard for Pulsar team to maintain and it also limits the scalability of the development. Thus we encourage developing language-native clients in the community.

There are a couple of options we can probably explore on providing _official_ clients.

1) implementing a dotnet wrapper over c/c++ library if it is a trivial enough to maintain.
2) setting up a pulsar-ecosystem repo and move some of the community implementations to be under its umbrella. similar as what https://flink.apache.org/ecosystem.html and https://flink-packages.org/

let me know what you think.

[rcollina]

@sijie thank you for your thorough reply, much appreciated.

Confluent is an official enough a name when it comes to evaluating Kafka’s pros and cons.

I can’t honestly comment on which scenario would work best, whether to follow Kafka’s model regarding project ownership.

What I know, given my current understanding of Pulsar’s featureset, it’s that it is closer to my requirements than Kafka’s. However, the project’s site does not even mention .NET Core. This unfortunately can turn potential users away.

I’m aware offering a proper client is no easy undertaking, especially considering the introduction of “memory friendly” constructs like Span in the latest .NET iterations, but it’s essential to increase adoption. My humble opinion, of course.

That being said, a gRPC endpoint + client package could be a viable solution, something to start with.

Thank you

[JarrodJ83]

@sijie I think either of those ideas would be a step in the right direction. Anything that will increase the confidence that, if you are using dotnet and want to use Pulsar, you'll have the ability to leverage the features you need and have good support from the community.

The two projects out there now seem to be going in different directions. One aiming to match the features of the java client and the other not as much. Neither have many contributors (yet). Neither of them have a solid roadmap (yet). Neither of them support all the features of the java client (yet). Specifically I see both are missing Schema which, in my mind, is a key feature. This all makes my confidence level that we will be able to leverage all of Pulsar's features in the near future using dotnet somewhat low.

I am in a similar situation as @rcollina where I think Pulsar is the better option for my company but for better or worse dotnet core is where we are headed with all future development. I worry we will not be able to use Pulsar to its full capability with the state of the two dotnet pulsar clients at present and it is a huge commitment to make without high confidence.

Hopefully this provides some insight that may help you move the dotnet support forward in some way.

[sijie]

@rcollina and @JarrodJ83 thank you for your input. I am reopening the issue for tracking this task. I will follow up with this soon.

[rcollina]

@sijie thank you

[JarrodJ83]

@sijie Building on @rcollina 's comment about Confluent, a potential third option may be to reach out to some of the companies that are building platforms around Pulsar (Streamlio?) and see if they would be interested in contributing to one of the existing repos or starting their own. This would give Pulsar an equivalent to Confluent which would greatly increase the confidence that dotnet and Pulsar will be a good pair moving forward, broadening the adoption of Pulsar.

[sijie]

@JarrodJ83 thank you for your input. I am following up with the owners of these projects since they are already working on these projects. it would be great to make sure we don't create another project to spread out the community efforts. regarding the comment around Confluent, we (StreamNative) are also happy to provide the support for dotnet client as well. but I really want the community can work together on building pulsar ecosystem before splitting the efforts.

[sijie]

For anyone who is interested in this topic, here is the summary of it.

I have reached out to both the owners of pulsar-client-dotnet and dotpulsar about contributing their pulsar client library back to the community. The maintainer of dotpulsar is interested in moving the project to the ASF. They have started the email thread for discussing the contribution.

https://lists.apache.org/thread.html/8ebe35027d818e595eec322da26a3f392324ec3d86c4c1b12b1ff498%40%3Cdev.pulsar.apache.org%3E

Everyone is welcome to chime in the topic and give the feedback and thoughts in that email thread.

Happy new year!

[JarrodJ83]

This is great news, @sijie! I've been following the email discussion but I don't know much about how a project gets brought into the Pulsar community. Maybe you could provide a high level update of what is happening and what this means moving forward for DotPulsar? Will there be a roadmap of some sort for it? When would we expect to start seeing new updates? etc.

As stated before I am still evaluating if we would be able to use Pulsar at my current company and any information about what the development story on dotnet will be moving forward will help inform that decision. Thank you!

[sijie]

@JarrodJ83 sorry for the late response here. have been pretty busy in the past. I am helping DotPulsar doing the IP Clearance in the ASF. Once it passes, we will transfer the repo to the ASF. Most of the updates are blocked at my side. I will try to push this forward as much as I can.

[sijie]

DotPulsar now is under pulsar project now. The PMC will work on maintaining dotpulsar.

https://github.com/apache/pulsar-dotpulsar

[rcollina]

Great news, thank you @sijie!