Confluent-kafka-dotnet: Embedded Kafka for integration tests

[tanoshii]

Description

Hi! I want to do integration tests in my project which uses Kafka Consumers and Producers. I have similar tests in Java in which I use an EmbeddedKafka provided by Spring, that allows me to test without having Kafka running previously.

I saw the integration tests in this repository and you use a real running Kafka to execute them. Do you have or have in mind to have something similar to an Embedded or In-Memory Kafka?

Thank you!

How to reproduce

Run integration tests without a running Kafka.

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version: 1.1.0
• [ ] Apache Kafka version.
• [ ] Client configuration.
• [x] Operating system: Ubuntu
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[larsbrekken]

FYI we do this with Docker and this image: https://github.com/wurstmeister/kafka-docker

[mhowlett]

We don't have specific plans, but we do plan to put quite a bit more effort into the integration tests. Something that's much more important to this client than others is we'd like a good story for testing on windows, and running Kafka on Windows is problematic (even under docker). Currently, I use the following to bring up services for testing (on all platforms): https://github.com/confluentinc/confluent-kafka-dotnet/pull/936

[larsbrekken]

Out of interest, what issues have you had running Kafka on Windows with Docker? We've had good experience running the image above plus the Zookeeper one with docker-compose, including running it over SSL. While it's not something I'd run in a production setting, it's been pretty stable for our integration tests.

[tanoshii]

We wanted to do integration tests during build time with minimum number of external dependencies, so it can be quickiest as possible. If not, Docker can be our next approach for integration tests.

Our final environment will be on linux, but I agree, not only Kafka but Docker on Windows is also problematic.