node-rdkafka does not compile on Windows

[webmakersteve]

Installing on windows does not work, and there is no way to make it work currently.

This is a low priority feature, but keeping it here so it can be tracked properly.

[serialLinker]

Hello,

As you can see in this thread : https://github.com/edenhill/librdkafka/issues/982, librdkafka can now run on Windows platform (master at this time). Can you tell me when you think you can add this feature in node-rdkafka ?

I'm working for a major insurance company and node-rdkafka had been chosen for our node applications, and we prefer to use the same library on Windows and Linux.

Thx

[mattness]

Windows support is not currently a priority for us. That said, we'd be happy to accept a pull request for it once a release of librdkafka has been made that includes edenhill/librdkafka#982.

[vipreshjha]

@mattness now that RC3 of librdkafka is released is there any possibility of providing a install for windows?

[webmakersteve]

The reason it doesn't work on Windows is not because the compile of librdkafka didn't work on Windows: we just aren't able to prioritize its implementation.

As @mattness said we are happy to accept a PR to implement the changes, but it isn't the librdkafka source that will be the work - it is adapting the gypfile to accommodate for all windows platform specific things, it is generating the header files required for the application to work (config.h) without assuming the presence of (ba)sh. It is testing the software on windows servers, and potentially integrating its testing into the CI pipeline.

I'd be happy to help anyone willing to take a crack at these problems so we could have windows support.

[daves2552]

@webmakersteve Can you provide a very rough estimate of how many developer weeks you'd expect that effort to take?

[adalinesimonian]

I'm currently working on Windows support for node-rdkafka. Right now it's building, but there are some linking issues to be resolved, and I don't have WITH_SASL working just yet.

However, it's only been a day so far that I've been working on it, so I expect to be making headway with some time.

[RyanMelenaNoesis]

Just out of curiosity, is it even necessary to build librdkafka for Windows or would it suffice to simply use the pre-built redistributable?

[webmakersteve]

There is potential for that pre-built binary to work on Windows, but you would need to configure node-gyp to look for the library dependency there. It, too, would probably require changes to the code base to work, but may satisfy some of the complexity.

However, I do not think librdkafka's compilation step is the problem stopping windows support from advancing. The vast majority of the problems relate to the gypfile and how it is generating the Visual Studio solution file given the current config. I imagine many windows specific pieces need to be added to the gypfile regarding that, especially regarding linking, to make this work.

[adalinesimonian]

Just a quick addition to what @webmakersteve said: In addition, node-rdkafka depends on unistd.h, which isn't a part of Visual C/C++, so a shim is needed (in my copy I am using an shim by Microsoft under the Apache licence). There are a few more details from what I remember, but I got sidelined with an important project that I need to take care of first before I can return to node-rdkafka.

[RyanMelenaNoesis]

@webmakersteve Thanks for the quick response!

@vsimonian Any idea how long you'll be sidelined? If you expect it will be a while, would you be able to check in what you've got so far to your branch? I'd be happy to take a look at what you have and see if I'm able to move things a bit further down the field.

[webmakersteve]

@vsimonian The only thing I'm using from unistd is usleep. You can just check what platform the code is running on and use the Windows equivalenet for that particular function instead of importing an entire polyfill.

[adalinesimonian]

@webmakersteve Only thing from unistd.h being used is usleep, yes, so I trimmed it down. However, Microsoft's stdlib.h also does not have rand_r, because their implementation of rand is thread-safe unlike GNU rand. I macro'd it as a replacement in the header file. And it looks like the polyfill I was talking about was actually for getopt, but there's no use of getopt in node-rdkafka, so I removed it.

@RyanMelenaNoesis It might be a while, so I totally don't mind posting up my work so far if you'd like to take a crack at it and get it up and running! Keep in mind I haven't had an opportunity to touch the code too much so you might find some questionable parts. But still I think it's a step forward, just needs to be brought to completion and then cleaned up.

[adalinesimonian]

@RyanMelenaNoesis I have the code up at https://github.com/vsimonian/node-rdkafka/tree/vsimonian-windows

I've sent you an invite for write access.

[RyanMelenaNoesis]

@vsimonian Thanks for setting up access. I ended up taking some of what you did but skipping the librdkafka compilation (for the time being) and instead utilizing the pre-built librdkafka redistributable binary. I branched the the Blizzard repo and created a pull request from there. Please let me know if you'd like me to integrate any of the changes back to your fork.

[adalinesimonian]

@RyanMelenaNoesis Glad it was of use! Your PR looks great! Sorry I don't have time to do a more thorough look through it, I only had time to give it a very quick glance.

[RyanMelenaNoesis]

@vsimonian Unfortunately, my PR was denied because it does not build librdkafka from source. A decision I find bewildering but that is how it goes.

For anyone interested in using node-rdkafka on Windows immediately, you can point your package.json dependency to our fork "node-rdkafka": "NoesisLabs/node-rdkafka" until such a time that someone chooses to implement a build-librdkafka-from-source-on-windows implementation.

[webmakersteve]

248 - Prelim pass at windows support that compiles librdkafka from source