Node-rdkafka: Support for SSL

[jdowning]

Hello! I am trying to configure a Producer client to connect to Kafka using ssl.

var kafka = require('node-rdkafka')
var producer = new kafka.Producer({
  'metadata.broker.list': ['kafka+ssl://broker1:9096', 'kafka+ssl://broker2:9096', 'kafka+ssl://broker3:9096'],
  'security.protocol': 'ssl',
  'ssl.key.location': '/path/to/ssl.key',
  'ssl.certificate.location': '/path/to/ssl.cert',
  'ssl.ca.location': '/path/to/ssl.ca'
})

This throws errors for the security and ssl keys:

/Users/jdowning/dev/node-rdkafka-demo/node_modules/node-rdkafka/lib/client.js:54
  this._client = new SubClientType(globalConf, topicConf);
                 ^

Error: Invalid value for configuration property "security.protocol"
    at Error (native)
    at Producer.Client (/Users/jdowning/dev/node-rdkafka-demo/node_modules/node-rdkafka/lib/client.js:54:18)
    at new Producer (/Users/jdowning/dev/node-rdkafka-demo/node_modules/node-rdkafka/lib/producer.js:71:10)
    at Object.<anonymous> (/Users/jdowning/dev/node-rdkafka-demo/index.js:22:16)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

These keys are listed in librdkafka CONFIGURATION. Do they need to be explicitly supported in this library?

[edoardocomar]

Hi did your npm install build librdkafka with ssl support?

which OS are you using ?
on Mac you'll need the brew version of openssl and set include and link flags to point to it,

CPPFLAGS=-I/usr/local/opt/openssl/include
LDFLAGS=-L/usr/local/opt/openssl/lib

[webmakersteve]

Following what @edoardocomar said, if SSL support has not been baked in it will not take SSL related configuration parameters.

That being said, the error message suggests you are giving it an incorrect value for the property, which means it is indeed passing the property through to the underlying librdkafka library, but it dislikes the value you gave it. This generally happens because dependencies are missing.

Try installing openssl-dev and recompiling the entire module and see if that fixes your problem. I was able to just verify that it does indeed fix the problem.

[jdowning]

@edoardocomar Thanks for the tip! Setting those flags when building on macOS worked well.

@webmakersteve Additionally, I wanted to use this on Heroku which already has the openssl development libraries. I can confirm this library compiles there too.

[raymondsiu]

If anyone is using RHEL, you just need to sudo yum install openssl-devel and then reinstall node-rdkafka and it should automatically include the SSL features when node-gyp does the build.