Vector: [Kafka][Source] topology error when using certs

[Diliz]

Hello there!

First issue here, I'm really new to vector, so maybe it's only a problem between my screen and my chair. I can change the issue format if needed as well.

I actually got an issue with the kafka source.

vector --version
vector 0.9.1 (v0.9.1 x86_64-unknown-linux-musl 2020-04-30)

Here's my vector.conf:

[sources.simple]
  type = "kafka" # required
  bootstrap_servers = "kafka.servers.ips"
  group_id = "group" # required
  topics = ["my_topic]
  tls.ca_path = "/path/to/ca.pem"
  tls.crt_path = "/path/to/crt.pem"
  tls.key_path = "/path/to/crt.key"
  tls.enabled = true

[sinks.stdin]
  inputs = ["simple"]
  type = "console"
  encoding.codec = "text"

Here's what I got when trying to use it:

vector --config vector.yml --dry-run -v
May 19 14:22:38.365  INFO vector: Log level "debug" is enabled.
May 19 14:22:38.365  INFO vector: Loading configs. path=["vector.yml"]
May 19 14:22:38.371  INFO vector: Vector is starting. version="0.9.1" git_version="v0.9.1" released="Thu, 30 Apr 2020 15:51:58 +0000" arch="x86_64"
May 19 14:22:38.371  INFO vector: Dry run enabled, exiting after config validation.
May 19 14:22:38.372 ERROR vector::topology: Configuration error: Source "pure_backend": Could not create Kafka consumer: Client creation error: `ssl.keystore.password` is mandatory when `ssl.keystore.location` is set

I tried to set an empty key_pass like this: tls.key_pass= "" but it's giving me the same error. Any idea on how to fix this?

Thanks for your work on vector, it's a cool tool!

[binarylogic]

Hi @Diliz, thanks for reporting. We'll have @bruceg take a look. It looks like a simple error.

[bruceg]

For reference, the Kafka source and sink pass the options on to the standard Kafka library, which uses its own SSL/TLS setup. In particular, tls.key_path translates to the option ssl.keystore.location, and tls.key_pass translates to ssl.keystore.password. Kafka will have its own rules on how those work.

However, are you trying to authenticate to the Kafka server using a client certificate? If not, the tls.crt_path and tls.key_path options are not necessary to enable TLS, as Vector acts as a client.

[Diliz]

Hello!

First, thanks for the quick reaction and answers on this issue :)

And yes, I need to consume Kafka with a certificate, I was already doing it with a fluentd agent + a python script, it's working well actually, but if I can use only one simple vector config file to replace both of my scripts it will be better actually, and I can use it without a key pass today with this solution :/

Regards

[bruceg]

I'm not sure how to resolve this. The error is coming from the Kafka library itself, something we don't have direct control over. What does your fluentd config contain for these options? I wonder if trying with a dummy (not empty) password would work. I will try a few options to see if I can figure out what is needed here.

[bruceg]

It turns out this was a bug in the setting name we passed to the Kafka library. I just submitted a pull request here that resolves the problem.

[Diliz]

Nice! Thanks for the fast fixing! Sorry for the delay, did not saw your questions about the fluentd config, but we don't put a default empty password in fluentd conf, maybe fluentd does, dunno :/

Anyway, thanks for your work, really appreciate :)