Vector: Kafka sink panics starting with 03/30 nightly release: "can call blocking only when running in a spawned task"

Created on 10 Apr 2020 · 4Comments · Source: timberio/vector

The following config causes a fatal panic in Vector on current releases, starting at the nightly release 2020-03-30. The config fails on the most current nightly release of 2020-04-08.

[sources.citus_worker_journald]
  type = "journald"
  current_boot_only = true
  units = [] # Empty array means all SystemD units

[sinks.log_kafka]
  type = "kafka"
  inputs = ["citus_worker_journald"]
  bootstrap_servers = "my_kafka_server"
  topic = "prod_logs_default"
  encoding = "json"

  [sinks.log_kafka.librdkafka_options]
    "security.protocol" = "SSL"

The error (with unhelpful backtrace) is:

Started Vector.
 Apr 09 22:07:44.048  INFO vector: Log level "info" is enabled.
 Apr 09 22:07:44.048  INFO vector: Loading configs. path=["/etc/vector/citus-worker.toml"]
 Apr 09 22:07:44.051  INFO vector: Vector is starting. version="0.9.0" git_version="v0.8.0-152-g686919a" released="Mon, 30 Mar 2020 14:20:28 +0000" arch="x86_64"
 thread 'main' panicked at 
'can call blocking only when running in a spawned task',
 /opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/thread_pool/worker.rs:27:34
 stack backtrace:
    0:           0xda153a - <unknown>
    1:           0x7645bc - <unknown>
    2:           0xda0ce6 - <unknown>
    3:           0xda0936 - <unknown>
    4:           0xd9ffca - <unknown>
    5:           0xd9fb58 - <unknown>
    6:           0x762290 - <unknown>
    7:           0x768842 - <unknown>
    8:          0x1398dab - <unknown>
    9:          0x19b770a - <unknown>
   10:           0xfe8d4f - <unknown>
   11:          0x1c1408e - <unknown>
   12:          0x1bba3c3 - <unknown>
   13:          0x1c01543 - <unknown>
vector-citus-worker.service: Main process exited, code=exited, status=101/n/a
vector-citus-worker.service: Failed with result 'exit-code'.

I bisected the nightly releases. The config works on the 03/18 release. There's no other nightly releases until 03/30, at which point it fails.

I tried to reproduce the error with different sinks. I was unable to reproduce the error with: