Confluent-kafka-go: librdkafka cannot be installed on Debian Stretch

[lizthegrey]

Description

libssl conflict results in inability to install librdkafka-dev=1.1.0~1confluent5.3.0-1 on Debian

How to reproduce

# add-apt-repository [...] && apt update && apt install librdkafka-dev

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 librdkafka-dev : Depends: librdkafka1 (= 1.1.0~1confluent5.3.0-1) but it is not going to be installed
                  Depends: librdkafka++1 (= 1.1.0~1confluent5.3.0-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Exited with code 100

# apt-get install librdkafka-dev=1.1.0~1confluent5.3.0-1 librdkafka1=1.1.0~1confluent5.3.0-1 librdkafka++1=1.1.0~1confluent5.3.0-1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 librdkafka1 : Depends: libssl1.0.0 (>= 1.0.0) but it is not installable

Checklist

Please provide the following information:

• [x] confluent-kafka-go and librdkafka version (LibraryVersion()): attempting to install 1.1.0~1confluent5.3.0-1 of librdkafka
• [x] Apache Kafka broker version: n/a
• [x] Client configuration: ConfigMap{...}: n/a
• [x] Operating system: Debian Stretch
• [x] Provide client logs (with "debug": ".." as necessary): n/a
• [x] Provide broker log excerpts: n/a
• [ ] Critical issue

[edenhill]

This is a known issue, the Confluent librdkafka debian packages are not compatible with Ubuntu >= 18., et.al, due to libssl1.0.0 not being available anymore.

There is no immediate fix in the pipeline for the debian package, but we're looking at bundling librdkafka with the Go client to void the need for librdkafka installed separately.
You can try it out by changing your confluent-kafka-go import to confluent-kafka-go-dev, see this example:
https://github.com/confluentinc/confluent-kafka-go-example/blob/master/main.go#L23

That is all experimental at this point, but the more feedback we get the sooner it will land in the proper confluent-kafka-go repo.

[lizthegrey]

Oh, yessss please omg <3 @edenhill I was literally just ranting that I didn't want to make my team separately install librdkafka. Twitter rant: https://twitter.com/lizthegrey/status/1157327638651969537

I'll use confluent-kafka-go-dev instead, which was literally the work I was about to duplicate myself.

[lizthegrey]

https://github.com/confluentinc/confluent-kafka-go-dev/issues/1 there you go, some feedback :) -- it works gloriously locally though, we just vendor everything for stability's sake.

[MichaelSaah]

is this libssl conflict also the reason why 1.1.0 isn't available for buster, bullseye, or sid?

https://packages.debian.org/sid/librdkafka-dev

[lizthegrey]

is this libssl conflict also the reason why 1.1.0 isn't available for buster, bullseye, or sid?

https://packages.debian.org/sid/librdkafka-dev

Yes.

[mxk1235]

@edenhill will the confluentinc/confluent-kafka-go-dev prebuilt librdkafka include libssl, libsasl, etc.?

[edenhill]

It will include libssl, but not libsasl2 (Kerberos support) due to the complexity of making its plugin-based mechanisms work with static linking. Kerberos users will still need to install librdkafka manually and build their go app with the -tags dynamic flag.

[theUm]

I'm not a clang expert but when I'm compiling librdkafka from sources there is no libssl errors and everything seems to build fine. Do you have any ideas why? Can I rely on this behavior?

I tested it with debian-buster with latest build-essential libssl-dev libsasl2-dev libzstd-dev
Tried to build from master and from v1.1.0 - both works

$ openssl version -a
OpenSSL 1.1.1c  28 May 2019

Configuration summary:
  prefix                   /usr/local
  MKL_DISTRO               debian
  SOLIB_EXT                .so
  ARCH                     x86_64
  CPU                      generic
  GEN_PKG_CONFIG           y
  ENABLE_SSL               y
  ENABLE_GSSAPI            y
  ENABLE_DEVEL             n
  ENABLE_VALGRIND          n
  ENABLE_REFCNT_DEBUG      n
  ENABLE_SHAREDPTR_DEBUG   n
  ENABLE_LZ4_EXT           y
  MKL_APP_NAME             librdkafka
  MKL_APP_DESC_ONELINE     The Apache Kafka C/C++ library
  CC                       gcc
  CXX                      g++
  LD                       ld
  NM                       nm
  OBJDUMP                  objdump
  STRIP                    strip
  CPPFLAGS                 -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align
  PKG_CONFIG               pkg-config
  INSTALL                  install
  LIB_LDFLAGS              -shared -Wl,-soname,$(LIBFILENAME)
  LDFLAG_LINKERSCRIPT      -Wl,--version-script=
  RDKAFKA_VERSION_STR      1.1.0
  MKL_APP_VERSION          1.1.0
  LIBS                     -lm -lzstd -lsasl2 -lssl -lcrypto -lz -ldl -lpthread -lrt -lpthread -lrt
  CFLAGS                   
  CXXFLAGS                 -Wno-non-virtual-dtor
  SYMDUMPER                $(NM) -D
  exec_prefix              /usr/local
  bindir                   /usr/local/bin
  sbindir                  /usr/local/sbin
  libexecdir               /usr/local/libexec
  datadir                  /usr/local/share
  sysconfdir               /usr/local/etc
  sharedstatedir           /usr/local/com
  localstatedir            /usr/local/var
  libdir                   /usr/local/lib
  includedir               /usr/local/include
  infodir                  /usr/local/info
  mandir                   /usr/local/man
  BUILT_WITH               GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW
Generated config.cache

Now type 'make' to build

[edenhill]

Source builds are fine, it will use the OpenSSL version installed on the system.
The problem is with the Debian packages that are built for an older Debian distro which has an openssl version which is not available on newer Debian distros, thus making the librdkafka packages uninstallable due to missing dependency.

[theUm]

thanks, that's clarifies everything

[natemurthy]

@edenhill I've also begun using the confluent-kafka-go-dev repo for building all my go projects that need the 1.1.0 version of this client. I'm happy to report (as others have above) that this pre-bundling approach has worked successfully for our builds as well (my team is using a base golang build image derived from debian stretch-slim).

We're using dep for vendor management like so:

[[constraint]]
  name = "github.com/confluentinc/confluent-kafka-go-dev"
  branch = "master"

And so my follow-up questions are:

• What guarantees do we get by building off the master branch of this experimental repo?
• When can we anticipate the self-static librdkafka bundle to be available in the canonical confluent-kafka-go repo? (I would expect this issue to be closed then).

[edenhill]

@natemurthy Happy to hear that it is working for you!

Answers:

• confluent-kafka-go-dev is a proof of concept, it comes with no guarantees and should not be used in production.
• we're looking to merge the static builds into the official confluent-kafka-go repo during Q4. This makes the static builds a supported feature.

[natemurthy]

Hi @edenhill wanted to follow up on this. It doesn't appear that static builds of the librdkafka library are supported yet in this official repo. Do you have a revised timeline?

I'm still experiencing confusion among teams I work with using this library for the first time expecting to have the same version of librdkafka as the version of confluent-kafka-go that they install for building their clients.

Many developers experience this in their builds, both in Debian and Alpine distros:

# gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
/go/src/gopkg.in/confluentinc/confluent-kafka-go.v1/kafka/00version.go:44:2: error: #error "confluent-kafka-go requires librdkafka v1.3.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"
 #error "confluent-kafka-go requires librdkafka v1.3.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"
  ^~~~~

And then the documentation in http://docs.confluent.io/current/installation.html isn't clear about how to resolve this dependency.

[edenhill]

We're looking to release confluent-kafka-go with embedded librdkafka during Q1.

For now you should install librdkafka through one of the confluent apt/rpm repositories, homebrew, or a source install from github.

[mcarbonneaux]

when you rebuild librdkafka for recent debian/ubuntu (on withch the source build work fine with the openssl installed version that are more recent than 1.0...) ?
because when we need c lib for kafka, we need c lib (not go lib)... if we need go lib we use go lib...