Confluent-kafka-go: librdkafka v0.11.4 does not exist

[livnoorbrar]

Description

The go get -u github.com/confluentinc/confluent-kafka-go/kafka is unable to install the client and produces the error
``# github.com/confluentinc/confluent-kafka-go/kafka ../github.com/confluentinc/confluent-kafka-go/kafka/00version.go:43:2: error: "confluent-kafka-go requires librdkafka v0.11.4 or later. Install the latest version of librdkafka from Homebrew by runningbrew install librdkafkaorbrew upgrade librdkafka`"

error "confluent-kafka-go requires librdkafka v0.11.4 or later. Install the latest version of librdkafka from Homebrew running brew install librdkafka or brew upgrade librdkafka"

^
1 error generated.

When `brew install librdkafa` is run it fetches the v0.11.3 which is the latest stable version released *https://github.com/edenhill/librdkafka/releases*

» brew info librdkafka
librdkafka: stable 0.11.3 (bottled), HEAD
The Apache Kafka C/C++ library
https://github.com/edenhill/librdkafka
/usr/local/Cellar/librdkafka/0.11.3 (14 files, 1.9MB) *
Poured from bottle on 2018-03-16 at 13:33:24
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/librdkafka.rb
==> Dependencies
Build: pkg-config ✔
Required: lzlib ✔, openssl ✔
Recommended: lz4 ✔
==> Options
--without-lz4
Build without lz4 support
--HEAD
Install HEAD version
```

• [ ] confluent-kafka-go version (0.11.0)
• [ ] Operating system: Mac OSX

[edenhill]

This is a bit of a synchronization issue, librdkafka v0.11.4 will be released within a couple of days, in the meantime build librdkafka from source (master branch or v0.11.4-RC1B tag)

[chaevnicher]

up!

[stim371]

Thanks for all your work, Magnus.

This tripped me up as well and caused some confusion when trying to use the projects for the first time.

If librdkafka and this lib use semver, then such tight coupling could be relaxed. And since homebrew doesn't pick up RCs, it means new devs will sometimes get everything automagically and other times need to hunt this down and then install from source outside their normal flow.

Just my 2 cents, and thanks again for the helpful tooling.

[stim371]

After a little research (as someone with rudimentary go knowledge), it looks like a side effect of go get pulling directly from master. Frustrating, but I get the sync issue now. Thanks again.

[pgt502]

Hello,

any idea of when this might be resolved?

Thanks

[edenhill]

Sorry for the mess, this is a sad side effect of the delay of librdkafka v0.11.4.
With a bit of luck we'll be able to release librdkafka tomorrow 🤞 and the homebrew packages should follow shortly after.

[StevenACoffman]

In the meantime, you can build from source, like in this Dockerfile example (Note: change the silly MYSPECIALPROJECT):

FROM golang:1.10.0-alpine3.7

RUN apk add --update --no-cache alpine-sdk bash ca-certificates \
      libressl \
      tar \
      git openssh openssl yajl-dev zlib-dev cyrus-sasl-dev openssl-dev build-base coreutils
WORKDIR /root
RUN git clone https://github.com/edenhill/librdkafka.git
WORKDIR /root/librdkafka
RUN /root/librdkafka/configure
RUN make
RUN make install
#For golang applications
RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

ENV MYSPECIALPROJECT "organization/repository"
WORKDIR /go/src/github.com/$MYSPECIALPROJECT

RUN go get -d -v github.com/confluentinc/confluent-kafka-go/kafka

COPY . .
RUN GOOS=linux go build -a -installsuffix cgo -o app .

ENV PORT 4000
EXPOSE ${PORT}
CMD ["./app"]

[martyphee]

Does v0.11.4 exist yet in the repos? I was trying to install it in circle ci and getting an older version.

#!/bin/bash -eo pipefail
sudo apt-get install -y -V librdkafka-dev
make ci_install
Reading package lists... Done


Building dependency tree       


Reading state information... Done

The following additional packages will be installed:
   librdkafka++1 (0.9.3-1)
   librdkafka1 (0.9.3-1)
The following NEW packages will be installed:
   librdkafka++1 (0.9.3-1)
   librdkafka-dev (0.9.3-1)
   librdkafka1 (0.9.3-1)

[edenhill]

@martyphee Use the Confluent APT repository for up-to-date packages:
https://docs.confluent.io/current/installation/installing_cp.html#debian-and-ubuntu

[martyphee]

I've been trying to get that to work without luck.

sudo apt-get update  
sudo apt-get install software-properties-common
sudo apt-get install apt-transport-https
wget -qO - https://packages.confluent.io/deb/4.1/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.1 stable main"
sudo apt-get update
sudo apt-get install -y -V librdkafka-dev

The following packages have unmet dependencies:
 librdkafka-dev : Depends: librdkafka1 (= 0.11.4~1confluent4.1.1-1) but it is not going to be installed
                  Depends: librdkafka++1 (= 0.11.4~1confluent4.1.1-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Changed it to

sudo apt-get install -y -V librdkafka-dev librdkafka1

Then get this.

The following packages have unmet dependencies:
 librdkafka1 : Depends: libssl1.0.0 (>= 1.0.0) but it is not installable
E: Unable to correct problems, you have held broken packages.
Exited with code 100

I've tried installing libssl-dev:1.1.0f and openssl and it hasn't helped.

openssl is already the newest version (1.1.0f-3+deb9u2).

[edenhill]

@martyphee What version of Ubuntu/Debian is this?

[martyphee]

It should be the latest.

[edenhill]

lsb_release -a

[martyphee]

Distributor ID: Debian
Description:    Debian GNU/Linux 9.4 (stretch)
Release:    9.4
Codename:   stretch

[martyphee]

@edenhill Bad version? Seems like the latest.

[edenhill]

~Confluent platform, and its clients, are not supported on Debian 9: https://docs.confluent.io/current/installation/versions-interoperability.html~

~(but we'll fix this!)~

Try getting librdkafka 0.11.4 from debian sid:
https://packages.debian.org/search?keywords=librdkafka1&searchon=names&suite=unstable&section=all

[edenhill]

Sorry, my bad, Debian 9 (thru Ubuntu 16.04) is supported, but the librdkafka package is not compatible, which is a bug.

We're on it.

[garlicbulb-puzhuo]

What is the highest compatible versions of confluent-kafka on librdkafka1(-dev) on Ubuntu 16.04 env?

[edenhill]

You should be using the latest version of both the go client and librdkafka, so using an older version is not really recommended.

I would suggest installing librdkafka v0.11.4 from Debian sid:
https://packages.debian.org/sv/sid/librdkafka1

[BrainMonkey]

I'm also having the same issue on alpine

FROM golang:1.10.3-alpine3.7

RUN apk -U add ca-certificates

# Install librdkafka
RUN apk add librdkafka-dev build-base

pulls version 0.11.1-r1 .

I also tried and set version with RUN apk add librdkafka-dev=0.11.4-r1 and can not find so it defaults to 0.11.1.-r1 also

[eduardompinto]

I had the same issue.

This worked for me:

apt-get update && apt-get install -y apt-transport-https ca-certificates git curl \
wget pkg-config software-properties-common python3-software-properties

wget -qO - https://packages.confluent.io/deb/5.0/archive.key | apt-key add -

add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/5.0 stable main"

apt-get update && apt-get install -y librdkafka-dev librdkafka1