Confluent-kafka-go: Got trouble with Alpine (Using docker) when install librdkafka-dev (Require librdkafka ver 1.1.0)

[QuanVan95]

Description

Hi Mr. Edenhill,
I'm trying to create a consumer with golang following your tutorial: https://github.com/confluentinc/confluent-kafka-go

I know that "This client for Go depends on librdkafka v1.1.0 or later...". I'm using docker with the base image is golang:1.11.12-alpine3.10, it seems the latest version of Alpine.

When I run dockerfile, the maximum version of librdkafka is only librdkafka-1.0.1-r1. I also tried to run it on MacOS and it worked well. I tried to find a solution, and one man told me that "use the alpine community for librdkafka v1.1.0". To be honest, I don't know how I can resolve this problem and upgrade librdkafka version with Alpine on Docker.

Could you please give me a solution?

How to reproduce

My dockerfile:
FROM golang:1.11.12-alpine3.10 AS build
RUN apk add --no-cache git
RUN apk add librdkafka-dev pkgconf build-base
WORKDIR /go/src/project/
COPY . /go/src/project/
RUN go get ./... => error because of librdkafka version

Checklist

Please provide the following information:

• [ ] confluent-kafka-go and librdkafka version (LibraryVersion()): librdkafka 1.0.1-r1
• [ ] Apache Kafka broker version:
• [ ] Client configuration: ConfigMap{...}
• [ ] Operating system: Alpine 3.10
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

We don't provide the librdkafka APK packages, so your best bet until it is updated by the community is to build librdkafka from source. This is quite straight forward with ./configure --install-deps since it will also install the required dependencies for you.

We'll soon provide librdkafka as part of the Go client to avoid external dependencies, please try it out (for testing, not production!) by changing your imports from ...confluent-kafka-go to ...confluent-kafka-go-dev.

Example here: https://github.com/confluentinc/confluent-kafka-go-example

[QuanVan95]

We don't provide the librdkafka APK packages, so your best bet until it is updated by the community is to build librdkafka from source. This is quite straight forward with ./configure --install-deps since it will also install the required dependencies for you.

We'll soon provide librdkafka as part of the Go client to avoid external dependencies, please try it out (for testing, not production!) by changing your imports from ...confluent-kafka-go to ...confluent-kafka-go-dev.

Example here: https://github.com/confluentinc/confluent-kafka-go-example

Thank Mr. Edenhill, I will try it to run Go with Kafka. Hope that you will "provide librdkafka as part of the Go client to avoid external dependencies soon".
Once again, thank you so much for your support <3

[alokkx]

@QuanVan95 Can you please share the complete Dockerfile?
Did the above trick work for you?

[obitech]

In case anyone is still trying to figure this out, you can install the latest version from the edge branch:

FROM golang:1.13.6-alpine3.10

RUN echo "@edge http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories
RUN echo "@edgecommunity http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories  
RUN apk add --no-cache alpine-sdk 'librdkafka@edgecommunity>=1.3.0' 'librdkafka-dev@edgecommunity>=1.3.0'

Be aware though, that pinning repos on edge is risky:

[...] most Alpine package repos contain an "edge" branch, which may drop package versions that are not deemed fit to make it into a stable branch. This means that pinning to a version on the edge branch may stop working after the package version is revoked from the repo.