Confluent-kafka-go: Build docker image with librdkafka

[apanagiotou]

Description

I created a Go consumer and I'm trying to build the docker image but it fails due to librdkafka.

My Dockerfile is this:

FROM golang:1.10.3-alpine
WORKDIR /go/src/github.com/apanagiotou/go-kafka-to-s3/
COPY . .
RUN apk add librdkafka
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o go-kafka-to-s3 .

FROM scratch
COPY --from=0 /go/src/github.com/apanagiotou/go-kafka-to-s3 .
ENTRYPOINT ["./go-kafka-to-s3"]

I assume that librdkafka is not enough but I cannot find any references on how to install it. Any guidance would be much appreciated.

How to reproduce

Building the docker produces the following error

ending build context to Docker daemon  2.969MB
Step 1/8 : FROM golang:1.10.3-alpine
 ---> 34d3217973fd
Step 2/8 : WORKDIR /go/src/github.com/apanagiotou/go-kafka-to-s3/
 ---> Using cache
 ---> 11afc81ec84d
Step 3/8 : COPY . .
 ---> f1640966d699
Step 4/8 : RUN apk add librdkafka
 ---> Running in 13c0eb07de47
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/3) Installing libgcc (6.4.0-r8)
(2/3) Installing libstdc++ (6.4.0-r8)
(3/3) Installing librdkafka (0.11.4-r1)
OK: 7 MiB in 17 packages
Removing intermediate container 13c0eb07de47
 ---> 8ef3a9bbf803
Step 5/8 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o go-kafka-to-s3 .
 ---> Running in 8af845e451e4
kafka/kafka.go:6:2: build constraints exclude all Go files in /go/src/github.com/apanagiotou/go-kafka-to-s3/vendor/github.com/confluentinc/confluent-kafka-go/kafka
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o go-kafka-to-s3 .' returned a non-zero code: 1

Checklist

Please provide the following information:

• [ ] Operating system: MacOS

[edenhill]

I'm guessing the build fails because you set CGO_ENABLED=0, and confluent-kafka-go is a cgo package.
Remove that part of your build commands.

[yacn]

^ That's exactly it. Additionally when you stop disabling CGO, while it will build, the binary won't work in a scratch container because said container will not have librdkafka inside.

I think there's work being done on building a completely static binary with this library that has to do with the go build tag static_all.

[apanagiotou]

Hello and thanks for the suggestions. I've fixed it. FYI I had to remove the CGO_ENABLED and, add the build-base package because is needed by librdkafka. I also, omitted the builder method because there are a few files that the last image needs from librdkafka and build-base and I didn't want to get involved with that. Here is the image that works:

FROM golang:1.10.3-alpine
WORKDIR /go/src/github.com/apanagiotou/go-kafka-to-s3/
RUN apk add librdkafka-dev build-base
COPY . .
RUN GOOS=linux go build -a -o go-kafka-to-s3 .
ENTRYPOINT ["./go-kafka-to-s3"]

[ankurjha7]

In case you are looking for an even lighter image since golang alpine image loads go depedendency you can just use alpine image in the second stage in a multi stage docker build . Something like

FROM golang:1.10.3-alpine as builder
WORKDIR /go/src/github.com/apanagiotou/go-kafka-to-s3/
RUN apk add librdkafka-dev build-base
COPY . .
RUN GOOS=linux go build -a -o go-kafka-to-s3 .
ENTRYPOINT ["./go-kafka-to-s3"]

use a minimal alpine image for second stage

FROM alpine:edge
WORKDIR /root
COPY --from=builder /go/src/github.com/apanagiotou/go-kafka-to-s3/go-kafka-to-s3 .
RUN apk add --no-cache librdkafka
ENTRYPOINT ["/root/go-kafka-to-s3"]

The resulting image will be much lighter