Confluent-kafka-go: How to build against Linux targets?

[mcandre]

When I use GOOS=linux go build against confluent-kafka-go, Go complains "no buildable Go source files".

Which is too bad, because I would like to be able to create a Linux, AMD64 binary of a Kafka app for use in Docker.

[edenhill]

Are you attempting to cross-compile it on osx?
That's not currently supported, I suggest you build it on a linux machine.

[mcandre]

Poop, debian jesse only has librdkafka-dev 0.8.5-2, but we need 0.9.2+.

[edenhill]

You sure do!

[mcandre]

Hmm, with librdkafka-0.9.3 and a Docker environment, go build says:

# export GOOS=linux
# export GOARCH=amd64
# go build
# github.datapipe.net/platform/brushpass/vendor/github.com/confluentinc/confluent-kafka-go/kafka
../../vendor/github.com/confluentinc/confluent-kafka-go/kafka/producer.go: In function 'do_produce':
../../vendor/github.com/confluentinc/confluent-kafka-go/kafka/producer.go:46:9: error: 'RD_KAFKA_V_END' undeclared (first use in this function)
         RD_KAFKA_V_END);
         ^
../../vendor/github.com/confluentinc/confluent-kafka-go/kafka/producer.go:46:9: note: each undeclared identifier is reported only once for each function it appears in

# dpkg --list librdkafka-dev
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                    Version                  Architecture             Description
+++-=======================================-========================-========================-===================================================================================
ii  librdkafka-dev:amd64                    0.9.3-1                  amd64                    library implementing the Apache Kafka protocol (development headers)

Is there a possibility that librdkafka 0.9.3 breaks kafka-go?

[edenhill]

That was an error in the Go client that was fixed recently:
https://github.com/confluentinc/confluent-kafka-go/commit/80d7f30be463f212d56e4381c633ff0963eb3cb1

Upgrade the client to latest master and it should work

[mcandre]

Cheers, that patch is helpful.

Which git ref should we use to obtain librdkafka source, since v0.9.4 isn't tagged yet?

[edenhill]

If you want a stable release you'll need to wait for 0.9.4 (probably within a month), otherwise be brave and use master (for development, PoC)

[mcandre]

Note: Docker users can cross-compile to Linux by mounting source directory into a container, installing librdkafka-dev and Go, and compiling with go build --ldflags '-extldflags "-static"'. This produces a Linux binary in the host directory, like ./cmd/<app>/<app>. Or simply go build, if dynamic linking is okay for some project.

Mac cGo binaries could theoretically be cross-compiled from non-Mac hosts by a Darwin virtual machine, though no idea what installing gcc / Xcode is like for Darwin.

If someone could produce a minimal Windows Vagrant box, that would help to make build bots.

Finally, if one day confluent-kafka-go replaced librdkafka with pure Go code, then cross-compilation would be as easy as tweaking the GOOS, GOARCH environment variables.

[edenhill]

@mcandre Does this mean that go build -tags static didn't work for you?

[mcandre]

@edenhill go build can't target GOOS=linux from a Mac host when C dependencies are involved.

[surendrabaid]

I'm facing same problem now.

skbaid$ make release OS=linux ARCH=amd64 VERSION=2.00.18
GOPATH=/Users/skbaid/GoglandProjects/abc-service-adapter GOARCH="amd64" GOOS="linux" \
go build -ldflags "-w -s -X main.Version=2.00.18 -X main.Name=abc-service-adapter" -o bin/abc-service-adapter src/abc/main.go
go build github.com/confluentinc/confluent-kafka-go/kafka: build constraints exclude all Go files in /Users/skbaid/GoglandProjects/abc-service-adapter/src/github.com/confluentinc/confluent-kafka-go/kafka
make: * [release] Error 1

has anyone found solution for this ?