Confluent-kafka-go: Cross-compilation support

[surendrabaid]

Description

I'm facing same problem now, not able to build for linux . I'm building on my mac.

skbaid$ make release OS=linux ARCH=amd64 VERSION=2.00.18
GOPATH=/Users/skbaid/GoglandProjects/abc-service-adapter GOARCH="amd64" GOOS="linux"
go build -ldflags "-w -s -X main.Version=2.00.18 -X main.Name=abc-service-adapter" -o bin/abc-service-adapter src/abc/main.go
go build github.com/confluentinc/confluent-kafka-go/kafka: build constraints exclude all Go files in /Users/skbaid/GoglandProjects/abc-service-adapter/src/github.com/confluentinc/confluent-kafka-go/kafka
make: * [release] Error 1

has anyone found solution for this ?

[edenhill]

Cross compilation is not currently supported

[surendrabaid]

Could you please elaborate?
when i use command 'make release OS=#darwin ARCH=amd64 VERSION=2.00.18' , build is successful but when i run 'make release OS=# linux ARCH=amd64 VERSION=2.00.18' , build is failing.

[edenhill]

We currently do not support building for another OS or ARCH than you are currently on:
in your case you try to build a Linux binary on an OSX machine.

[nkristianto]

hi @edenhill
does this case solved on the new version of confluent-kafka-go?

[edenhill]

@nkristianto Sorry no, cross builds are currently not supported.

Having said that, cross compilation is most likely possible but it will take a fair bit of environment-specific work to get working. Please share your success.

[yctn]

Would be nice to get this fixed in the future

[darkdefender27]

Definitely need cross compilation. :/

[Smadarl]

I was able to get this to work using XGo: https://github.com/karalabe/xgo.
I created a simple docker image based off their generic image (5GB), using this docker file:

FROM karalabe/xgo-latest
RUN git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure && make && make install && cd .. && rm -rf librdkafka

Then, I was able to compile my program like this:
xgo --targets=linux/amd64 -image build-go-kafka ./consumers/actions
Worked like a charm.

[mikekamornikov]

@edenhill I'm trying to use confluent-kafka-go in a monorepo driven by bazel. The thing is most of our devs use OSX for daily work while the final producer/consumer services are dockerized. And it means we need to cross-compile anyway. I wonder:

• How hard it is to make it possible to cross-compile librdkafka?
• Is it super-hard or just low priority because no one is asking for it?
• How do devs in the wild use confluent-kafka-go? Especially what is the recommended way to handle librdkafka now? Should I prepare a docker base with everything ready OOB? Should I prepare cc toolchain somehow to cross-compile? Please advice

[jubyvictor]

Trying to use v 1.4.2, go build for linux fails on OSX. Assuming its a cross-compilation issue ? go build for osx succeeds.

` ebuspub git:(master) ✗ GOOS=linux GOARCH=amd64 go build

ebuspub/ebusnative

ebusnative/ebusnative.go:21:20: undefined: kafka.Producer
ebusnative/ebusnative.go:53:23: undefined: kafka.NewProducer
ebusnative/ebusnative.go:53:42: undefined: kafka.ConfigMap
ebusnative/ebusnative.go:83:37: undefined: kafka.Message
ebusnative/ebusnative.go:97:9: undefined: kafka.Message
`

[shivakumarss]

This is very much required. Please prioritise this ticket.

[edenhill]

We recommend that you use a docker container to build non-native binaries, if possible.

[michaelcoder33]

This is still a issue!?

[NewestUser]

I have a similar issue.
I try to build the project inside a docker container.
The build fails with:

../kfka/producer.go:11:12: undefined: kafka.NewProducer
../kfka/producer.go:11:31: undefined: kafka.ConfigMap
../kfka/producer.go:23:5: undefined: kafka.Producer
../kfka/producer.go:32:21: undefined: kafka.Message
ERROR: Service 'my-app' failed to build : The command '/bin/sh -c go mod download && cd cmd && go build -o my-app' returned a non-zero code: 2

At the same time when I build it on my osx laptop it succeeds.
Here is the docker file that I use to run the build in:

FROM golang:1.14.6-alpine3.12

# create a build dir
RUN mkdir /build
WORKDIR /build

# it seems gcc is not available on plain alpine
ENV CGO_ENABLED=0

# copy module dependencies
COPY go.mod .
COPY go.sum .

# copy the project itself
COPY . .

RUN go mod download && cd cmd && go build -o my-app

CMD ["./cmd/my-app"]

Here is my go.mod file

module github.com/newestuser/my-app

go 1.14

require (
    github.com/confluentinc/confluent-kafka-go v1.4.2
    github.com/go-playground/universal-translator v0.17.0 // indirect
    github.com/gorilla/mux v1.8.0
    github.com/leodido/go-urn v1.2.0 // indirect
    github.com/stretchr/testify v1.6.1
    go.mongodb.org/mongo-driver v1.4.2
    gopkg.in/go-playground/assert.v1 v1.2.1 // indirect
    gopkg.in/go-playground/validator.v9 v9.31.0
)

[dotmodusgeorge]

@NewestUser

Check out https://github.com/confluentinc/confluent-kafka-go#librdkafka.

When building on Alpine it requires additional flags to be set.

[NewestUser]

@dotmodusgeorge thanks for pointing this out.
I confirm that if I remove ENV CGO_ENABLED=0 from my Dockerfile the code compiles successfully.

[mejedi]

I wonder why it is not fixed in 2 years. Our project has recently switched from another Kafka connector, and cross compilation stopped working. This is a serious usability issue.

@edenhill Could you please share what are the issues preventing cross-compilation?

[edenhill]

The standard Go toolchain does not support cross-compilation of CGo out of the box, we recommend using docker for this purpose.
This looks promising: https://github.com/gythialy/golang-cross

[mejedi]

@edenhill Wasn't aware that the standard Go toolchain does not support cross-compilation of CGo out of the box. Thank you for the info.