Confluent-kafka-go: cannot find module for path github.com/confluentinc/confluent-kafka-go/kafka/librdkafka

[rogaha]

Description

I'm not able to run go test ./... on ubuntu (GitHub Actions).

Error msg:

cannot find module for path github.com/confluentinc/confluent-kafka-go/kafka/librdkafka
84
##[error]Process completed with exit code 1.

go get ./... works.

go: extracting github.com/tdewolff/minify v2.3.6+incompatible
140
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0
141
go: extracting github.com/cpuguy83/go-md2man/v2 v2.0.0
142
go: downloading github.com/russross/blackfriday v1.5.2
143
go: extracting golang.org/x/sys v0.0.0-20200620081246-981b61492c35
144
go: extracting github.com/russross/blackfriday v1.5.2
145
go: downloading github.com/russross/blackfriday/v2 v2.0.1
146
go: extracting github.com/russross/blackfriday/v2 v2.0.1
147
go: downloading github.com/tinylib/msgp v1.1.2
148
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
149
go: downloading github.com/tdewolff/parse v2.3.4+incompatible
150
go: downloading gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc
151
go: downloading github.com/go-playground/form v3.1.4+incompatible
152
go: extracting github.com/tinylib/msgp v1.1.2
153
go: extracting github.com/shurcooL/sanitized_anchor_name v1.0.0
154
go: downloading golang.org/x/text v0.3.3
155
go: extracting github.com/tdewolff/parse v2.3.4+incompatible
156
go: extracting gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc
157
go: downloading github.com/philhofer/fwd v1.0.0
158
go: extracting github.com/philhofer/fwd v1.0.0
159
go: extracting github.com/go-playground/form v3.1.4+incompatible
160
go: extracting github.com/go-playground/locales v0.13.0
161
go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
162
go: extracting github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
163
go: downloading github.com/confluentinc/confluent-kafka-go v1.4.2
164
go: extracting golang.org/x/text v0.3.3
165
go: extracting github.com/confluentinc/confluent-kafka-go v1.4.2

How to reproduce

$ go test ./...

librdkafka installed

        sudo apt-get update && sudo apt-get install build-essential pkg-config git librdkafka-dev
        git clone https://github.com/edenhill/librdkafka.git
        cd librdkafka && git checkout v1.4.2
        sudo ./configure --prefix /usr
        sudo make
        sudo make install

[edenhill]

You should not need to install librdkafka since it is bundled with the Go client since v1.4.0.

Can you try to reproduce this with go test -v -x ./... so we can see what is going on?

[divo77]

Do you have example of the docker image where I don't need to install librdkafka ?

[edenhill]

You don't need to install librdkafka in or outside of docker at all.

[seremenko-wish]

Check CGO_ENABLED env var, it must not be 0

[newcarrotgames]

I'm seeing this error from our build system (GitLab), and I've confirmed the CGO_ENABLED env var is 0. I'm using v1.5.2 according to go.mod. I'm able to build locally on Ubuntu (via WSL), but I'm seeing the cannot find module for path github.com/confluentinc/confluent-kafka-go/kafka/librdkafka error in the gitlab build logs. I noticed this note from the readme:

If you are building for Alpine Linux (musl), -tags musl must be specified.
go build -tags musl ./...

Our build system is using golang:1.13-alpine as the base image, but the build script isn't using the command line parameter listed above (-tags musl). Would omitting those tags lead to this error specifically somehow? I'm investigating ways I can pass in flags through our CI configuration, but was hoping someone here could confirm/deny this.

EDIT: I just tried adding the -tags musl parameter and no luck there, still seeing this error.

[newcarrotgames]

I created a github repo to reproduce this issue:

https://github.com/newcarrotgames/confluent-kafka-test

If you clone that repo, and run docker build -t conkafkatest . in the same folder you cloned the repo in, you should see this error.

EDIT - If something happens to the repo, here's the files you need to reproduce the issue (you need to run go mod init && go mod tidy before the docker build):

Dockerfile:

FROM golang:1.13-alpine AS builder

ENV GOOS linux
ENV GOARCH amd64
ENV CGO_ENABLED 0

RUN apk update && apk upgrade && \
    apk add --no-cache git

WORKDIR /build

COPY . ./

# confluent-kafka-go recommends musl tags for alpine linux builds
RUN go build -tags musl

main.go (just needed for the import):

package main

import (
    "fmt"

    "github.com/confluentinc/confluent-kafka-go/kafka"
)

func main() {
    fmt.Printf("%+v", kafka.ConfigMap{})
}

[edenhill]

It seems like the internal import of librdkafka_vendor/ (previously librdkafka/) is unversioned, which makes sense but is annoying.
We'd really just like to have a relative import, but that doesn't seem to be supported.

Could a workaround be to add the librdkafka_vendor/ directory as a go mod import?
cc @KJTsanaktsidis

[newcarrotgames]

Was able to get this to work using this dockerfile:

FROM golang:1.13-alpine AS builder

ENV GOOS linux
ENV GOARCH amd64
# missed this very obvious change 
ENV CGO_ENABLED 1

RUN apk update && apk upgrade && \
    apk add --no-cache git

# needed for gcc
RUN apk add --no-cache git build-base

WORKDIR /build

COPY . ./

# confluent-kafka-go recommends musl tags for alpine linux builds
RUN go build -tags musl

[KJTsanaktsidis]

Heya, just got back from holidays and stared at it a bit until the answer hit me in the face 😄 Of course, you can't compile a confluent-kafka-go app without cgo enabled because it links against the librdkafka C library.

I suppose this is the error message you were trying to improve on in https://github.com/confluentinc/confluent-kafka-go/commit/d2e0d83bd58738501f8c98a6785e2e0de07b5f3d and had to revert because of https://github.com/confluentinc/confluent-kafka-go/issues/581 .

I definitely see how this is a very annoying slipup for everyone to keep making and a better error message would be nice. Unfortunately I can't think of any good way to do this....