Confluent-kafka-go: Clang error when building with v1.4.0

[jhuggart]

Description

I upgraded to the latest version of the go client (v1.4.0) and ran brew upgrade librdkafka. But the client is looking for librdkafka in the package directory and it is not present.

How to reproduce

In an existing project that uses confluent-kafka-go, run go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka and go build ./...

You should receive something similar to:

clang: error: no such file or directory: '$GOHOME/src/github.com/your-user/your-service/vendor/gopkg.in/confluentinc/confluent-kafka-go.v1/kafka/librdkafka
/librdkafka_darwin.a'

Checklist

Please provide the following information:

• [x] confluent-kafka-go and librdkafka version (LibraryVersion()): v1.4.0 (both)
• [ n/a ] Apache Kafka broker version:
• [ n/a ] Client configuration:
• [x] Operating system: Mac OS X 10.15.4
• [ n/a ] Provide client logs
• [ n/a ] Provide broker log excerpts
• [ ] Critical issue

[jhuggart]

I fixed this by manually copying in the static librdkafka deps. But it looks like that dir is not making it into the published module.

[solarhell]

I ran into the same issue:

# github.com/confluentinc/confluent-kafka-go/kafka
vendor/github.com/confluentinc/confluent-kafka-go/kafka/00version.go:24:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.

[edenhill]

My guess would be that you already have gopkg.in...v1 cached somewhere (to an older version, e.g. v1.3.0), try removing whatever confluent-kafka-go caches you find in $GOHOME and try again.

[solarhell]

after run go clean -modcache and go mod tidy && go mod vendor
i got this issue
# github.com/confluentinc/confluent-kafka-go/kafka clang: error: no such file or directory: '/Users/solarhell/go/src/github.com/solarhell/confluent-kafka-go-consumer/vendor/github.com/confluentinc/confluent-kafka-go/kafka/librdkafka/librdkafka_darwin.a'

[edenhill]

can you do ls -laR /Users/solarhell/go/src/github.com/solarhell/confluent-kafka-go-consumer/vendor/github.com/confluentinc/confluent-kafka-go/kafka/ and provide the output?

[edenhill]

It looks like your vendor:ed confluent-kafka-go is missing the kafka/librdkafka directory. Make sure it is at version/tag v1.4.0

[solarhell]

i am sure i am using the 1.4.0 version.

the output is

total 984
drwxr-xr-x  32 solarhell  staff    1024 Apr  9 19:41 .
drwxr-xr-x   4 solarhell  staff     128 Apr  9 19:41 ..
-rw-r--r--   1 solarhell  staff      50 Apr  9 19:41 .gitignore
-rw-r--r--   1 solarhell  staff    2087 Apr  9 19:41 00version.go
-rw-r--r--   1 solarhell  staff    3235 Apr  9 19:41 README.md
-rw-r--r--   1 solarhell  staff   34496 Apr  9 19:41 adminapi.go
-rw-r--r--   1 solarhell  staff    7221 Apr  9 19:41 adminoptions.go
-rw-r--r--   1 solarhell  staff  225603 Apr  9 19:41 api.html
-rw-r--r--   1 solarhell  staff     402 Apr  9 19:41 build_darwin.go
-rw-r--r--   1 solarhell  staff     210 Apr  9 19:41 build_dynamic.go
-rw-r--r--   1 solarhell  staff     420 Apr  9 19:41 build_glibc_linux.go
-rw-r--r--   1 solarhell  staff     432 Apr  9 19:41 build_musl_linux.go
-rw-r--r--   1 solarhell  staff    7705 Apr  9 19:41 config.go
-rw-r--r--   1 solarhell  staff   24171 Apr  9 19:41 consumer.go
-rw-r--r--   1 solarhell  staff     967 Apr  9 19:41 context.go
-rw-r--r--   1 solarhell  staff    4434 Apr  9 19:41 error.go
-rw-r--r--   1 solarhell  staff    2911 Apr  9 19:41 error_gen.go
-rw-r--r--   1 solarhell  staff   10185 Apr  9 19:41 event.go
-rw-r--r--   1 solarhell  staff   21804 Apr  9 19:41 generated_errors.go
-rw-r--r--   1 solarhell  staff    1180 Apr  9 19:41 glue_rdkafka.h
-rw-r--r--   1 solarhell  staff    9431 Apr  9 19:41 handle.go
-rw-r--r--   1 solarhell  staff    1942 Apr  9 19:41 header.go
-rw-r--r--   1 solarhell  staff   15294 Apr  9 19:41 kafka.go
-rw-r--r--   1 solarhell  staff    2240 Apr  9 19:41 log.go
-rw-r--r--   1 solarhell  staff    5844 Apr  9 19:41 message.go
-rw-r--r--   1 solarhell  staff    5121 Apr  9 19:41 metadata.go
-rw-r--r--   1 solarhell  staff     854 Apr  9 19:41 misc.go
-rw-r--r--   1 solarhell  staff    3965 Apr  9 19:41 offset.go
-rw-r--r--   1 solarhell  staff   30378 Apr  9 19:41 producer.go
-rw-r--r--   1 solarhell  staff     192 Apr  9 19:41 testconf-example.json
-rw-r--r--   1 solarhell  staff    5650 Apr  9 19:41 testhelpers.go
-rw-r--r--   1 solarhell  staff    1508 Apr  9 19:41 time.go

[edenhill]

It is missing the librdkafka directory, which I think is because it does not have any .go files in it.
But this seems to work for most people, so I wonder what is different with your setup / tooling.

[solarhell]

I can provide my dockerfile and sample app in a repo. Wait for a moment, pls.

[edenhill]

That'd be great, thank you!

[solarhell]

@edenhill https://github.com/solarhell/confluentKafkaGoExample

I am using alpine(musl linux) and i can see the similar error output when i was using macOS.

# github.com/confluentinc/confluent-kafka-go/kafka gcc: error: vendor/github.com/confluentinc/confluent-kafka-go/kafka/librdkafka/librdkafka_glibc_linux.a: No such file or directory

Now i must use -tags dynamic to use the system's librdkafka.

Please check the two dockerfile's difference.

[edenhill]

Seems to be hitting this issue of go mod vendor:
https://github.com/golang/go/issues/26366

[edenhill]

Let's see if sticking a .go file in librdkafka/ solves the issue.

[solarhell]

Last time i ran into the similar issue and i used the following code solved the issue.

import (
    _ "mypackage"
)

[edenhill]

@solarhell Can you try if a

import _ "librdkafka"

from e.g. kafka/kafka.go fixes the problem on your end?

[solarhell]

The root problem is that the "librdkafka" package is not vendored.
So i can't import it.

[edenhill]

Can you try importing the vendor again using tag v1.0.7-RC1 (this is a dummy tag that will be removed, it is actually 1.4.0+)?

The librdkafka/ subdir should now be included.

Thanks

[solarhell]

It works!

And the program works fine.

[edenhill]

Whoop!

Thank you so much for the quick testing!

I'll prepare a new release, v1.4.1, to fix this proper.

[solarhell]

Thanks for your work! 🎉

[edenhill]

Just to verify, the building and running of your application now works correctly with the bundled librdkafka, yes?

[solarhell]

Yes. I used the producer example to verify it on macOS.

[edenhill]

Perfect, thank you!

[blaketastic2]

Should I be using v1.0.7-RC1, or is there a v1.4.1 version with this fix?

[jhuggart]

Any updates on this @edenhill ? I tried go get gopkg.in/confluentinc/[email protected] but get invalid version: unknown revision v1.0.7-RC1.

[fnuneztw]

I can't find any v1.0.7-RC1 tag to use as a temporary fix as mentioned here. Was it already removed?

[edenhill]

Yes, it was removed, we're aiming to release a proper v1.4.2 this week, will have a release-candidate ready soon for you to try out.

[edenhill]

Verified on golang:1.13.10-alpine3.10 (docker):

librdkafka link information: static musl_linux from librdkafka-static-bundle-v1.4.2-RC1.tgz
%7|1587992099.237|OPENSSL|rdkafka#producer-1| [thrd:app]: librdkafka built with statically linked OpenSSL version 0x1000215f

%7|1587992099.237|CACERTS|rdkafka#producer-1| [thrd:app]: Setting default CA certificate location to /etc/ssl/cert.pem, override with ssl.ca.location

^^^^ this is the probed location

%7|1587992099.249|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.2-RC1 (0x10402ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, STATIC_LINKING GCC GXX PKGCONFIG GNULD LDS C11THREADS LIBDL PLUGINS STATIC_LIB_zlib ZLIB STATIC_LIB_libcrypto STATIC_LIB_libssl SSL STATIC_LIB_libzstd ZSTD HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x200)

%7|1587992099.391|SSLVERIFY|rdkafka#producer-1| [thrd:sasl_ssl://.....confluent.cloud:9092/boot]: sasl_ssl://....confluent.cloud:9092/bootstrap: Broker SSL certificate verified

[fnuneztw]

I guess now I can use librdkafka v1.4.2-RC1 as a temporary fix, right?

[edenhill]

librdkafka still needs to be bundled with the Go client.
To keep the Go repo size down we'll only do this once librdkafka v1.4.2 is released which is hopefully later this week.

[edenhill]

The workaround for Alpine is to set "ssl.ca.location": "/etc/ssl/cert.pem" in the ConfigMap

[daniesy]

Any news on the release? Or a temporary fix?

[edenhill]

Fixed in v1.4.2

[kutty-kumar]

@edenhill this isn't working using dep

[edenhill]

@kutty-kumar Please provide more information. Steps to reproduce, error messages, file listings, mod files, etc

[mostafatalebi]

The problem is fixed with v1.4.2. But we should import
_ "gopkg.in/confluentinc/confluent-kafka-go.v1/kafka/librdkafka"

Thanks!