Confluent-kafka-go: Package rdkafka was not found in the pkg-config search path.

Created on 30 Aug 2018 · 15Comments · Source: confluentinc/confluent-kafka-go

Description

[root@localhost librdkafka]# go get -u github.com/confluentinc/confluent-kafka-go/kafka

pkg-config --cflags rdkafka

Package rdkafka was not found in the pkg-config search path.
Perhaps you should add the directory containing `rdkafka.pc'
to the PKG_CONFIG_PATH environment variable
No package 'rdkafka' found
pkg-config: exit status 1

anything missing ?
on centos

How to reproduce

Checklist

Please provide the following information:

[ ] confluent-kafka-go and librdkafka version (LibraryVersion()):
[ ] Apache Kafka broker version:
[ ] Client configuration: ConfigMap{...}
[ ] Operating system:
[ ] Provide client logs (with "debug": ".." as necessary)
[ ] Provide broker log excerpts
[ ] Critical issue

Source

OrangePieiep

Most helpful comment

brew install librdkafka

antsbean on 3 Jul 2019

👍15 🚀5 ❤5 🎉4

All 15 comments

See installation instructions here:
https://github.com/confluentinc/confluent-kafka-go#getting-started

edenhill on 30 Aug 2018

👎13 👍1

I try to build from source code.
use steps below.

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
./configure --prefix /usr
make
sudo make install

I got the same error.

[dev@bastion librdkafka]$ go get -u github.com/confluentinc/confluent-kafka-go/kafka

pkg-config --cflags rdkafka

Package rdkafka was not found in the pkg-config search path.
Perhaps you should add the directory containing `rdkafka.pc'
to the PKG_CONFIG_PATH environment variable
No package 'rdkafka' found

not work for centos ?

OrangePieiep on 30 Aug 2018

Interesting, can you provide the full output from the librdkafka build steps? (configure + make + install)

edenhill on 31 Aug 2018

Exactly the same issue there on VS Code and Windows 10 64.
"For Windows, see the librdkafka.redist NuGet package." instruction didn't make life easier.
Whole internet is full of questions on how to get pkg-config and confluent working in Windows, but no clear instructions are provided.
Could be great to provide CRYSTAL CLEAR step by step setup instruction for Windows Go developers in package documentation.
BTW - this helped for me on Windows - Some Blog

oswee on 2 Sep 2018

👍2

It seems the pkg-config problem, need to specify the searching path manually .

find the position of your "rdkafka.pc" file, In my centos it's /usr/lib/pkgconfig/

2.add the env variable to the file ~/.bash_profile
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/

3.source ~/.bash_profile and restart the terminal

4.then,go get -u github.com/confluentinc/confluent-kafka-go/kafka works

It works for me.
env: centos 7

woshidama323 on 26 Sep 2018

👍9

having the same issue, it works on my system. Upon deployment there is the same error

Edit: @woshidama323 answer:
apt-get install pkg-config
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/
seems to do the trick

amikar on 22 Oct 2018

👍3 👎1

For me it is not working. On my mac-osx darwin-amd64 machine the rdkafka.pc file is located at location: /usr/local/Cellar/librdkafka/0.11.6/lib/pkgconfig/rdkafka.pc and also at /usr/local/lib/pkgconfig/rdkafka.pc
I have set the path of env as below:
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/local/lib/pkgconfig/:/usr/local/Cellar/librdkafka/0.11.6/lib/pkgconfig/

but still getting error:

pkg-config --cflags -- rdkafka

GoutamHub on 3 Feb 2019

@GoutamHub what happens if you do:
PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/local/lib/pkgconfig/:/usr/local/Cellar/librdkafka/0.11.6/lib/pkgconfig/ pkg-config --libs rdkafka

edenhill on 4 Feb 2019

@edenhill I have executed the command in below sequence:
goutamdas@admins-MacBook-Pro-125 ~/Documents/go/src $ export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/local/lib/pkgconfig/:/usr/local/Cellar/librdkafka/0.11.6/lib/pkgconfig/
goutamdas@admins-MacBook-Pro-125 ~/Documents/go/src $ pkg-config --libs rdkafka
-L/usr/local/lib -lrdkafka
goutamdas@admins-MacBook-Pro-125 ~/Documents/go/src $ xgo github.com/confluentinc/confluent-kafka-go/kafka

ERROR:

pkg-config --cflags -- rdkafka

GoutamHub on 5 Feb 2019

@GoutamHub Oh, I see that you are trying to cross-compile. That is not currently officially supported with golang client due to native C dependencies. We're actively working on solving that though, but in the meantime I suggest you build your target binaries in the appropriate docker container for your target platform.

edenhill on 5 Feb 2019

brew install librdkafka

antsbean on 3 Jul 2019

👍15 🚀5 ❤5 🎉4

I tried everything said on this thread but this is the final result:

go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka

# gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AlterConfigs_result_resources':
/tmp/go-build/cgo-gcc-prolog:132: undefined reference to `rd_kafka_AlterConfigs_result_resources'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_is_read_only':
/tmp/go-build/cgo-gcc-prolog:151: undefined reference to `rd_kafka_ConfigEntry_is_read_only'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_is_sensitive':
/tmp/go-build/cgo-gcc-prolog:170: undefined reference to `rd_kafka_ConfigEntry_is_sensitive'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_is_synonym':
/tmp/go-build/cgo-gcc-prolog:189: undefined reference to `rd_kafka_ConfigEntry_is_synonym'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_name':
/tmp/go-build/cgo-gcc-prolog:207: undefined reference to `rd_kafka_ConfigEntry_name'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_source':
/tmp/go-build/cgo-gcc-prolog:226: undefined reference to `rd_kafka_ConfigEntry_source'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_synonyms':
/tmp/go-build/cgo-gcc-prolog:245: undefined reference to `rd_kafka_ConfigEntry_synonyms'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigEntry_value':
/tmp/go-build/cgo-gcc-prolog:263: undefined reference to `rd_kafka_ConfigEntry_value'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_configs':
/tmp/go-build/cgo-gcc-prolog:282: undefined reference to `rd_kafka_ConfigResource_configs'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_error':
/tmp/go-build/cgo-gcc-prolog:313: undefined reference to `rd_kafka_ConfigResource_error'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_error_string':
/tmp/go-build/cgo-gcc-prolog:331: undefined reference to `rd_kafka_ConfigResource_error_string'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_name':
/tmp/go-build/cgo-gcc-prolog:349: undefined reference to `rd_kafka_ConfigResource_name'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_new':
/tmp/go-build/cgo-gcc-prolog:369: undefined reference to `rd_kafka_ConfigResource_new'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_set_config':
/tmp/go-build/cgo-gcc-prolog:390: undefined reference to `rd_kafka_ConfigResource_set_config'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_type':
/tmp/go-build/cgo-gcc-prolog:409: undefined reference to `rd_kafka_ConfigResource_type'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigSource_name':
/tmp/go-build/cgo-gcc-prolog:428: undefined reference to `rd_kafka_ConfigSource_name'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_CreatePartitions_result_topics':
/tmp/go-build/cgo-gcc-prolog:463: undefined reference to `rd_kafka_CreatePartitions_result_topics'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_CreateTopics_result_topics':
/tmp/go-build/cgo-gcc-prolog:498: undefined reference to `rd_kafka_CreateTopics_result_topics'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DeleteTopic_new':
/tmp/go-build/cgo-gcc-prolog:528: undefined reference to `rd_kafka_DeleteTopic_new'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DeleteTopics_result_topics':
/tmp/go-build/cgo-gcc-prolog:563: undefined reference to `rd_kafka_DeleteTopics_result_topics'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DescribeConfigs_result_resources':
/tmp/go-build/cgo-gcc-prolog:598: undefined reference to `rd_kafka_DescribeConfigs_result_resources'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewPartitions_new':
/tmp/go-build/cgo-gcc-prolog:631: undefined reference to `rd_kafka_NewPartitions_new'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewPartitions_set_replica_assignment':
/tmp/go-build/cgo-gcc-prolog:656: undefined reference to `rd_kafka_NewPartitions_set_replica_assignment'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewTopic_new':
/tmp/go-build/cgo-gcc-prolog:690: undefined reference to `rd_kafka_NewTopic_new'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewTopic_set_config':
/tmp/go-build/cgo-gcc-prolog:711: undefined reference to `rd_kafka_NewTopic_set_config'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewTopic_set_replica_assignment':
/tmp/go-build/cgo-gcc-prolog:736: undefined reference to `rd_kafka_NewTopic_set_replica_assignment'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ResourceType_name':
/tmp/go-build/cgo-gcc-prolog:755: undefined reference to `rd_kafka_ResourceType_name'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_AlterConfigs_result':
/tmp/go-build/cgo-gcc-prolog:818: undefined reference to `rd_kafka_event_AlterConfigs_result'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_CreatePartitions_result':
/tmp/go-build/cgo-gcc-prolog:836: undefined reference to `rd_kafka_event_CreatePartitions_result'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_CreateTopics_result':
/tmp/go-build/cgo-gcc-prolog:854: undefined reference to `rd_kafka_event_CreateTopics_result'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_DeleteTopics_result':
/tmp/go-build/cgo-gcc-prolog:872: undefined reference to `rd_kafka_event_DeleteTopics_result'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_DescribeConfigs_result':
/tmp/go-build/cgo-gcc-prolog:890: undefined reference to `rd_kafka_event_DescribeConfigs_result'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_topic_result_error':
/tmp/go-build/cgo-gcc-prolog:1049: undefined reference to `rd_kafka_topic_result_error'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_topic_result_error_string':
/tmp/go-build/cgo-gcc-prolog:1067: undefined reference to `rd_kafka_topic_result_error_string'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_topic_result_name':
/tmp/go-build/cgo-gcc-prolog:1085: undefined reference to `rd_kafka_topic_result_name'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AdminOptions_destroy':
/tmp/go-build/cgo-gcc-prolog:100: undefined reference to `rd_kafka_AdminOptions_destroy'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AlterConfigs':
/tmp/go-build/cgo-gcc-prolog:116: undefined reference to `rd_kafka_AlterConfigs'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_ConfigResource_destroy':
/tmp/go-build/cgo-gcc-prolog:297: undefined reference to `rd_kafka_ConfigResource_destroy'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_CreatePartitions':
/tmp/go-build/cgo-gcc-prolog:447: undefined reference to `rd_kafka_CreatePartitions'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_CreateTopics':
/tmp/go-build/cgo-gcc-prolog:482: undefined reference to `rd_kafka_CreateTopics'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DeleteTopic_destroy':
/tmp/go-build/cgo-gcc-prolog:513: undefined reference to `rd_kafka_DeleteTopic_destroy'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DeleteTopics':
/tmp/go-build/cgo-gcc-prolog:547: undefined reference to `rd_kafka_DeleteTopics'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_DescribeConfigs':
/tmp/go-build/cgo-gcc-prolog:582: undefined reference to `rd_kafka_DescribeConfigs'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewPartitions_destroy':
/tmp/go-build/cgo-gcc-prolog:613: undefined reference to `rd_kafka_NewPartitions_destroy'
/tmp/go-build208544301/b001/_x003.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_NewTopic_destroy':
/tmp/go-build/cgo-gcc-prolog:671: undefined reference to `rd_kafka_NewTopic_destroy'
/tmp/go-build208544301/b001/_x004.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AdminOptions_new':
/tmp/go-build/cgo-gcc-prolog:53: undefined reference to `rd_kafka_AdminOptions_new'
/tmp/go-build208544301/b001/_x004.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AdminOptions_set_operation_timeout':
/tmp/go-build/cgo-gcc-prolog:76: undefined reference to `rd_kafka_AdminOptions_set_operation_timeout'
/tmp/go-build208544301/b001/_x004.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AdminOptions_set_request_timeout':
/tmp/go-build/cgo-gcc-prolog:99: undefined reference to `rd_kafka_AdminOptions_set_request_timeout'
/tmp/go-build208544301/b001/_x004.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_AdminOptions_set_validate_only':
/tmp/go-build/cgo-gcc-prolog:122: undefined reference to `rd_kafka_AdminOptions_set_validate_only'
/tmp/go-build208544301/b001/_x008.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_fatal_error':
/tmp/go-build/cgo-gcc-prolog:54: undefined reference to `rd_kafka_fatal_error'
/tmp/go-build208544301/b001/_x008.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_test_fatal_error':
/tmp/go-build/cgo-gcc-prolog:76: undefined reference to `rd_kafka_test_fatal_error'
/tmp/go-build208544301/b001/_x009.o: In function `chdrs_to_tmphdrs':
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/event.go:38: undefined reference to `rd_kafka_header_get_all'
/tmp/go-build208544301/b001/_x009.o: In function `_rk_queue_poll':
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/event.go:67: undefined reference to `rd_kafka_message_headers'
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/event.go:68: undefined reference to `rd_kafka_header_cnt'
/tmp/go-build208544301/b001/_x009.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_config_string':
/tmp/go-build/cgo-gcc-prolog:74: undefined reference to `rd_kafka_event_config_string'
/tmp/go-build208544301/b001/_x009.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_event_error_is_fatal':
/tmp/go-build/cgo-gcc-prolog:93: undefined reference to `rd_kafka_event_error_is_fatal'
/tmp/go-build208544301/b001/_x011.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_oauthbearer_set_token':
/tmp/go-build/cgo-gcc-prolog:95: undefined reference to `rd_kafka_oauthbearer_set_token'
/tmp/go-build208544301/b001/_x011.o: In function `_cgo_71e8d970da39_Cfunc_rd_kafka_oauthbearer_set_token_failure':
/tmp/go-build/cgo-gcc-prolog:115: undefined reference to `rd_kafka_oauthbearer_set_token_failure'
/tmp/go-build208544301/b001/_x018.o: In function `tmphdrs_to_chdrs':
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/producer.go:43: undefined reference to `rd_kafka_headers_new'
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/producer.go:46: undefined reference to `rd_kafka_header_add'
/tmp/go-build208544301/b001/_x018.o: In function `do_produce':
../../../../pkg/mod/gopkg.in/confluentinc/[email protected]/kafka/producer.go:112: undefined reference to `rd_kafka_headers_destroy'
collect2: error: ld returned 1 exit status

any help would be very appreciated

imans777 on 30 Sep 2019

@imans777 Your librdkafka is too old, in fact it seems that you have multiple versions of librdkafka installed.
Remove all versions of librdkafka on your system, then install v1.1.0 or later through the Confluent Platform APT/RPM repositories or build librdkafka from source and install.

The upcoming version of confluent-kafka-go will come with librdkafka prebuilt and bundled.

edenhill on 7 Oct 2019

👍2

@edenhill Yeah I figured it out right after I posted that comment :)
Because I tried apt-get and weren't successful and I tried installing from binary but I took the master branch and it wasn't the latest version. So I checked out to v1.2.0 and tried installing from binary again and everything worked as expected!

Thanks and sorry!

imans777 on 7 Oct 2019

👍1

It seems the pkg-config problem, need to specify the searching path manually .
1. find the position of your "rdkafka.pc" file, In my centos  it's /usr/lib/pkgconfig/
2.add the env variable to the file ~/.bash_profile
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/

3.source ~/.bash_profile and restart the terminal

4.then,go get -u github.com/confluentinc/confluent-kafka-go/kafka works

It works for me.
env: centos 7

hey actually ive kinda weird problem, i was installing go on wrong path it installed to usr/ instead of usr/local but i didnt care about it, and installed again to usr/local, so after awhile i was having the same problem kinda like this, but its solved,

and now i was deleting the folder from wrong install before, and much worse i delete them permanently because im new to ubuntu i just search how to delete home file, and i got this problem again, ive tried the pkg-config-path thing,

so to check pkg_config_path is by echo pkg_config_path right?, and after i set export pkg_config and restart terminal its still the same , no path was setted,

any idea please? i just got a job as a junior developer and i really screw by a lot of bugs and wrong configuration, error after error coming after me, depression day by day is increasing, im thinking that i should reinstall my ubuntu then go to install thing again if i dont have the solution, but i was hestitate because i was already installing ubuntu recently and got a lot of problem when opening my project,

please help me, much much thanks, and would be really big big appreciated guys, thank you