php-rdkafka self compiled for php7.1 not working

[geruetzel]

• PHP version: 7.1.31, rdkafka compiled against 7.1.15
• librdkafka version: 0.8.5, compiled agains >= 0.11
• php-rdkafka version: 3.1.3-dev/3.1.2
• kafka version: -

I manually compiled rdkafka for php7.2 and php7.1.

php7.2 seems to be working, at least the extension information shows up in the phpinfo() output and php7.2 starts without error messages.

rdkafka for php7.1 does not work when I compile it as I get the following error message on php7.1 startup:
[23-Aug-2019 11:45:33 UTC] PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20160303/rdkafka.so' - /usr/lib/php/20160303/rdkafka.so: undefined symbol: zend_ce_countable in Unknown on line 0
I tried it with php-rdkafka-version 3.1.2 and 3.1.3-dev, both with the same outcome.

My steps for compilation:

1. git clone
2. phpize7.1
3. ./configure
4. make all

[nick-zh]

@geruetzel only librdkafka 0.11.x upwards is compatible, 0.8.5 is too old

[geruetzel]

@nick-zh thanks for your reply.
if it is like you say, the documentation here
https://arnaud.le-blanc.net/php-rdkafka/phpdoc/rdkafka.requirements.html
is misleading/outdated.

[nick-zh]

@geruetzel thank you very much, we only updated the readme, i will take care of it!

[Steveb-p]

@geruetzel support for <0.10 was only recently dropped. 3.1.0 should work with older versions, but due to internal differences in librdkafka maintaining those old versions proved to cost a lot of additional code & maintenance.

[nick-zh]

@geruetzel let us know if it works properly for you with 0.11.x, i would advise to go with 1.1.0 if possible

[geruetzel]

@nick-zh @Steveb-p I get the same error with librdkafka 1.1.0 and php-rdkafka 3.1.2:
[24-Aug-2019 09:25:06 UTC] PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20160303/rdkafka.so' - /usr/lib/php/20160303/rdkafka.so: undefined symbol: zend_ce_countable in Unknown on line 0

root@php-misc:/usr/lib/php/20160303# ldd rdkafka.so 
linux-vdso.so.1 (0x00007ffd59ec4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0752c3a000)
librdkafka.so.1 => /usr/local/lib/librdkafka.so.1 (0x00007f0752891000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f07524f2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0753155000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f07522d7000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f075206b000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f0751bd2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f07519b8000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f07517b4000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0751597000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f075138f000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f0751178000)

As you can see here, rdkafka uses my self-compiled version of librdkafka. Both librdkafka and php-rdkafka were built on Debian Stretch, php7.1 is running on a Debian Stretch machine as well.

The same setup just works in php7.2 for me. The only difference in my build is using phpize7.2 for php7.2 before building php-rdkafka.

This is the output in the phpinfo() Page of my php72 now:

rdkafka support enabled
version 3.1.2
build date  Aug 24 2019 11:38:58
librdkafka version (runtime)    1.1.0
librdkafka version (build)  1.1.0.255

[nick-zh]

@geruetzel thanks for the thorough follow up, i tried to reproduce this with the following dockerfile:

FROM php:7.1.31-fpm-stretch

RUN apt-get update && \
    apt-get install -y git

RUN git clone --depth 1 --branch v1.1.0 https://github.com/edenhill/librdkafka.git \
    && cd librdkafka \
    && ./configure \
    && make \
    && make install

RUN git clone --depth 1 --branch 3.1.2 https://github.com/arnaud-lb/php-rdkafka.git \
    && cd php-rdkafka \
    && phpize \
    && ./configure \
    && make all -j 5 \
    && make install

RUN docker-php-ext-enable rdkafka

Where i try to do the same you did. Somehow this succeeds. I will try to dig deeper and find out why you are getting undefined symbol: zend_ce_countable.

[nick-zh]

In your initial comment i saw you have mentioned this:
PHP version: 7.1.31, rdkafka compiled against 7.1.15
In your second try with librdkafka:1.1.0 and rdkafka.3.1.2 did you compile everything again with the same php version (meaning php:7.1.31)?

[geruetzel]

@nick-zh I was able to even make it a little simpler, tried to load the module on the host it was built on in php-cli:

root@buildhost:~ # php7.1 -v
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20160303/rdkafka.so' - /usr/lib/php/20160303/rdkafka.so: undefined symbol: zend_ce_countable in Unknown on line 0
PHP 7.1.15-1+0~20180306120119.15+jessie~1.gbp78327e (cli) (built: Mar  6 2018 12:41:50) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.1.15-1+0~20180306120119.15+jessie~1.gbp78327e, Copyright (c) 1999-2018, by Zend Technologies

so the module fails to load even if the version it was compiled against is the same it is run on. the local php-cli version is from sury: https://packages.sury.org/php/

I will try to upgrade to a php7.1 version that is made for debian stretch (the one I am using now is for jessie) and will let you know how it went

[nick-zh]

@geruetzel ok great, thx for the info, so i see now it is mentioned that this php build is for jessie, correct? So to reproduce i should try to use debian jessie and this php from sury?
Just making sure, because you mentioned stretch before

[nick-zh]

Tried to reproduce with debian:stretch, php:7,1 from sury

FROM debian:oldstable

# add sury repo for php 7.1
RUN apt-get update && \
    apt-get install -y apt-transport-https lsb-release ca-certificates wget gcc && \
    wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg && \
    echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list

RUN apt-get update && \
    apt-get install -y git php7.1 php7.1-dev

RUN git clone --depth 1 --branch v1.1.0 https://github.com/edenhill/librdkafka.git \
    && cd librdkafka \
    && ./configure \
    && make \
    && make install

RUN git clone --depth 1 --branch 3.1.2 https://github.com/arnaud-lb/php-rdkafka.git \
    && cd php-rdkafka \
    && phpize \
    && ./configure \
    && make all -j 5 \
    && make install \
    && echo "extension=rdkafka.so" > /etc/php/7.1/cli/conf.d/20-rdkafka.ini

CMD tail -f /dev/null

Everything went fine, my guess is, you have somewhere either versions or packages from distro not alligned. Let me know if you found something, that helps me narrow it down @geruetzel, thx!

EDIT: Also tried with jessie, by using FROM debian:oldoldstable was still working fine

[geruetzel]

@nick-zh thanks for your support. I did not think of building this in a docker container, which is of course an easy and clean way to do it. I was able to extract the compiled extension from the image created by your Dockerfile and it ran without errors on my php install. Thanks for your patience despite the fact, that you software is not the cause of my problem :)

[Steveb-p]

That's @nick-zh for you ;)

:+1: