Libelektra: Ubuntu Focal (20.04) Python Bindings

[markus2330]

It looks like Ubuntu Focal (20.04) has incompatible SWIG/Python versions according to:

https://github.com/ElektraInitiative/libelektra/blob/cbb0909f6ed189d1a04872818ae48556f3d44d8d/src/bindings/swig/python/CMakeLists.txt#L28-L30

which gets exactly triggered by:

https://packages.ubuntu.com/focal/swig 4.0.1-5build1
https://packages.ubuntu.com/focal/python3 3.8.2-0ubuntu2

@manuelm do you remember which incompatibility is addressed there?

The first question is, if our check is correct. Maybe 4.0.1-5build1 already contains a patch that fixes this incompatibility.

If they really are incompatible, then we could try to upgrade swig, which hopefully still will lead to functioning Ubuntu Focal (20.04) packages, as swig should not show up as run-time dependency.

@robaerd Can you already build Ubuntu Focal (20.04) packages for the upcoming release? (At the moment we also do not even have a Docker image for 20.04, so it is probably quite a bit of work.) There were already several requests to have it, e.g. #3412

@mpranj It would be great if this issue can be fixed for 0.9.3. (If the swig upgrade is necessary and works, nothing is to be changed in the source of Elektra.)

@dev2718 Thank you for reporting this problem!

[mpranj]

The original issue regarding this incompatibility is #3379. It was resolved by adding the version check.

If ubuntu backported the fix, which I do not think, there is no guarantee that other distris did the same.

[manuelm]

If ubuntu backported the fix, which I do not think, [...]

They didn't backport the commit.

See https://bugs.launchpad.net/ubuntu/+source/swig/+bug/1876764

[manuelm]

While further looking: tp_print is marked deprecated. So the compilation should still succeed and the generated bindings should work fine. Do you guys treat __attribute__((__deprecated__)) as a failure? If so this should be fixed and #3379 reverted.

[markus2330]

Thank you so much for looking into this!

If -Werror is passed we treat most warnings as errors in master, in #3447 f90ec817965b86402743583c0f7af9cd31338707 @kodebach changed it, as again plenty of new warnings came up. We should probably get this commit to master for less troubles in general.

While further looking: tp_print is marked deprecated. So the compilation should still succeed and the generated bindings should work fine.

So we can make this check less strict? Are we talking about this tp_print https://github.com/cython/cython/issues/2976? Which python version removed tp_print?

[manuelm]

Add -Wno-error=deprecated-declarations if -Werror is enabled. Treating deprecations as fatal errors seems wrong.

Which python version removed tp_print?

3.9 will remove tp_print. See https://github.com/python/cpython/commit/d917cfe4051d45b2b755c726c096ecfcc4869ceb

[markus2330]

So it will not help for Ubuntu Focal as 3.8 still has tp_print?

But this means that we can make the check less strict (allow 4.0.1 with >=3.9).

Did someone try SWIG 4.0.2 on Ubuntu Focal?

[mpranj]

Afair the error was related to tp_vectorcall. Open a PR where the check is removed and see where it fails.

[robaerd]

@robaerd Can you already build Ubuntu Focal (20.04) packages for the upcoming release? (At the moment we also do not even have a Docker image for 20.04, so it is probably quite a bit of work.)

Yes, I will also create the docker image for 20.04.

[markus2330]

Any update here? Did someone manage to build the Ubuntu Focal (20.04) packages with Python Bindings?

[robaerd]

The python bindings are building on Ubuntu Focal with SWIG 4.0.2.

For building of the deb packages I changed the swig dependencies in debian/control and debian/rules to point to the new swig binary, but the the swig 4.0.2 package couldn't be detected by dpkg-checkbuilddeps.

...
dpkg-checkbuilddeps: error: Unmet build dependencies: swig
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting

Removing swig from the Build-Depends section in debian/control temporarily solved this issue and the dep package could be built.

[markus2330]

debian/control and debian/rules to point to the new swig binary

What exactly is the change? Can you create a PR or point me to the repo?

package couldn't be detected by dpkg-checkbuilddeps.

Please run cmake in a shell to not confuse dpkg and cmake errors. The error shown above is a dpkg error: Simply remove the swig dependency from debian/control which is not fulfilled as you probably did not install swig via the package manager (as 4.0.2 is not available).

Is there another error earlier (from cmake)? Does CMakeFiles/CMakeError.log say something about why the swig executable is unsuitable? (It is possible cmake did not even run as dpkg found a dependency problem earlier.)

[robaerd]

What exactly is the change? Can you create a PR or point me to the repo?

I will soon add the changes to the pipeline and the new Ubuntu Focal Dockerfile to my last PR.
The installation of swig 4.0.2 creates following executable /usr/bin/swig. So the flag -DSWIG_EXECUTABLE would need to be either changed to ' /usr/bin/swig', or a symlink /usr/bin/swig3.0 created which points to /usr/bin/swig. I think latter would be the better approach, since we wouldn't need to maintain two versions (debian buster would still use swig3.0) or update the debian image to use swig4.0.2.

Simply remove the swig dependency from debian/control which is not fulfilled as you probably did not install swig via the package manager (as 4.0.2 is not available).

Will do.

Please run cmake in a shell to not confuse dpkg and cmake errors.

I did. I just wanted to mention that building of the package would require changes to debian/control and debian/rules

Is there another error earlier (from cmake)?

Yes but i don't think it has something to do with swig. All tests pass as well.

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_1edfc/fast && /usr/bin/make -f CMakeFiles/cmTC_1edfc.dir/build.make CMakeFiles/cmTC_1edfc.dir/build
make[1]: Entering directory '/home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_1edfc.dir/src.c.o
/usr/bin/cc   -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal -DCMAKE_HAVE_LIBC_PTHREAD   -o CMakeFiles/cmTC_1edfc.dir/src.c.o   -c /home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_1edfc
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_1edfc.dir/link.txt --verbose=1
/usr/bin/cc  -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal -DCMAKE_HAVE_LIBC_PTHREAD    CMakeFiles/cmTC_1edfc.dir/src.c.o  -o cmTC_1edfc 
/usr/bin/ld: CMakeFiles/cmTC_1edfc.dir/src.c.o: in function `main':
src.c:(.text+0x46): undefined reference to `pthread_create'
/usr/bin/ld: src.c:(.text+0x52): undefined reference to `pthread_detach'
/usr/bin/ld: src.c:(.text+0x63): undefined reference to `pthread_join'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_1edfc.dir/build.make:87: cmTC_1edfc] Error 1
make[1]: Leaving directory '/home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_1edfc/fast] Error 2


Source file was:
#include <pthread.h>

void* test_func(void* data)
{
  return data;
}

int main(void)
{
  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_d2c29/fast && /usr/bin/make -f CMakeFiles/cmTC_d2c29.dir/build.make CMakeFiles/cmTC_d2c29.dir/build
make[1]: Entering directory '/home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_d2c29.dir/CheckFunctionExists.c.o
/usr/bin/cc   -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal -DCHECK_FUNCTION_EXISTS=pthread_create   -o CMakeFiles/cmTC_d2c29.dir/CheckFunctionExists.c.o   -c /usr/share/cmake-3.16/Modules/CheckFunctionExists.c
Linking C executable cmTC_d2c29
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_d2c29.dir/link.txt --verbose=1
/usr/bin/cc  -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_d2c29.dir/CheckFunctionExists.c.o  -o cmTC_d2c29  -lpthreads 
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_d2c29.dir/build.make:87: cmTC_d2c29] Error 1
make[1]: Leaving directory '/home/jenkins/DEV-elektra/libelektra/build/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_d2c29/fast] Error 2

[markus2330]

I did. I just wanted to mention that building of the package would require changes to debian/control and debian/rules

Yes, but probably these changes will stay compatible so that also Debian Buster can build with it. (As you already noticed about the swig binary. And removing a dep will not break the Debian builds.) Long term we probably need a better solution as it will probably not always be possible to cover all Debian derivatives from a single source.

Yes but i don't think it has something to do with swig. All tests pass as well.

Some tests in CMakeFiles/CMakeError.log are supposed to fail, e.g. when CMake checks some linker flags a test might fail but CMake knows about an alternative, as in this case: CMAKE_HAVE_LIBC_PTHREAD checks if libc has pthread but on this system, pthread is a separate library. So no need to worry here, nothing seems to be related to swig.

[markus2330]

I now tested https://build.libelektra.org/job/libelektra-release/70/artifact/artifacts/ubuntu-focal/ubuntu-focal-release.tar.gz
where the python bindings worked without problems under Ubuntu Focal.