Rocksdb: Which glibc should we support in binary releases?

[adamretter]

At the moment the RocksJava releases for Linux are built on CentOS 5.6 which provides glibc 2.5 and libstdc++ 4.1.

Obviously that version of glibc and libstdc++ is quite outdated now; However I am trying to understand forwards and backwards compatibility between glibc versions.

Using a librocksdbjni-linux64.so file built on CentOS 5.6 (glibc 2.5) on CentOS 7.2 (glibc 2.17) seems to work fine.

However a version of librocksdbjni-linux64.so built on TinyCore Linux 7.2 (glibc 2.22) will not work on CentOS 7.2 and instead results in the error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni8724249777028282150.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /tmp/librocksdbjni8724249777028282150.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1968)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1893)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
    at java.lang.Runtime.load0(Runtime.java:795)
    at java.lang.System.load(System.java:1062)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

Running ldd /tmp/librocksdbjni8724249777028282150.so shows:

./librocksdbjni-linux64.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./librocksdbjni-linux64.so)
./librocksdbjni-linux64.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./librocksdbjni-linux64.so)
./librocksdbjni-linux64.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./librocksdbjni-linux64.so)
    linux-vdso.so.1 =>  (0x00007ffeb2cd0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff06229e000)
    librt.so.1 => /lib64/librt.so.1 (0x00007ff062096000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff061d8d000)
    libm.so.6 => /lib64/libm.so.6 (0x00007ff061a8b000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff061875000)
    libc.so.6 => /lib64/libc.so.6 (0x00007ff0614b2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff062b90000)

Can someone with a better understanding of glibc linking issues shed some light on this? I am wondering ultimately if we need different builds for different Linux distributions depending on what glibc versions they provide?

[JonChesterfield]

I can think of four answers to distribution of glibc based binaries.

1/ Link to whatever your CI server is using and hope for the best

2/ Link against many glibc versions and offer as downloads sorted by OS version. This works, but is a serious inconvenience for all involved

3/ Link against as old a glibc as we can get away with, usually from an old centos distribution. This will work (modulo bugs) on anything newer thanks to symbol versioning. Occasionally it may be worth explicitly using an older version of e.g. memcpy just to use an older glibc. The cost is we lose out on all the good work done by the glibc team on newer versions

4/ Detect the available version of glibc at runtime and use that. This appeals from an engineering perspective but is quite a lot of work to get right. In particular there's a nasty bootstrapping problem to gain access to dlopen in order to use dlopen to access glibc, so it's likely to be implemented as option 3 + runtime lookup for newer versions

Personal preference is for option 3, which I believe rocksdb currently uses. So I think we're good as we are.

[zedware]

Option 3 is a good choice.

[adamretter]

I also note that glibc 2.6 is now required to be able to use sync_file_range syscall which is needed by ROCKSDB_RANGESYNC_PRESENT. glibc 2.5 on CentOS 5 has an unsupported/broken sync_file_range. See https://bugzilla.redhat.com/show_bug.cgi?id=518581

As such our next best bet is likely CentOS 6 which provides glibc 2.12.

[sagar0]

Seems like Redhat discourages static linking, by saying (under '2.1.1. Static Linking' section):

Static linking is emphatically discouraged for all Red Hat Enterprise Linux releases. Static linking causes far more problems than it solves, and should be avoided at all costs.

The main drawback of static linking is that it is only guaranteed to work on the system on which it was built, and even then only until the next release of glibc or libstdc++ (in the case of C++). There is no forward or backward compatibility with a static build. Furthermore, any security fixes (or general-purpose fixes) in subsequent updates to the libraries will not be available unless the affected statically linked executables are re-linked.

[adamretter]

@sagar0 That may be the case but I think it has worked well for us so far. Linking against an older version of glibc has worked also when running against more modern versions, and it enables us to publish a Jar file which contains everything that a Java user needs (without them having to get into C++ and compiling and installing dependencies (e.g. snappy, zstd etc)

[gfosco]

Closing this via automation due to lack of activity. If discussion is still needed here, please re-open or create a new/updated issue.