Rocksdb: Build error on powerpc64 with Clang 11.0.0

[pkubaj]

Expected behavior

rocksdb doesn't build.

Actual behavior

It should build.

Steps to reproduce the behavior

Build rocksdb using Clang 11.0.0 on powerpc64 - I'm using FreeBSD head, but building on Linux should also yield the same results.

Building fails with:

util/crc32c.cc:449:54: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'unsigned int' [-Werror,-Wshorten-64-to-32]
  return crc32c_ppc(crc, (const unsigned char *)buf, size);
         ~~~~~~~~~~                                  ^~~~

[jay-zhuang]

I guess we should change unsigned len to unsigned long len here: https://github.com/facebook/rocksdb/blob/e731f4402258554812c46334dc0d9483e6cc769b/util/crc32c_ppc.c#L70
And
https://github.com/facebook/rocksdb/blob/e731f4402258554812c46334dc0d9483e6cc769b/util/crc32c_ppc.c#L29-L30
@pkubaj we don't have ppc environment with clang11 for test, would you please try that out?

[adamretter]

At the moment our ppc64le Jenkins CI and Travis CI systems both use gcc 5.4.0-6ubuntu1~16.04.12 5.4.0 20160609.

I could attempt to setup a ppc64le CLANG 11 environment on the Jenkins CI, assuming that I can get and install a Clang 11 for Ubuntu 16 on ppc64le.

[adamretter]

It looks like Clang 11 only became available in the Ubuntu package repo in Ubuntu 20.10 (groovy).
However, it looks like the LLVM project provide pre-built binaries. I will see what I can do...

[adamretter]

The ppc64le Clang 11 binaries are available from here: https://releases.llvm.org/download.html#11.0.0

The exact download is: https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/clang+llvm-11.0.0-powerpc64le-linux-ubuntu-16.04.tar.xz

Unfortunately whilst it claims to be usable for Ubuntu 16.04, it appears the binaries have been compiled against GLIBC 2.27, yet Ubuntu 16.04 has GLIBC 2.23. It is Ubuntu 18.04 that ships with GLIBC 2.27.

I have reported the issue to the LLVM people - https://bugs.llvm.org/show_bug.cgi?id=48219

[adamretter]

@pkubaj @jay-zhuang I was able to compile LLVM11 from source code on Ubuntu 16.04. I was also able to reproduce the same error:

  CC       util/random.o
util/crc32c.cc:453:54: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'unsigned int' [-Werror,-Wshorten-64-to-32]
  return crc32c_ppc(crc, (const unsigned char *)buf, size);
         ~~~~~~~~~~                                  ^~~~
1 error generated.
Makefile:2349: recipe for target 'util/crc32c.o' failed
make: *** [util/crc32c.o] Error 1

I will add an LLVM11 target to the ppc64le CI and test your proposed patch...

[adamretter]

Okay I had to start again with LLVM 11 on pcc64le as the instructions I had followed to build it resulted in a debug build which was too slow to be useful. I have now built and bootstrapped llvm from source. I also setup a build Matrix in the CI.

[adamretter]

@jay-zhuang Unfortunately your proposed fix doesn't seem to be enough and reveals other errors. I will see if I can get to the bottom of it...

[adamretter]

Okay so with the attached patch ppc-clang11.patch.txt, I am now hitting a different error related to gcc vs clang and asm headers on ppc:

util/crc32c_ppc_asm.S:9:10: fatal error: 'ppc-asm.h' file not found
#include <ppc-asm.h>
         ^~~~~~~~~~~
1 error generated.
Makefile:2346: recipe for target 'util/crc32c_ppc_asm.o' failed
make: *** [util/crc32c_ppc_asm.o] Error 1
make: *** Waiting for unfinished jobs....

It seems that they already solved this in MariaDB - https://bugs.llvm.org/show_bug.cgi?id=28286... So I will take a look at how they fixed that there (https://github.com/antonblanchard/crc32-vpmsum/commit/ffc8018efc1e4f05d22a9fc8dde57109dd09368b) and see if we can do something similar.

It seems we would need to include a file from GCC... I will send a PR, but I guess the license implications of this will have to be discussed.

[pkubaj]

Thanks!