Cryptopp: OS X compile failures with Clang from Xcode 9.3 for 32-bit

[alanbirtles]

c++ -c sha.cpp -arch i386 is failing to build with the following error:

sha.cpp:1005:5: error: invalid operand for instruction
    ASJ(    jnz,     0, b)
    ^
./cpu.h:623:23: note: expanded from macro 'ASJ'
        #define ASJ(x, y, z) GNU_ASJ(x, y, z)
                             ^
./cpu.h:617:27: note: expanded from macro 'GNU_ASJ'
        #define GNU_ASJ(x, y, z) #x " " #y #z ";" NEW_LINE
                                 ^
<scratch space>:105:2: note: expanded from here
"jnz"
 ^
<inline asm>:80:1: note: instantiated into assembly here
jnz 0b;
^

64-bit builds work. The clang version is:
Apple LLVM version 9.1.0 (clang-902.0.39.1)
vmac.cpp fails with the same error.
I presume this is a bug with clang?

[noloader]

Thanks @alanbirtles,

@mouse07410 is experiencing a similar problem. I don't have access Xcode 9.3 so I have not been able to duplicate it.

I suspect it has something to do with the integrated assembler and whitespace. Also see cpu.h : 40:

// Applies to both X86/X32/X64 and ARM32/ARM64
#if defined(CRYPTOPP_LLVM_CLANG_VERSION) || defined(CRYPTOPP_APPLE_CLANG_VERSION) || defined(CRYPTOPP_CLANG_INTEGRATED_ASSEMBLER)
    #define NEW_LINE "\n"
    #define INTEL_PREFIX ".intel_syntax;"
    #define INTEL_NOPREFIX ".intel_syntax;"
    #define ATT_PREFIX ".att_syntax;"
    #define ATT_NOPREFIX ".att_syntax;"
#elif defined(__GNUC__)
    #define NEW_LINE
    #define INTEL_PREFIX ".intel_syntax prefix;"
    #define INTEL_NOPREFIX ".intel_syntax noprefix;"
    #define ATT_PREFIX ".att_syntax prefix;"
    #define ATT_NOPREFIX ".att_syntax noprefix;"
#else
    #define NEW_LINE
    #define INTEL_PREFIX
    #define INTEL_NOPREFIX
    #define ATT_PREFIX
    #define ATT_NOPREFIX
#endif

As a workaround you might try removing the first and/or second #if block entirely.

[alanbirtles]

I've narrowed it down to:

void test()
{
    __asm__ __volatile__
    (
    ".intel_syntax;\n"
    "0:\n"
    "jnz 0b;\n"
    );
}

which fails on standard clang 5 and 6: https://godbolt.org/g/ikFCrk

[alanbirtles]

Looks like https://bugs.llvm.org/show_bug.cgi?id=36144

[noloader]

@alanbirtles, @mouse07410,

I believe Uri was having trouble with this:

 "1:\n"
 "jnz 1b;\n"

This is from a sidebar email where we were discussing things offline:

g++ -maes -mpclmul -mrdrnd -msse2 -mssse3 -msse4.2 -mtune=native -Os -Ofast -std=c++11 -fPIC -Wa,-q -DCRYPTOPP_CLANG_INTEGRATED_ASSEMBLER=1 -pthread -pipe -msse4.1 -maes -c rijndael-simd.cpp
rdrand.cpp:192:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:192:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:349:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:349:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:192:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:192:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:349:1: error: invalid operand for instruction
jnc 1b;
^
rdrand.cpp:349:1: error: invalid operand for instruction
jnc 1b;
^
make: *** [rdrand.o] Error 1

[Joel-Mckay]

MacOSX Xcode symbolically links the g++ paths to clang, and even compiling the real gcc from brew will not help some issues.... What we did find was:

cryptopp/rdrand.cpp

• clang and real g++ both will throw ambiguous definition error for ByteSwap() ...common on MacOS, but usually can suppress with type casting.

cryptopp/scrypt.cpp

• dumps several screens of assembly with g++- compiler even if CRYPTOPP_DISABLE_ASM flag is defined

@alanbirtles, @mouse07410,
You may want to try a build with CRYPTOPP_DISABLE_ASM flag set.

[alanbirtles]

@Joel-Mckay with the patch I posted crypto++ builds perfectly, no need to disable assembly

[mouse07410]

First, no it does not build perfectly - unless you use Apple GCC, in which case I'd ask why bother to begin with, as Apple Clang IMHO is far superior to Apple GCC. And if you do use a "real" (aka current) GCC - it still does not compile properly (see https://github.com/weidai11/cryptopp/pull/640#issuecomment-380923866).

And yes, it seems to be a bug in Clang.

However, the whole point is to build the library with assembler speedups enabled.

[alanbirtles]

What doesn't work in clang? My build definitely compiled and was working in our application.

[noloader]

@alanbirtles,

What doesn't work in clang?

Answering this narrowly, I believe it is the integrated assembler. The compiler parses things OK and generates the intermediate code OK. But Clang's integrated assembler cannot consume the intermediate representation of the code.

At least that has been mine and Uri's experience with it in the past.

Another thing to watch out for is, LLVM and Apple use different numbering schemes for Clang. That is why we have LLVM_CLANG_VERSION and APPLE_CLANG_VERSION; and were are constantly tuning work arounds. Once things are straightened out with Apple Clang, we will have to approximate where it needs to be applied for LLVM Clang. Also see Yamaya | Xcode clang version record.

[noloader]

@Joel-Mckay,

You may want to try a build with CRYPTOPP_DISABLE_ASM flag set.

This is one of those last resort remediations. CRYPTOPP_DISABLE_ASM used to be required for OS X , but Uri and I removed the limitations some time ago (circa Crypto++ 5.6.3). You should avoid it nowadays. My apologies you have been using it.

In the future, if things don't "just work" for you, then ping me or Uri directly. Our goal for distros is "things just work". My email is noloader, Gmail; and Uri's email is mouse008, Gmail.

[Joel-Mckay]

Thanks @noloader, and I did try the repo head build again with xcode 9.3 clang without error (the master zip snapshot still reads as corrupted with the MacOS keka extractor).

./cryptest.exe v   
... all tests pass on unit test

I do like permissive atomic c/c++ lib solutions as they can be audited and ported more easily.
Great work guys. ;-)

[noloader]

Cleared at Commit a0f91aeb2587.

[epicstar]

I'm experiencing this problem with Cryptopp 5.6.5 (Android x86 only). The patch has also fixed the issue.