Cryptopp: Cross Compile for ARM Failure

Created on 16 Jan 2018 · 15Comments · Source: weidai11/cryptopp

State the operating system and version (Ubutnu 17 x86_64, Windows 7 Professional x64, etc)
State the version of the Crypto++ library (Crypto++ 5.6.5, Master, etc)
State how you built the library (Makefile, Cmake, distro, etc)
Show a typical command line (the output of the compiler for cryptlib.cpp)
Show the link command (the output of the linker for libcryptopp.so or cryptest.exe)
Show the exact error message you are receiving (copy and paste it); or
Clearly state the undesired behavior (and state the expected behavior)

Building on Ubuntu 16.10
Crytpopp 5.6.5
Build using directions at ARM Embedded (Command Line). In the setenv-embedded.sh script, I changed version of gnueabi to 6. Docs use 4.7.3
File successfully made into object:

/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -Wl,--fix-cortex-a8 -I/usr/arm-linux-gnueabi/include/c++/6 -I/usr/arm-linux-gnueabi/include/c++/6/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c fips140.cpp

Doesn't reach linking
Error:

/usr/lib/gcc-cross/arm-linux-gnueabi/6/include/arm_neon.h:12253:1: error: inlining failed in call to always_inline ‘uint64x2_t veorq_u64(uint64x2_t, uint64x2_t)’: target specific option mismatch
 veorq_u64 (uint64x2_t __a, uint64x2_t __b)
 ^~~~~~~~~
gcm-simd.cpp:285:32: note: called from here
     *(uint64x2_t*)a = veorq_u64(*(uint64x2_t*)b, *(uint64x2_t*)c);
GNUmakefile-cross:461: recipe for target 'gcm-simd.o' failed

The build process fails for cross-compilation

Bug arm

Source

jackfrye

All 15 comments

Thanks @jackfrye,

Yeah, this is kind of a known problem with 5.6.5. You have a few choices.

You can move to Master, which is the upcoming Crypto++ 6.0. It is stable so you will be OK with it. I believe the cross-compiles are working as expected in Master (ping @anonimal).
You can use -mfpu=vfpv3-d16 but you will need to add CRYPTOPP_DISABLE_ASM to your CXXFLAGS.
You can add -mfpu=neon and compile as expected

The difference between (2) and (3) is the presence of a NEON coprocessor. You likely have one, but you may be missing it on your hardware.

noloader on 16 Jan 2018

I may have provided the wrong version of the software. I cloned it today, so I am assuming my local branch was synced with latest master, which apparently you are saying it 6.0 not 5.6.5. I'm looking at compiling for a Xilinx Zynq 7000 Cortex A9, so I will probably need that NEON, or at least it should not do any harm. I will ask @anonimal for further guidance if needed. thanks.

jackfrye on 16 Jan 2018

@anonimal are the cross compilers working?

it looks like I can compile all source files to object when I do:

uncomment
export ARM_EMBEDDED_FLAGS="-march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -Wl,--fix-cortex-a8 -I$ARM_EMBEDDED_CXX_HEADERS -I$ARM_EMBEDDED_CXX_HEADERS/arm-linux-gnueabi"
in setenv-embedded.sh

and

CXXFLAGS ?= -DNDEBUG -g2 -O3 -fPIC -pipe -DCRYPTOPP_DISABLE_ASM
in the GNUMakefile-cross

However, when it tries to generate the executable, it fails because it cannot recognize the format of the linker (I am guessing built for wrong architecture)

g++ -o cryptest.exe -DNDEBUG -g2 -O3 -fPIC -pipe -DCRYPTOPP_DISABLE_ASM adhoc.o test.o bench1.o bench2.o validat0.o validat1.o validat2.o validat3.o datatest.o regtest1.o regtest2.o regtest3.o fipsalgt.o dlltest.o ./libcryptopp.a  
/usr/bin/ld: ./libcryptopp.a(cryptlib.o): Relocations in generic ELF (EM: 40)
/usr/bin/ld: ./libcryptopp.a(cryptlib.o): Relocations in generic ELF (EM: 40)
...
/usr/bin/ld: ./libcryptopp.a(cryptlib.o): Relocations in generic ELF (EM: 40)
/usr/bin/ld: ./libcryptopp.a(cryptlib.o): Relocations in generic ELF (EM: 40)
./libcryptopp.a: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
GNUmakefile-cross:415: recipe for target 'cryptest.exe' failed
make: *** [cryptest.exe] Error 1

When I comment out that ARM_EMBEDDED_FLAGS variable in setenv-embedded as above in the environment variables so that it is set to default

export ARM_EMBEDDED_FLAGS="-mcpu=cortex-m3 -I$ARM_EMBEDDED_CXX_HEADERS -I$ARM_EMBEDDED_CXX_HEADERS/arm-linux-gnueabi"

and I do not add that -DCRYPTOPP_DISABLE_ASM to the GNUMakefile-cross, I get these errors.

```g++ -DNDEBUG -g2 -O3 -fPIC -pipe -c aria-simd.cpp
In file included from aria-simd.cpp:22:0:
/usr/lib/gcc/x86_64-linux-gnu/6/include/tmmintrin.h: In function ‘void CryptoPP::ARIA_ProcessAndXorBlock_Xor_SSSE3(const byte, CryptoPP::byte, const byte, CryptoPP::word32)’:
/usr/lib/gcc/x86_64-linux-gnu/6/include/tmmintrin.h:136:1: error: inlining failed in call to always_inline ‘__m128i _mm_shuffle_epi8(__m128i, __m128i)’: target specific option mismatch
_mm_shuffle_epi8 (__m128i __X, __m128i __Y)
^~~~
aria-simd.cpp:135:18: note: called from here

~~~~^~~~~~

~~~~~~~~~~~~
_mm_shuffle_epi8(_mm_load_si128(CONST_M128_CAST(rk)), MASK)));
~~~~~~~~~~~~~
In file included from aria-simd.cpp:22:0:
/usr/lib/gcc/x86_64-linux-gnu/6/include/tmmintrin.h:136:1: error: inlining failed in call to always_inline ‘__m128i _mm_shuffle_epi8(__m128i, __m128i)’: target specific option mismatch
_mm_shuffle_epi8 (__m128i __X, __m128i __Y)
^~~~~~
aria-simd.cpp:135:18: note: called from here

~~~~^~~~~~

~~~~~~~~~~~~
_mm_shuffle_epi8(_mm_load_si128(CONST_M128_CAST(rk)), MASK)));
~~~~~~~~~~~~~~~
GNUmakefile-cross:445: recipe for target 'aria-simd.o' failed
make: * [aria-simd.o] Error 1
```

jackfrye on 16 Jan 2018

@jackfrye, @anonimal,

Yeah, it looks like you are using 5.6.5 (and not Master).

The ARIA/ARM bug was reported at Issue 491, Android compiling doesn't seem to work with NDK r15c. But I don't see a reference to the commit that fixed things in config.h, aria.cpp or aria-simd.cpp. This may be the commit we want: f0c2324f6be1a3b9.

Ubuntu arm cross-compile toolchains are a can of worms. I have had a lot trouble with them. Its one of the reasons I gave up trying to cross compile. I now SSH into the board and endure the longer compile times to avoid the toolchain troubles.

noloader on 16 Jan 2018

When I try to pull from master, it says I am up to date. When I try to pull from Master, it says the branch does not exist

native compiling is exactly what Xilinx tells me not to do, and at times like this I really begin to wonder why.

jackfrye on 16 Jan 2018

@jackfrye,

Working from Master I cannot duplicate the issue after updating setenv-embedded.sh at commit c8389f89ea0e using stock settings:

$ make -f GNUmakefile-cross 
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c cryptlib.cpp
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c cpu.cpp
...

But we are back to the broken Ubuntu toolchain. It fails to link things:

/usr/bin/arm-linux-gnueabi-g++ -o cryptest.exe -DNDEBUG -g2 -O3 -fPIC -pipe -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi adhoc.o test.o bench1.o bench2.o validat0.o validat1.o validat2.o validat3.o datatest.o regtest1.o regtest2.o regtest3.o fipsalgt.o dlltest.o ./libcryptopp.a  
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/libc.so.6 inside /usr/arm-linux-gnueabi
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/libc_nonshared.a inside /usr/arm-linux-gnueabi
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/ld-linux.so.3 inside /usr/arm-linux-gnueabi
collect2: error: ld returned 1 exit status
GNUmakefile-cross:415: recipe for target 'cryptest.exe' failed
make: *** [cryptest.exe] Error 1

In fact, the following is broke, too:

$ cat test.cxx
#include <iostream>

int main(int argc, char* argv[])
{
    std::cout << "Hello world" << std::endl;
    return 0;
}

It results in:

$ /usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi test.cxx -o test.exe
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/libc.so.6 inside /usr/arm-linux-gnueabi
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/libc_nonshared.a inside /usr/arm-linux-gnueabi
/usr/lib/gcc-cross/arm-linux-gnueabi/5/../../../../arm-linux-gnueabi/bin/ld: cannot find /usr/arm-linux-gnueabi/lib/ld-linux.so.3 inside /usr/arm-linux-gnueabi
collect2: error: ld returned 1 exit status

There's not much that can be done with a broken toolchain.

I seem to recall I filed a couple of bug reports with Ubuntu about it a few years ago. I tried to file a new bug, but Ubuntu's broken shit extends beyond it's toolchains. I gave up after about 6 of these:

ubuntu-broken-shit

noloader on 17 Jan 2018

@jackfrye,

By the way, this looks wrong for GNU systems:

-march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16

As far as I know, all GNU systems use hard floats, not soft ones. Hard floats pass floats in registers. Soft floats pass them on the stack. As I understand things, they are not compatible.

Working from Master does not experience the problem. You should probably switch to Master. Also see Github and Git on the Crypto++ wiki.

$ make -f GNUmakefile-cross 
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c cryptlib.cpp
...
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c arc4.cpp
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c aria-simd.cpp
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c aria.cpp
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c ariatab.cpp
...

But it looks like you will need to add -DCRYPTOPP_DISABLE_ASM for other source files, like gcm-simd.cpp:

$ make -f GNUmakefile-cross 
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -DCRYPTOPP_DISABLE_ASM -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c cryptlib.cpp
/usr/bin/arm-linux-gnueabi-g++ -DNDEBUG -g2 -O3 -fPIC -pipe -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -DCRYPTOPP_DISABLE_ASM -I/usr/arm-linux-gnueabi/include/c++/5.4.0 -I/usr/arm-linux-gnueabi/include/c++/5.4.0/arm-linux-gnueabi --sysroot=/usr/arm-linux-gnueabi -c cpu.cpp
...

noloader on 17 Jan 2018

It looks like there is only one branch "master". I have tried to pull from "origin Master" (capital M) but it did not work.

jackfrye on 17 Jan 2018

native compiling is exactly what Xilinx tells me not to do, and at times like this I really begin to wonder why.

I looked at the company's site. It looks like they ship a lot of different gadgets, including some PLCs. Some of their boards probably don't have the resources to run a SSH server, let alone a C++ compiler.

Its too bad Fedora does not provide an ARM cross-compiler. I wonder if CentOS supplies one. (Fedora offers an ARM cross-compiler, but it is for building ARM kernels, not userland).

It looks like there is only one branch "master". I have tried to pull from "origin Master" (capital M) but it did not work.

No idea. Git is a mystery to me. I stand in awe at how simple workflows are made difficult to impossible.

noloader on 17 Jan 2018

😄1

@jackfrye,

If you are working from a Fork (as opposed to a Clone), then this might help. This is how my personal fork of Wei's Crypto++ is setup. I use it for testing bigger changes before merging back to Wei's master.

cryptopp-fork$ cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://github.com/noloader/cryptopp
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[remote "upstream"]
        url = https://github.com/weidai11/cryptopp
        fetch = +refs/heads/*:refs/remotes/upstream/*

To update the Crypto++ code my fork is derived from, I issue:

cryptopp-fork$ git pull upstream master
Enter passphrase for key '/home/jwalton/.ssh/id_ed25519':
From ssh://github.com/weidai11/cryptopp
 * branch            master     -> FETCH_HEAD
Already up-to-date.

And if you want to reset your fork back to the same state as Wei's Master, then you can run reset-fork.sh. Its a helper script we keep online, and it is available here.

cp TestScripts/reset-fork.sh .
./reset-fork.sh

noloader on 17 Jan 2018

Having spoken to the Xilinx representative, it looks like you guys have a bitbake recipe specifically for the Linux I am trying to build on ARM. I am going to try that route. I will let you know how it goes.

jackfrye on 17 Jan 2018

@jackfrye,

Ack, thanks. I am going to close this out.

I don't have Yocto accounts and test gear, so I can't directly test for you/with you. But I'm happy to assist, especially when the work affects a distro like Yocto . Ping me at noloader, gmail account.

noloader on 17 Jan 2018

Sorry folks, I'm just now getting the pings (my github notifications are flooded on a daily basis from other repos).

We (kovri) doesn't cross-compile ARM; we build natively. Details at https://build.getmonero.org/waterfall/.

anonimal on 23 Jan 2018

👍1

@anonimal, Yeah, I know what you mean. Cross-compiles are a pain.

@jackfrye, We just released Crypto++ 6.0. It has several build system fixes that may be of interest to you. @anonimal, @Skycoder42, me and several others spent a fair amount of time cleaning up cross-compiles after the BASE+SIMD change.

noloader on 23 Jan 2018