Lightning: Cross compilation support for ARM Linux

[hsjoberg]

I've been trying to get cross compilation to work for Linux.
The dependencies (sqlite3 and libgmp) compiles fine using the arm-linux-gnueabi toolchain.

My issue so far has been to understand how CCAN could work together with cross compilation.
A binary ("configurator") seems to be compiled and later executed in the Makefile, which causes conflicts if you would naively just try to compile using the ARM compiler (CC=linux-arm-gnueabi-cc) when running the makefile.

Any advice on how I should reason around this issue and/or how CCAN should operate with cross compilation?

If you need any more information about the specifics regarding the issues/errors I can provide them. :)

[cdecker]

Does this work:

make CC=clang clean ccan/tools/configurator/configurator
make CC=linux-arm-gnueabi-cc default

That'd compile the configurator for the current arch, and then uses it when cross-compiling.

[hsjoberg]

@cdecker Thanks! Yes it moves forward at least.

Libs using autotools/autoconfig will fail:

checking whether we are cross compiling... configure: error: in `/home/coco/Projects/C/lightning-arm-linux/external/libwally-core':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
external/Makefile:40: recipe for target 'external/libwally-core/src/secp256k1/libsecp256k1.la' failed
make: *** [external/libwally-core/src/secp256k1/libsecp256k1.la] Error 1

I'll try to manually edit the Makefile(s) to make sure configure runs with the host flag and see if it's possible to progress further.

[cdecker]

Perfect, looking forward to this, will be really helpful for deterministic builds as well #423.

[icota]

@hsjoberg This PR will help you overcome that hurdle. Now I have the build failing due to ARM binaries like ccan/ccan/cdump/tools/cdump-enumstr trying to run on x86. I guess it's a step in the right direction :smile:

[hsjoberg]

Very nice @icota!
That's as far as I got as well, but good that you've started a PR about this.

I'll try to pick up this again soon and perhaps ping rusty on IRC. :)

[icota]

As at d4d1c4acb08efb6be4f491cdee5cb6dd4b84ddf7 you can cross compile but make sure to install qemu-user. Run make CC=clang clean ccan/tools/configurator/configurator and in case you get stuck at the cdump-enumstr step do a make clean -C ccan/ccan/cdump/tools && make CC=clang -C ccan/ccan/cdump/tools. I'm not sure how to tweak Makefile not to cross-compile these two bits.

On Android I need to drop the -Werror flag but on a more vanilla variant of Linux I'm sure the whole process is smooth sailing.

[cdecker]

With #1035 merged this should work now. Can you confirm this @hsjoberg?

[hsjoberg]

@cdecker @icota I can confirm that this compiles and seems to be producing working binaries (only tested lightning-cli so far), _if_ "machine-native" cc is being used instead of the $(CC) variable (which would be set to ARM gcc) in ccan/config.h: ccan/tools/configurator/configurator:

# make clean ccan/tools/configurator/configurator # Clean ccan configurator

make CC=arm-linux-gnueabi-gcc HOST=arm-linux-gnueabi LDFLAGS="-L/<arm lib path>" default

# When cdump-enumstr fails:
make clean -C ccan/ccan/cdump/tools && make CC=clang -C ccan/ccan/cdump/tools

# And then make default again:
make CC=arm-linux-gnueabi-gcc HOST=arm-linux-gnueabi LDFLAGS="-L/<arm lib path>" default

[cdecker]

Thanks @hsjoberg, any way we can make it foolproof, e.g., by not using $(CC) for the configurator? Or can we disentangle CC and HOST?

[cdecker]

This was addressed in 229f288eb5f94770c159e759aa327a80cef2cf8f

[droark]

Hi. Came across this while looking into some CCAN stuff. Just curious, would anybody mind posting the directions they're using to cross-compile now that everything is (supposedly) good to go? I'm attempting to cross-compile for Raspberry Pi on Ubuntu 17.10 and am having issues.

Thanks!

[icota]

Hi @droark. There are Android instructions in INSTALL.md. What issues are you having?

[droark]

@icota - Whoops! I missed that doc somehow. I'll take a second swing at it and see what comes up.

[droark]

Okay. Here are my steps. This is targeted toward RPi3 boards, using the arm-linux-nueabihf toolset. sqlite and gmp have been compiled for it (with appropriate host/CC/etc. changes vis-à-vis what's in the directions). I don't know if there's anything else I need to do; I'm not an old hand at cross-compiling (although I'm getting comfortable with it!). Something might be messed up in GMP, as make check fails horribly. (Then again, maybe the checks are supposed to be run on the relevant architectures?)

COMMANDS

export PATH=$PATH:/path/to/arm-linux-gnueabihf/bin
target_host=arm-linux-gnueabihf
export AR=$target_host-ar
export AS=$target_host-as
export CC=$target_host-gcc
export CXX=$target_host-g++
export LD=$target_host-ld
export STRIP=$target_host-strip
make CC=gcc clean ccan/tools/configurator/configurator
BUILD=x86_64 HOST=arm-linux-gnueabihf make PIE=1 DEVELOPER=0 CONFIGURATOR_CC="arm-linux-gnueabihf-gcc -static" LDFLAGS="-L/path/to/gmp-and-sqlite/lib"

RELEVANT SECOND MAKE OUTPUT

arm-linux-gnueabihf-cc -L/path/to/gmp-and-sqlite/lib  ccan/ccan/cdump/tools/cdump-enumstr.o ccan-cdump.o ccan-strmap.o ccan-asort.o ccan-autodata.o ccan-breakpoint.o ccan-crypto-hmac.o ccan-crypto-hkdf.o ccan-crypto-ripemd160.o ccan-crypto-sha256.o ccan-crypto-shachain.o ccan-crypto-siphash24.o ccan-daemonize.o ccan-err.o ccan-fdpass.o ccan-htable.o ccan-ilog.o ccan-io-io.o ccan-intmap.o ccan-io-poll.o ccan-io-fdpass.o ccan-isaac.o ccan-isaac64.o ccan-list.o ccan-mem.o ccan-noerr.o ccan-opt-helpers.o ccan-opt-parse.o ccan-opt-usage.o ccan-opt.o ccan-pipecmd.o ccan-ptr_valid.o ccan-read_write_all.o ccan-str-hex.o ccan-str.o ccan-take.o ccan-tal-grab_file.o ccan-tal-link.o ccan-tal-path.o ccan-tal-str.o ccan-tal.o ccan-time.o ccan-timer.o  -Lexternal -lwallycore -lsecp256k1 -lsodium -ljsmn -lbase58 -lbacktrace -L/usr/local/lib -lm -lgmp -lsqlite3  -o ccan/ccan/cdump/tools/cdump-enumstr
ccan/ccan/cdump/tools/cdump-enumstr common/htlc_state.h > common/gen_htlc_state_names.h
/lib/ld-linux-armhf.so.3: No such file or directory
common/Makefile:64: recipe for target 'common/gen_htlc_state_names.h' failed
make: *** [common/gen_htlc_state_names.h] Error 255
rm external/libwally-core/src/secp256k1/libsecp256k1.la

[icota]

When you hit that second output try running:

 make clean -C ccan/ccan/cdump/tools \
  && make CC=gcc -C ccan/ccan/cdump/tools \
  && BUILD=x86_64 HOST=arm-linux-gnueabihf make PIE=1 DEVELOPER=0 CONFIGURATOR_CC="arm-linux-gnueabihf-gcc -static" LDFLAGS="-L/path/to/gmp-and-sqlite/lib"

[droark]

Thanks. I ended up having to add CFLAGS in the end. Here's the revised set of commands that led to everything compiling properly. I haven't run it yet and may not be able to do so for awhile. I'm happy to submit a documentation PR, though, and see if it'll get a bit of attention from somebody in a better position to test everything out.

export PATH=$PATH:/path/to/arm-linux-gnueabihf/bin
target_host=arm-linux-gnueabihf
export AR=$target_host-ar
export AS=$target_host-as
export CC=$target_host-gcc
export CXX=$target_host-g++
export LD=$target_host-ld
export STRIP=$target_host-strip
make CC=gcc clean ccan/tools/configurator/configurator
BUILD=x86_64 HOST=arm-linux-gnueabihf make PIE=1 DEVELOPER=0 CONFIGURATOR_CC="arm-linux-gnueabihf-gcc -static" LDFLAGS="-L/path/to/gmp-and-sqlite/lib" CFLAGS="-std=gnu11 -I /path/to/gmp-and-sqlite/include -I . -I ccan -I external/libwally-core/src/secp256k1/include -I external/libsodium/src/libsodium/include -I external/jsmn -I external/libwally-core/include -I external/libbacktrace -I external/libbase58"
make clean -C ccan/ccan/cdump/tools
make CC=gcc -C ccan/ccan/cdump/tools
BUILD=x86_64 HOST=arm-linux-gnueabihf make PIE=1 DEVELOPER=0 CONFIGURATOR_CC="arm-linux-gnueabihf-gcc -static" LDFLAGS="-L/path/to/gmp-and-sqlite/lib" CFLAGS="-std=gnu11 -I /path/to/gmp-and-sqlite/include -I . -I ccan -I external/libwally-core/src/secp256k1/include -I external/libsodium/src/libsodium/include -I external/jsmn -I external/libwally-core/include -I external/libbacktrace -I external/libbase58"

[droark]

Okay. Documentation PR created. Let's continue the conversation over there. :) Thanks.

[arowser]

I always got error:
ccan/tools/configurator/configurator: Test for HAVE_ALIGNOF failed with 512: ./configurator.out: 1: ./configurator.out: Syntax error: word unexpected (expecting ")")

when execute:
BUILD=x86_64 HOST=arm-linux-gnueabihf make PIE=1 DEVELOPER=0 CONFIGURATOR_CC="arm-linux-gnueabihf-gcc -static" LDFLAGS="-Llibs/lib" CFLAGS="-std=gnu11 -I libs/include -I . -I ccan -I external/libwally-core/src/secp256k1/include -I external/libsodium/src/libsodium/include -I external/jsmn -I external/libwally-core/include -I external/libbacktrace -I external/libbase58"
I tested on Ubuntu 14.04.5 LTS with https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/arm-linux-gnueabihf

[droark]

@arowser - I got that at one point too. IIRC, it went away when I set the compiler, linker, etc. as seen above. Are you doing that? Are you following the directions in my PR (#1263)? If it's not working, try sudo apt-get install qemu-user-static and let me know if that helps (or, better yet, just post in the PR). I did that at some point.

[arowser]

I only installed qemu-user, so I install qemu-user-static.
The error seems gone after I installed qemu-user-static , but got a new error,
ccan/tools/configurator/configurator: Test for HAVE_OPENMP failed with 35584: qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault
Its should be qemu issue, I'll try update qemu.

[arowser]

Finally I build success with https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/arm-linux-gnueabihf on a new ubuntu Ubuntu 16.04.3 LTS, and use qemu-user or qemu-user-static is all good, but I did others to make build pass:

1. need set export QEMU_LD_PREFIX=/opt/tools/arm-bcm2708/arm-linux-gnueabihf/arm-linux-gnueabihf/sysroot to fix
   /lib/ld-linux-armhf.so.3: No such file or directory
2. copy sqlite and gmp lib to $QEMU_LD_PREFIX to fix
   error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory

@droark maybe should mention it on #1263 ?

[droark]

@arowser - Thanks. I need to do a fixup and will do so in a bit.