lightningd 0.8.1 breaks during startup on Void Linux

[nzwallaby]

Issue and Steps to Reproduce

I'm running an up to date install of Void Linux voidlinux.org. I've compiled the program from the release source v0.8.1, all seems fine.

I can run lightningd, and it start off well for a little while but then I get a bunch of BROKEN messages with FATAL SIGNAL 6. Sorry for not including too detailed logs, I'm not sure what all the hex messages mean and I don't want to expose any secrets. I replaced all the hexes with 0xg, except for the final one in the stacktrace which is 0xffffffffffffffff

2020-04-12T16:32:58.129Z DEBUG wallet: Restored 0 outgoing HTLCS
2020-04-12T16:32:58.129Z DEBUG wallet: Restored 0 outgoing HTLCS
2020-04-12T16:32:58.129Z DEBUG wallet: Restored 0 outgoing HTLCS
2020-04-12T16:32:58.129Z DEBUG wallet: Restored 0 outgoing HTLCS
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG connectd: REPLY WIRE_CONNECTCTL_ACTIVATE_REPLY with 0 fds
2020-04-12T16:32:58.129Z INFO lightningd: --------------------------------------------------
2020-04-12T16:32:58.129Z INFO lightningd: Server started with public key abcdefgxxxxxxxx, alias wallabynz (color #000000) and lightningd 0.8.1
2020-04-12T16:32:58.129Z DEBUG xxx-chan#123: Will try reconnect in 1 seconds
2020-04-12T16:32:58.129Z DEBUG xxx-chan#1234: Will try reconnect in 2 seconds
2020-04-12T16:32:58.129Z DEBUG xxx-chan#12345: Will try reconnect in 3 seconds
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.129Z DEBUG hsmd: Client: Received message 10 from client
2020-04-12T16:32:58.130Z DEBUG plugin-autoclean: autocleaning not active
2020-04-12T16:32:58.140Z DEBUG lightningd: Adding block 625509: 000000000000000000124ac82bc0639989a68ac12f020429ae49fdc65d6a70eb
lightningd: FATAL SIGNAL 6 (version 0.8.1)
0xg send_backtrace
        common/daemon.c:39
0xg crashdump
        common/daemon.c:52
0xg ???
        /builddir/glibc-2.30/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0xg __GI_raise
        ../sysdeps/unix/sysv/linux/raise.c:51
0xg __GI_abort
        /builddir/glibc-2.30/stdlib/abort.c:79
0xg call_error
        ccan/ccan/tal/tal.c:93
0xg allocate
        ccan/ccan/tal/tal.c:247
0xg tal_alloc_
        ccan/ccan/tal/tal.c:423
0xg tal_alloc_arr_
        ccan/ccan/tal/tal.c:466
0xg bitcoin_tx_output_get_script
        bitcoin/tx.c:206
0xg topo_add_utxos
        lightningd/chaintopology.c:658
0xg add_tip
        lightningd/chaintopology.c:679
0xg get_new_block
        lightningd/chaintopology.c:765
0xg getrawblockbyheight_callback
        lightningd/bitcoind.c:367
0xg plugin_response_handle
        lightningd/plugin.c:258
0xg plugin_read_json_one
        lightningd/plugin.c:356
0xg plugin_read_json
        lightningd/plugin.c:388
0xg next_plan
        ccan/ccan/io/io.c:59
0xg do_plan
        ccan/ccan/io/io.c:407
0xg io_ready
        ccan/ccan/io/io.c:417
0xg io_loop
        ccan/ccan/io/poll.c:445
0xg io_loop_with_timers
        lightningd/io_loop_with_timers.c:24
0xg main
        lightningd/lightningd.c:928
0xg __libc_start_main
        ../csu/libc-start.c:308
0xg ???
        ../sysdeps/x86_64/start.S:120
0xffffffffffffffff ???
        ???:0
2020-04-12T16:32:58.207Z **BROKEN** lightningd: FATAL SIGNAL 6 (version 0.8.1)
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: common/daemon.c:44 (send_backtrace) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: common/daemon.c:52 (crashdump) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: /builddir/glibc-2.30/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ../sysdeps/unix/sysv/linux/raise.c:51 (__GI_raise) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: /builddir/glibc-2.30/stdlib/abort.c:79 (__GI_abort) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:93 (call_error) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:247 (allocate) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:423 (tal_alloc_) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:466 (tal_alloc_arr_) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: bitcoin/tx.c:206 (bitcoin_tx_output_get_script) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:658 (topo_add_utxos) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:679 (add_tip) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:765 (get_new_block) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:367 (getrawblockbyheight_callback) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:258 (plugin_response_handle) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:356 (plugin_read_json_one) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:388 (plugin_read_json) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:928 (main) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: ../sysdeps/x86_64/start.S:120 ((null)) 0xg
2020-04-12T16:32:58.207Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
Log dumped in crash.log.20200412163258
autoclean: fundchannel: Lost connection to the RPC socket.
Lost connection to the RPC socket.
pay: Lost connection to the RPC socket.Aborted
[nzwallaby@localhost ~]$ 
lightning_connectd: Failed parsing get_addrs_reply gossipctl:  (version 0.8.1)
0xg send_backtrace
        common/daemon.c:39
0xg status_failed
        common/status.c:206
0xg add_gossip_addrs
        connectd/connectd.c:1375
0xg try_connect_peer
        connectd/connectd.c:1416
0xg connect_to_peer
        connectd/connectd.c:1480
0xg recv_req
        connectd/connectd.c:1549
0xg handle_read
        common/daemon_conn.c:31
0xg next_plan
        ccan/ccan/io/io.c:59
0xg do_plan
        ccan/ccan/io/io.c:407
0xg io_ready
        ccan/ccan/io/io.c:417
0xg io_loop
        ccan/ccan/io/poll.c:445
0xg main
        connectd/connectd.c:1644
0xg __libc_start_main
        ../csu/libc-start.c:308
0xg ???
        ../sysdeps/x86_64/start.S:120
0xffffffffffffffff ???
        ???:0

[rustyrussell]

You ran out of memory. We don't handle that gracefully, but it's very unusual. How much RAM do you have?

[whitslack]

It's actually not an out-of-memory condition. I've run into this problem before. It comes about because your libwallycore was compiled with a different value of BUILD_ELEMENTS than your C-Lightning was. That causes the size of struct wally_tx_output to differ between the shared library and the executable, and thus when C-Lightning tries to examine wtx->outputs[1] (or any subsequent output), it's not looking in the correct location, and KABOOM!

Arguably this is poor ABI design on the part of libwallycore. It shouldn't be possible for the ABI to silently break depending on a preprocessor definition in the consuming code. But that's the case with libwallycore.

To resolve your problem, be sure you are configuring libwallycore with --enable-elements. And if you have any other programs linked against libwallycore, you'll have to recompile them with -DBUILD_ELEMENTS=1 so that they understand libwallycore's structures in the same way as libwallycore does.

[whitslack]

Oh, and one other thing I should mention: libwallycore's configure.ac contains Bashisms, so if your /bin/sh isn't Bash, you'll have to set CONFIG_SHELL=/bin/bash before configuring libwallycore. If you don't do that, then things break. The BUILD_ELEMENTS setting is one of them, if I recall correctly.

[nzwallaby]

It's actually not an out-of-memory condition. I've run into this problem before. It comes about because your libwallycore was compiled with a different value of BUILD_ELEMENTS than your C-Lightning was.

My understanding is that libwallycore is compiled automatically when compiling c-lightning, so I'm not sure I understand how it would have been compiled with a different value of BUILD_ELEMENTS? I did this on a fresh extract of the zip file with the source.

[whitslack]

libwallycore configures using Autotools. C-Lightning uses ccan configurator. Just set CONFIG_SHELL=/bin/bash in your environment before starting the build.