Lightning: Crash: When shutting down the daemon
Issue and Steps to Reproduce
When stop my node using lightning-cli stop I run into this crashlog stacktrace message. I do not know if this is 'normal' but at least it doesn't look right. Happens on both of my machines on mainnet and testnet.
Details
- Revision: 8ecc6c7 but I have seen this on all recent versions.
- Network: The attached stacktrace happend on testnet, but I have seen it on mainnet as well.
getinfo output (after second restart)
{
"id" : "022368baffc9c5a46e7e0aa9690dd753cba2889e42498369c3643efbb422b492f9",
"alias" : "dev.schmoock.net",
"color" : "ffdd00",
"num_peers" : 5,
"num_pending_channels" : 0,
"num_active_channels" : 5,
"num_inactive_channels" : 0,
"address" : [],
"binding" : [
{
"type" : "ipv6",
"address" : "::",
"port" : 9735
},
{
"type" : "ipv4",
"address" : "0.0.0.0",
"port" : 9735
}
],
"version" : "v0.7.0-376-g8ecc6c7",
"blockheight" : 1518346,
"network" : "testnet",
"msatoshi_fees_collected" : 0,
"fees_collected_msat" : "0msat"
}
Stacktrace
2019-05-25T10:23:02.874Z lightningd(15541):jcon fd 35: JSON-RPC shutdown
2019-05-25T10:23:12.877Z lightning_connectd(15557): Status closed, but waitpid 15557 says No child processes
2019-05-25T10:23:22.879Z lightning_gossipd(15558): Status closed, but waitpid 15558 says No child processes
2019-05-25T10:23:32.882Z lightning_hsmd(15556): Status closed, but waitpid 15556 says No child processes
2019-05-25T10:23:32.892Z lightningd(15541): FATAL SIGNAL 6 (version v0.7.0-376-g8ecc6c7)
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: common/daemon.c:45 (send_backtrace) 0x56492b61fe7c
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: common/daemon.c:53 (crashdump) 0x56492b61fecc
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0x7fa1063d78af
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0x7fa1063d782f
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0x7fa1063c2671
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:93 (call_error) 0x56492b6708e8
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:165 (check_bounds) 0x56492b670ab0
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:175 (to_tal_hdr) 0x56492b670afe
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:678 (tal_parent) 0x56492b671ba2
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: common/daemon.c:102 (steal_notify) 0x56492b620090
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:237 (notify) 0x56492b670cbc
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:508 (tal_steal_) 0x56492b671610
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:789 (tal_dup_) 0x56492b67207e
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: common/msg_queue.c:19 (do_enqueue) 0x56492b6266e9
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: common/msg_queue.c:33 (msg_enqueue) 0x56492b626776
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: lightningd/subd.c:728 (subd_send_msg) 0x56492b619b42
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: lightningd/channel.c:37 (channel_set_owner) 0x56492b5e9577
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: lightningd/channel.c:98 (destroy_channel) 0x56492b5e9766
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:235 (notify) 0x56492b670c95
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:397 (del_tree) 0x56492b671184
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: ccan/ccan/tal/tal.c:481 (tal_free) 0x56492b671510
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: lightningd/lightningd.c:430 (shutdown_subdaemons) 0x56492b5fada7
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: lightningd/lightningd.c:838 (main) 0x56492b5fb6da
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0x7fa1063c3ce2
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0x56492b5e3abd
2019-05-25T10:23:32.892Z lightningd(15541): backtrace: (null):0 ((null)) 0xffffffffffffffff
Another Stacktrace with the same symptom and cause to reproduce:
019-05-25T10:54:16.130Z lightningd(15880):jcon fd 35: JSON-RPC shutdown
2019-05-25T10:54:26.134Z lightning_connectd(15896): Status closed, but waitpid 15896 says No child processes
2019-05-25T10:54:36.136Z lightning_gossipd(15897): Status closed, but waitpid 15897 says No child processes
2019-05-25T10:54:46.140Z lightning_hsmd(15895): Status closed, but waitpid 15895 says No child processes
2019-05-25T10:54:46.172Z lightningd(15880): FATAL SIGNAL 6 (version v0.7.0-376-g8ecc6c7)
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: common/daemon.c:45 (send_backtrace) 0x55d53209ce7c
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: common/daemon.c:53 (crashdump) 0x55d53209cecc
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf208af
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf2082f
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf0b671
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf62e77
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf69789
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf69fb7
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf6c367
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf6db56
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb36fc11
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb36f304
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb37aac2
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb39a85a
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb38272d
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb382bfd
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb383775
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edb369ba2
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: wallet/db.c:646 (db_commit_transaction) 0x55d5320c7ca0
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: lightningd/lightningd.c:449 (shutdown_subdaemons) 0x55d532077e3d
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: lightningd/lightningd.c:838 (main) 0x55d5320786da
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x7f6edaf0cce2
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0x55d532060abd
2019-05-25T10:54:46.173Z lightningd(15880): backtrace: (null):0 ((null)) 0xffffffffffffffff
m-schmoock
All 28 comments
These may be two unrelated crashes!
First crash
I start with destroy_channel. Stacktrace points:
Seems kosher. First argument is the channel being destroyed, second is the new owner (NULL means nobody will own it now), last is whether to reconnect (false means we do not reconnect, which makes sense, we are shutting down).
Now to next stacktrace point inside channel_set_owner. And now it is interesting:
If we are connected, and the subdaemon handling the channel is one which talks directly to the peer (anything that is not onchaind, I think), we tell connectd that the peer is disconnected.
Except.... prior to line 430 inside shutdown_subdaemons:
We already gave connectd 10 seconds to kill itself.
So it looks like cleaning up the channels may lead to the connectd being contacted after it has been told to kill itself.
@cdecker @rustyrussell @niftynei can you confirm my analysis?
Second crash
This is a lot more disturbing.
The last point we can see in the stacktrace is:
Everything else in the stacktrace, unfortunately, seems to be inside libsqlite3.
Strange not to have the debug symbols....
ZmnSCPxj
on 25 May 2019
👍1
@ZmnSCPxj Good analysis. Also this happens occasionally on shutdown:
2019-05-25T15:26:24.534Z lightning_connectd(18602): Status closed, but waitpid 18602 says No child processes
2019-05-25T15:26:34.537Z lightning_gossipd(18603): Status closed, but waitpid 18603 says No child processes
2019-05-25T15:26:44.540Z lightning_hsmd(18601): Status closed, but waitpid 18601 says No child processes
2019-05-25T15:26:44.549Z lightningd(18586): FATAL SIGNAL 11 (version v0.7.0-380-gbbec8f6)
2019-05-25T15:26:44.601Z lightningd(18586): backtrace: common/daemon.c:45 (send_backtrace) 0x56411fe5fe7c
2019-05-25T15:26:44.601Z lightningd(18586): backtrace: common/daemon.c:53 (crashdump) 0x56411fe5fecc
2019-05-25T15:26:44.601Z lightningd(18586): backtrace: (null):0 ((null)) 0x7f547adda8af
2019-05-25T15:26:44.601Z lightningd(18586): backtrace: (null):0 ((null)) (nil)
Edit: These are all I can have on 'graceful' shutdowns. At least one of them happens each time.
m-schmoock
on 25 May 2019
That is a very short backtrace containing nothing at all after we have returned from main.... sigh. It looks to be memory corruption? That would be bad, very bad, need to valgrind this.
By any chance, are you running on a single board computer (Rasp Pi or similar) with some additional devices connected? Those can have unreliable memory unless you upgrade the amperage of the power supply.
ZmnSCPxj
on 26 May 2019
Reviewing the backtrace of the first crash... it looks to be memory corruption. The part that crashed is a simple utility to ensure we do not have cycles in our object graph (which will not get freed, like in CPython). It does this by tracing the parent (owner) of each object until it reaches a parentless object. Along the way, it crashed because a parent pointer was pointing to an object outside of our heap. But all our objects are still valid and the "outside the heap" check is really very rough (i.e. it will still say "inside the heap" for objects that were already freed, this is really a very rough check). That suggests to me memory corruption. The other crashes are code that is outside lightningd --- libsqlite3 in the second crashdump, probably libc in the latest.
Edit: These are all I can have on 'graceful' shutdowns. At least one of them happens each time.
What do you mean? These three backtraces are the only backtraces you have? Or do you mean that there is always a crash at each shutdown but backtraces are different each time?
ZmnSCPxj
on 26 May 2019
Hi @m-schmoock, have you considered the possibility of hardware problem as suggested above?
ZmnSCPxj
on 27 May 2019
Hi, @ZmnSCPxj @m-schmoock
can confirm this on Rasp Pi's with latest builds too
So far I found a working build from 11 April that crashed from other reasons
The current situation is this:
If I do a lightning-cli stop
The crash.log looks always like this
+294.815576483 lightningd(4569):jcon fd 24:IO_OUT: 7b20226a736f6e727063223a2022322e30222c2022696422203a20226c696768746e696e672d636c692d34363733222c2022726573756c7422203a20225368757474696e6720646f776e22207d0a0a
+294.815663149 lightningd(4569):jcon fd 24:UNUSUAL: JSON-RPC shutdown
+294.815841377 lightning_connectd(4581):DEBUG: Shutting down
+304.816843816 lightning_connectd(4581):DEBUG: Status closed, but not exited. Killing
+304.818507554 lightning_gossipd(4582):DEBUG: Shutting down
+314.819446147 lightning_gossipd(4582):DEBUG: Status closed, but not exited. Killing
+314.824543038 lightning_hsmd(4580):DEBUG: Shutting down
+324.825476011 lightning_hsmd(4580):DEBUG: Status closed, but not exited. Killing
+324.827055896 lightningd(4569):BROKEN: FATAL SIGNAL 11 (version v0.7.0-390-g8b4ba8b)
+905.883629782 lightningd(7765):jcon fd 24:IO_OUT: 7b20226a736f6e727063223a2022322e30222c2022696422203a20226c696768746e696e672d636c692d38303535222c2022726573756c7422203a20225368757474696e6720646f776e22207d0a0a
+905.883724000 lightningd(7765):jcon fd 24:UNUSUAL: JSON-RPC shutdown
+905.883966551 lightning_connectd(7776):DEBUG: Shutting down
+915.885057087 lightning_connectd(7776):DEBUG: Status closed, but not exited. Killing
+915.886676398 lightning_gossipd(7777):DEBUG: Shutting down
+925.887839762 lightning_gossipd(7777):DEBUG: Status closed, but not exited. Killing
+925.891213540 lightning_hsmd(7775):DEBUG: Shutting down
+935.892281762 lightning_hsmd(7775):DEBUG: Status closed, but not exited. Killing
+935.893921073 lightningd(7765):BROKEN: FATAL SIGNAL 11 (version v0.7.0-397-g22600fa)
Saibato
on 28 May 2019
@ZmnSCPxj Its happening on my new 6core i7 Laptop and on my hosted dedicated root Server. Wastly most of the times its the first stacktrace (>90%). Both machines do not have any issues in normal Operation.
Edit: Both system use an up-to-date Archlinux
m-schmoock
on 28 May 2019
Here is another one similar but slightly different from the first one. And really, I only have stacktrace on 'graceful' shutdowns. no other problems whatsoever during runtime.
2019-05-28T23:43:57.161Z lightningd(16599):jcon fd 35: JSON-RPC shutdown
2019-05-28T23:44:07.162Z lightning_connectd(16615): Status closed, but waitpid 16615 says No child processes
2019-05-28T23:44:17.165Z lightning_gossipd(16616): Status closed, but waitpid 16616 says No child processes
2019-05-28T23:44:27.167Z lightning_hsmd(16614): Status closed, but waitpid 16614 says No child processes
2019-05-28T23:44:27.199Z lightningd(16599): FATAL SIGNAL 11 (version v0.7.0-392-gd7b1d3b)
2019-05-28T23:44:27.199Z lightningd(16599): backtrace: common/daemon.c:45 (send_backtrace) 0x55dd4bbe0cb8
2019-05-28T23:44:27.199Z lightningd(16599): backtrace: common/daemon.c:53 (crashdump) 0x55dd4bbe0d08
2019-05-28T23:44:27.199Z lightningd(16599): backtrace: (null):0 ((null)) 0x7f7ab03578af
2019-05-28T23:44:27.199Z lightningd(16599): backtrace: ccan/ccan/tal/tal.c:713 (tal_resize_) 0x55dd4bc335e4
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: common/msg_queue.c:19 (do_enqueue) 0x55dd4bbe74e3
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: common/msg_queue.c:33 (msg_enqueue) 0x55dd4bbe75bc
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: lightningd/subd.c:726 (subd_send_msg) 0x55dd4bbda97e
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: lightningd/channel.c:37 (channel_set_owner) 0x55dd4bbaa577
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: lightningd/channel.c:98 (destroy_channel) 0x55dd4bbaa766
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: ccan/ccan/tal/tal.c:235 (notify) 0x55dd4bc32559
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: ccan/ccan/tal/tal.c:397 (del_tree) 0x55dd4bc32a48
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: ccan/ccan/tal/tal.c:481 (tal_free) 0x55dd4bc32dd4
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: lightningd/lightningd.c:430 (shutdown_subdaemons) 0x55dd4bbbbd39
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: lightningd/lightningd.c:838 (main) 0x55dd4bbbc66c
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: (null):0 ((null)) 0x7f7ab0343ce2
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: (null):0 ((null)) 0x55dd4bba4abd
2019-05-28T23:44:27.200Z lightningd(16599): backtrace: (null):0 ((null)) 0xffffffffffffffff
Looks like some kind of memory corruption caused by us, then, which is very bad.
ZmnSCPxj
on 29 May 2019
@ZmnSCPxj Can I help in investigating, since not all users seem to have this? I use an up-to-date Archlinux.
@Saibato you said you also have these shutdown crashes, what System/OS are you using?
m-schmoock
on 29 May 2019
How slow does it get under valgrind? If too slow, strace but really we need to know what memory we are poking into...
ZmnSCPxj
on 29 May 2019
When using valgrind doing lightnign-cli stop will cause a lot of complains...
CLICK TO EXPAND
2019-05-29T13:16:04.519Z lightningd(11839):jcon fd 35: JSON-RPC shutdown
2019-05-29T13:16:14.544Z lightning_connectd(11855): Status closed, but waitpid 11855 says No child processes
2019-05-29T13:16:24.551Z lightning_gossipd(11856): Status closed, but waitpid 11856 says No child processes
2019-05-29T13:16:34.570Z lightning_hsmd(11854): Status closed, but waitpid 11854 says No child processes
2019-05-29T13:16:34.808Z lightningd(11839): FATAL SIGNAL 6 (version v0.7.0-401-g1141243)
2019-05-29T13:16:34.809Z lightningd(11839): backtrace: (null):0 ((null)) 0x14c232
2019-05-29T13:16:34.809Z lightningd(11839): backtrace: (null):0 ((null)) 0x14c282
2019-05-29T13:16:34.809Z lightningd(11839): backtrace: (null):0 ((null)) 0x4dde8af
2019-05-29T13:16:34.809Z lightningd(11839): backtrace: (null):0 ((null)) 0x4dde82f
2019-05-29T13:16:34.809Z lightningd(11839): backtrace: (null):0 ((null)) 0x4dc9671
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x19d190
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x19e525
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x152a53
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x152b2c
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x145ed4
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x1155ee
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x1157dd
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x19d53d
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x19da2c
2019-05-29T13:16:34.810Z lightningd(11839): backtrace: (null):0 ((null)) 0x19ddb8
2019-05-29T13:16:34.811Z lightningd(11839): backtrace: (null):0 ((null)) 0x12708d
2019-05-29T13:16:34.811Z lightningd(11839): backtrace: (null):0 ((null)) 0x1279c0
2019-05-29T13:16:34.811Z lightningd(11839): backtrace: (null):0 ((null)) 0x4dcace2
2019-05-29T13:16:34.811Z lightningd(11839): backtrace: (null):0 ((null)) 0x10fadd
2019-05-29T13:16:34.811Z lightningd(11839): backtrace: (null):0 ((null)) 0xffffffffffffffff
========= NOW VALGRIND OUTPUT ==========
[will@will-lap lightning.git (master)]$ ==11881== Invalid read of size 8
==11881== at 0x145E73: subd_send_msg (subd.c:725)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5750340 is 96 bytes inside a block of size 176 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19DB40: tal_alloc_ (tal.c:423)
==11881== by 0x14598B: new_subd (subd.c:622)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881== by 0x1276CE: main (lightningd.c:700)
==11881==
==11881== Invalid read of size 8
==11881== at 0x145EC2: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5750378 is 152 bytes inside a block of size 176 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19DB40: tal_alloc_ (tal.c:423)
==11881== by 0x14598B: new_subd (subd.c:622)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881== by 0x1276CE: main (lightningd.c:700)
==11881==
==11881== Invalid read of size 8
==11881== at 0x152A23: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5750b58 is 40 bytes inside a block of size 48 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19DB40: tal_alloc_ (tal.c:423)
==11881== by 0x1529D7: msg_queue_new (msg_queue.c:12)
==11881== by 0x145BBE: new_subd (subd.c:659)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881== by 0x1276CE: main (lightningd.c:700)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D384: to_tal_hdr (tal.c:174)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f48 is 24 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D39C: to_tal_hdr (tal.c:175)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f30 is 0 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3AB: to_tal_hdr (tal.c:176)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f38 is 8 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3BB: to_tal_hdr (tal.c:177)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3C8: to_tal_hdr (tal.c:177)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 1
==11881== at 0x19D24A: is_literal (tal.c:128)
==11881== by 0x19D3D3: to_tal_hdr (tal.c:177)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== Address 0x5750da0 is 0 bytes inside a block of size 32 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAC8: del_tree (tal.c:414)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19D700: add_notifier_property (tal.c:300)
==11881== by 0x19DFEA: tal_add_notifier_ (tal.c:544)
==11881== by 0x14C497: add_steal_notifier (daemon.c:111)
==11881== by 0x19D564: notify (tal.c:237)
==11881== by 0x19DC04: tal_alloc_ (tal.c:437)
==11881== by 0x19DD01: tal_alloc_arr_ (tal.c:466)
==11881== by 0x1529FD: msg_queue_new (msg_queue.c:13)
==11881== by 0x145BBE: new_subd (subd.c:659)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3DF: to_tal_hdr (tal.c:178)
==11881== by 0x19D418: to_tal_hdr_or_null (tal.c:186)
==11881== by 0x19E300: tal_bytelen (tal.c:632)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19E311: tal_bytelen (tal.c:634)
==11881== by 0x152A2D: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f50 is 32 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19E4B9: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5750b58 is 40 bytes inside a block of size 48 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19DB40: tal_alloc_ (tal.c:423)
==11881== by 0x1529D7: msg_queue_new (msg_queue.c:12)
==11881== by 0x145BBE: new_subd (subd.c:659)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881== by 0x1276CE: main (lightningd.c:700)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D384: to_tal_hdr (tal.c:174)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f48 is 24 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D39C: to_tal_hdr (tal.c:175)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f30 is 0 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3AB: to_tal_hdr (tal.c:176)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f38 is 8 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3BB: to_tal_hdr (tal.c:177)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3C8: to_tal_hdr (tal.c:177)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid read of size 1
==11881== at 0x19D24A: is_literal (tal.c:128)
==11881== by 0x19D3D3: to_tal_hdr (tal.c:177)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== Address 0x5750da0 is 0 bytes inside a block of size 32 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAC8: del_tree (tal.c:414)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483877F: malloc (vg_replace_malloc.c:299)
==11881== by 0x19D5A7: allocate (tal.c:245)
==11881== by 0x19D700: add_notifier_property (tal.c:300)
==11881== by 0x19DFEA: tal_add_notifier_ (tal.c:544)
==11881== by 0x14C497: add_steal_notifier (daemon.c:111)
==11881== by 0x19D564: notify (tal.c:237)
==11881== by 0x19DC04: tal_alloc_ (tal.c:437)
==11881== by 0x19DD01: tal_alloc_arr_ (tal.c:466)
==11881== by 0x1529FD: msg_queue_new (msg_queue.c:13)
==11881== by 0x145BBE: new_subd (subd.c:659)
==11881== by 0x145D5C: new_global_subd (subd.c:687)
==11881== by 0x119F91: connectd_init (connect_control.c:354)
==11881==
==11881== Invalid read of size 8
==11881== at 0x19D3DF: to_tal_hdr (tal.c:178)
==11881== by 0x19E4C3: tal_resize_ (tal.c:689)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f40 is 16 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881== Invalid free() / delete / delete[] / realloc()
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5cb9f30 is 0 bytes inside a block of size 40 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DA7E: del_tree (tal.c:407)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Block was alloc'd at
==11881== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==11881== by 0x19E50E: tal_resize_ (tal.c:694)
==11881== by 0x152C34: msg_dequeue (msg_queue.c:54)
==11881== by 0x14583C: msg_send_next (subd.c:582)
==11881== by 0x18EB0A: next_plan (io.c:59)
==11881== by 0x18F687: do_plan (io.c:407)
==11881== by 0x18F717: io_ready (io.c:423)
==11881== by 0x19171A: io_loop (poll.c:445)
==11881== by 0x1279A2: main (lightningd.c:823)
==11881==
==11881==
==11881== Process terminating with default action of signal 6 (SIGABRT): dumping core
==11881== at 0x4DDE82F: raise (in /usr/lib/libc-2.29.so)
==11881== by 0x4DC9671: abort (in /usr/lib/libc-2.29.so)
==11881== by 0x19D190: call_error (tal.c:93)
==11881== by 0x19E525: tal_resize_ (tal.c:696)
==11881== by 0x152A53: do_enqueue (msg_queue.c:19)
==11881== by 0x152B2C: msg_enqueue (msg_queue.c:33)
==11881== by 0x145ED4: subd_send_msg (subd.c:726)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881==
==11881== HEAP SUMMARY:
==11881== in use at exit: 8,430,033 bytes in 119,937 blocks
==11881== total heap usage: 649,653 allocs, 529,716 frees, 123,556,135 bytes allocated
==11881==
==11881== LEAK SUMMARY:
==11881== definitely lost: 0 bytes in 0 blocks
==11881== indirectly lost: 0 bytes in 0 blocks
==11881== possibly lost: 1,782,976 bytes in 409 blocks
==11881== still reachable: 6,647,057 bytes in 119,528 blocks
==11881== suppressed: 0 bytes in 0 blocks
==11881== Rerun with --leak-check=full to see details of leaked memory
==11881==
==11881== For counts of detected and suppressed errors, rerun with: -v
==11881== ERROR SUMMARY: 20 errors from 20 contexts (suppressed: 0 from 0)
Thank you. I am investigating 2687 for now, I shall try to investigate your dumped logs in a day or two.
ZmnSCPxj
on 29 May 2019
Well look at that:
==11881== Invalid read of size 8
==11881== at 0x145E73: subd_send_msg (subd.c:725)
==11881== by 0x1155EE: channel_set_owner (channel.c:37)
==11881== by 0x1157DD: destroy_channel (channel.c:98)
==11881== by 0x19D53D: notify (tal.c:235)
==11881== by 0x19DA2C: del_tree (tal.c:397)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x12708D: shutdown_subdaemons (lightningd.c:430)
==11881== by 0x1279C0: main (lightningd.c:838)
==11881== Address 0x5750340 is 96 bytes inside a block of size 176 free'd
==11881== at 0x48399AB: free (vg_replace_malloc.c:530)
==11881== by 0x19DAFA: del_tree (tal.c:416)
==11881== by 0x19DDB8: tal_free (tal.c:481)
==11881== by 0x146047: subd_shutdown (subd.c:773)
==11881== by 0x12703E: shutdown_subdaemons (lightningd.c:411)
==11881== by 0x1279C0: main (lightningd.c:838)
For reference, this is line 411 of lightningd.c:
Now, like I mentioned in https://github.com/ElementsProject/lightning/issues/2677#issuecomment-495922697 if a channel is destroyed, we move responsibility for the peer back to connectd.
However at the point that we do this, we have already destroyed connectd, and apparently also the object representing it has already been freed as well.
This can lead to memory corruption since some malloc implementations might deliberately write garbage 0xDEADBEEF data into freed memory, or the memory might be reused in some temporary objects (which overwrites parts of the object). Worse, at the end of the valgrind log we see a double-free, which can destroy the malloc implementation heap structure, leading to strange crashes later (as in the libsqlite crash and the crash inside libc after main has returned).
There are two solutions as I see it:
- Move destruction of global daemons to after the destruction of the channel objects.
- Clear
ld->channeld(and all the other global daemons, for symmetry) and insidedestroy_channel, only pass toconnectdif it is non-NULL.
ZmnSCPxj
on 30 May 2019
It should be possible to replicate this bug in a test that is enabled in VALGRIND mode, by setting up two nodes with a live channel (in CHANNELD_NORMAL) between them, and doing a shutdown on one node. I shall try to make such a test replicating this bug some hours or a few days from now, then actually fix the bug (probably by suppressing the transfer to connectd, I am wary of moving the connectd shutdown to later in the program)
ZmnSCPxj
on 30 May 2019
@Saibato you said you also have these shutdown crashes, what System/OS are you using?
Linux raspberrypi 4.9.35-v7+ # 1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
潭a潭n潭d潭 or
Linux raspberrypi 4.9.80-v7+ # 1098 SMP Fri Mar 9 19:11:42 GMT 2018 armv7l GNU/Linux
Saibato
on 30 May 2019
@Saibato same stacktraces as I have?
Especially the first one is very often
m-schmoock
on 30 May 2019
same stacktraces as I have?
Especially the first one is very often
@m-schmoock @ZmnSCPxj , sorry no backtrace I have backrace disabled on the pi's
and on my debian's (x86) I never had this problem
But I found a workaround that prevents the crash on my pi's. ( Diff in Patch.txt )
.
When the channels are tal free'd before the semi manual free of the deamons
the progie exits peacefully without a crash.
Hope that helps
Saibato
on 31 May 2019
Can you also try this? https://github.com/ZmnSCPxj/lightning/tree/fix-2677
I am unable to replicate the bug in a simple test.
In fact, it seems that subd_shutdown is inconsistent about whether it frees the struct subd or not.
If the daemon is force-killed by subd_shutdown, then the struct subd is freed, but if the daemon completes immediately, then the struct subd is not freed. So the replication needs to somehow emulate connectd taking more than 10 seconds to handle getting killed.
ZmnSCPxj
on 31 May 2019
tACK https://github.com/ZmnSCPxj/lightning/commit/64b9517c70f3166ab5a33c472b3914f81fc3b760 :+1:
Yupp works on Linux raspberrypi 4.9.35-v7+ # 1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
Saibato
on 31 May 2019
Thank you.
ZmnSCPxj
on 31 May 2019
❤1
That didnt do it for me:
I cheery-picked your 64b9517c70f3166ab5a33c472b3914f81fc3b760 on a clean master:
* bbe63711 (HEAD -> master) lightningd/channel.c: Transfer peer to connectd only if connectd alive.
* 11412435 (origin/master, origin/HEAD) Channeld: init channel with remote announcement info when restart
Still 'graceful' shutdown issues:
2019-05-31T09:02:47.292Z lightningd(7297):jcon fd 33: JSON-RPC shutdown
2019-05-31T09:02:57.295Z lightning_connectd(7311): Status closed, but waitpid 7311 says No child processes
2019-05-31T09:03:07.299Z lightning_gossipd(7312): Status closed, but waitpid 7312 says No child processes
2019-05-31T09:03:17.301Z lightning_hsmd(7310): Status closed, but waitpid 7310 says No child processes
2019-05-31T09:03:17.331Z lightningd(7297): FATAL SIGNAL 6 (version v0.7.0-402-gbbe6371)
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: common/daemon.c:45 (send_backtrace) 0x5567d9fc724a
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: common/daemon.c:53 (crashdump) 0x5567d9fc729a
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0x7f9c9b40d8af
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0x7f9c9b40d82f
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0x7f9c9b3f8671
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:93 (call_error) 0x5567da0181a8
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:165 (check_bounds) 0x5567da018370
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:174 (to_tal_hdr) 0x5567da0183af
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:186 (to_tal_hdr_or_null) 0x5567da018430
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:632 (tal_bytelen) 0x5567da019318
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: common/msg_queue.c:19 (do_enqueue) 0x5567d9fcda45
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: common/msg_queue.c:33 (msg_enqueue) 0x5567d9fcdb44
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: lightningd/subd.c:726 (subd_send_msg) 0x5567d9fc0eec
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: lightningd/channel.c:45 (channel_set_owner) 0x5567d9f90606
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: lightningd/channel.c:108 (destroy_channel) 0x5567d9f907f5
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:235 (notify) 0x5567da018555
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:397 (del_tree) 0x5567da018a44
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: ccan/ccan/tal/tal.c:481 (tal_free) 0x5567da018dd0
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: lightningd/lightningd.c:430 (shutdown_subdaemons) 0x5567d9fa20a5
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: lightningd/lightningd.c:838 (main) 0x5567d9fa29d8
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0x7f9c9b3f9ce2
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0x5567d9f8aadd
2019-05-31T09:03:17.331Z lightningd(7297): backtrace: (null):0 ((null)) 0xffffffffffffffff
@ZmnSCPxj And when using valgrind doing lightnign-cli stop still this:
CLICK TO EXPAND
==24161== at 0x145E8B: subd_send_msg (subd.c:725)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x575f680 is 96 bytes inside a block of size 176 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19DB58: tal_alloc_ (tal.c:423)
==24161== by 0x1459A3: new_subd (subd.c:622)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161== by 0x1276E6: main (lightningd.c:700)
==24161==
==24161== Invalid read of size 8
==24161== at 0x145EDA: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x575f6b8 is 152 bytes inside a block of size 176 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19DB58: tal_alloc_ (tal.c:423)
==24161== by 0x1459A3: new_subd (subd.c:622)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161== by 0x1276E6: main (lightningd.c:700)
==24161==
==24161== Invalid read of size 8
==24161== at 0x152A3B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x575fe98 is 40 bytes inside a block of size 48 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19DB58: tal_alloc_ (tal.c:423)
==24161== by 0x1529EF: msg_queue_new (msg_queue.c:12)
==24161== by 0x145BD6: new_subd (subd.c:659)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161== by 0x1276E6: main (lightningd.c:700)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D39C: to_tal_hdr (tal.c:174)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebf8 is 24 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3B4: to_tal_hdr (tal.c:175)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebe0 is 0 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3C3: to_tal_hdr (tal.c:176)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebe8 is 8 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3D3: to_tal_hdr (tal.c:177)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3E0: to_tal_hdr (tal.c:177)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 1
==24161== at 0x19D262: is_literal (tal.c:128)
==24161== by 0x19D3EB: to_tal_hdr (tal.c:177)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== Address 0x57600e0 is 0 bytes inside a block of size 32 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DAE0: del_tree (tal.c:414)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19D718: add_notifier_property (tal.c:300)
==24161== by 0x19E002: tal_add_notifier_ (tal.c:544)
==24161== by 0x14C4AF: add_steal_notifier (daemon.c:111)
==24161== by 0x19D57C: notify (tal.c:237)
==24161== by 0x19DC1C: tal_alloc_ (tal.c:437)
==24161== by 0x19DD19: tal_alloc_arr_ (tal.c:466)
==24161== by 0x152A15: msg_queue_new (msg_queue.c:13)
==24161== by 0x145BD6: new_subd (subd.c:659)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3F7: to_tal_hdr (tal.c:178)
==24161== by 0x19D430: to_tal_hdr_or_null (tal.c:186)
==24161== by 0x19E318: tal_bytelen (tal.c:632)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19E329: tal_bytelen (tal.c:634)
==24161== by 0x152A45: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ec00 is 32 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19E4D1: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x575fe98 is 40 bytes inside a block of size 48 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19DB58: tal_alloc_ (tal.c:423)
==24161== by 0x1529EF: msg_queue_new (msg_queue.c:12)
==24161== by 0x145BD6: new_subd (subd.c:659)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161== by 0x1276E6: main (lightningd.c:700)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D39C: to_tal_hdr (tal.c:174)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebf8 is 24 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3B4: to_tal_hdr (tal.c:175)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebe0 is 0 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3C3: to_tal_hdr (tal.c:176)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebe8 is 8 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3D3: to_tal_hdr (tal.c:177)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3E0: to_tal_hdr (tal.c:177)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid read of size 1
==24161== at 0x19D262: is_literal (tal.c:128)
==24161== by 0x19D3EB: to_tal_hdr (tal.c:177)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== Address 0x57600e0 is 0 bytes inside a block of size 32 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DAE0: del_tree (tal.c:414)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483877F: malloc (vg_replace_malloc.c:299)
==24161== by 0x19D5BF: allocate (tal.c:245)
==24161== by 0x19D718: add_notifier_property (tal.c:300)
==24161== by 0x19E002: tal_add_notifier_ (tal.c:544)
==24161== by 0x14C4AF: add_steal_notifier (daemon.c:111)
==24161== by 0x19D57C: notify (tal.c:237)
==24161== by 0x19DC1C: tal_alloc_ (tal.c:437)
==24161== by 0x19DD19: tal_alloc_arr_ (tal.c:466)
==24161== by 0x152A15: msg_queue_new (msg_queue.c:13)
==24161== by 0x145BD6: new_subd (subd.c:659)
==24161== by 0x145D74: new_global_subd (subd.c:687)
==24161== by 0x119FA9: connectd_init (connect_control.c:354)
==24161==
==24161== Invalid read of size 8
==24161== at 0x19D3F7: to_tal_hdr (tal.c:178)
==24161== by 0x19E4DB: tal_resize_ (tal.c:689)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebf0 is 16 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161== Invalid free() / delete / delete[] / realloc()
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x1270A5: shutdown_subdaemons (lightningd.c:430)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Address 0x5a7ebe0 is 0 bytes inside a block of size 40 free'd
==24161== at 0x48399AB: free (vg_replace_malloc.c:530)
==24161== by 0x19DB12: del_tree (tal.c:416)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DA96: del_tree (tal.c:407)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161== by 0x14605F: subd_shutdown (subd.c:773)
==24161== by 0x127056: shutdown_subdaemons (lightningd.c:411)
==24161== by 0x1279D8: main (lightningd.c:838)
==24161== Block was alloc'd at
==24161== at 0x483AD7B: realloc (vg_replace_malloc.c:826)
==24161== by 0x19E526: tal_resize_ (tal.c:694)
==24161== by 0x152C4C: msg_dequeue (msg_queue.c:54)
==24161== by 0x145854: msg_send_next (subd.c:582)
==24161== by 0x18EB22: next_plan (io.c:59)
==24161== by 0x18F69F: do_plan (io.c:407)
==24161== by 0x18F72F: io_ready (io.c:423)
==24161== by 0x191732: io_loop (poll.c:445)
==24161== by 0x1279BA: main (lightningd.c:823)
==24161==
==24161==
==24161== Process terminating with default action of signal 6 (SIGABRT): dumping core
==24161== at 0x4DDE82F: raise (in /usr/lib/libc-2.29.so)
==24161== by 0x4DC9671: abort (in /usr/lib/libc-2.29.so)
==24161== by 0x19D1A8: call_error (tal.c:93)
==24161== by 0x19E53D: tal_resize_ (tal.c:696)
==24161== by 0x152A6B: do_enqueue (msg_queue.c:19)
==24161== by 0x152B44: msg_enqueue (msg_queue.c:33)
==24161== by 0x145EEC: subd_send_msg (subd.c:726)
==24161== by 0x115606: channel_set_owner (channel.c:45)
==24161== by 0x1157F5: destroy_channel (channel.c:108)
==24161== by 0x19D555: notify (tal.c:235)
==24161== by 0x19DA44: del_tree (tal.c:397)
==24161== by 0x19DDD0: tal_free (tal.c:481)
==24161==
==24161== HEAP SUMMARY:
==24161== in use at exit: 3,674,969 bytes in 51,722 blocks
==24161== total heap usage: 260,441 allocs, 208,719 frees, 36,950,499 bytes allocated
==24161==
==24161== LEAK SUMMARY:
==24161== definitely lost: 0 bytes in 0 blocks
==24161== indirectly lost: 0 bytes in 0 blocks
==24161== possibly lost: 270,056 bytes in 62 blocks
==24161== still reachable: 3,404,913 bytes in 51,660 blocks
==24161== suppressed: 0 bytes in 0 blocks
==24161== Rerun with --leak-check=full to see details of leaked memory
==24161==
==24161== For counts of detected and suppressed errors, rerun with: -v
==24161== ERROR SUMMARY: 20 errors from 20 contexts (suppressed: 0 from 0)
@m-schmoock there are two commits on that tree, you should have cherry-picked both, not just the tip. The first commit is preparation for the second (i.e. the first commit actually clears the pointer in lightningd, the second commit suppresses moving to connectd if the pointer is cleared).
ZmnSCPxj
on 31 May 2019
i.e. cherrypick 165dec9 and 64b9517
ZmnSCPxj
on 31 May 2019
👍2
Whoops, let me retry :D
m-schmoock
on 31 May 2019
🚀2
@m-schmoock to come clear with my changes I dropped mine and checkout the complete https://github.com/ZmnSCPxj/lightning/tree/fix-2677
Saibato
on 31 May 2019
Now, this works like a charm, no more stacktraces. I would prefer a log entry that shutdown was completed though:
2019-05-31T09:38:26.700Z lightningd(5024):jcon fd 33: JSON-RPC shutdown
2019-05-31T09:38:36.702Z lightning_connectd(5038): Status closed, but waitpid 5038 says No child processes
2019-05-31T09:38:46.704Z lightning_gossipd(5039): Status closed, but waitpid 5039 says No child processes
2019-05-31T09:38:56.707Z lightning_hsmd(5037): Status closed, but waitpid 5037 says No child processes
$ lightning-cli stop
"Shutting down"
$ echo $?
0
🎉1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
brunoaduarte
路
4Comments
jonasnick
路
3Comments
saubyk
路
3Comments
brunoaduarte
路
5Comments
cdecker
路
4Comments
Most helpful comment
Whoops, let me retry :D