Lightning: Three crashes every time it sync'd with blockchain.

[Ulmo]

Issue and Steps to Reproduce

I just woke up and checked, and yesterday there were three crashes in a row when it sync'd with the blockchain. Except for the version line, I only include the part of the log from end of sync to end of crash.

crash_after_sync_crash1.txt
crash_after_sync_crash2.txt
crash_after_sync_crash3.txt

getinfo output

Here's the version on all three:

2018-03-01T22:10:23.390Z lightningd(7188): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77
2018-03-01T23:31:05.746Z lightningd(15417): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77
2018-03-02T00:52:44.465Z lightningd(12189): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77

[Ulmo]

"owned by none" definitely immediately preceeds all of those crashes:

egrep -n -e 'owned by none' -e FATAL -e start log.12.14989.0
112:2018-03-01T22:10:23.390Z lightningd(7188): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77
...
19897:2018-03-01T23:30:45.255Z lightningd(7188): 0347469dea682bd59255a5e60eaed8079186f85b5b5ea6dfd24565212e474b30c6 chan #98: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
19901:2018-03-01T23:30:45.256Z lightningd(7188): 03204b29fe80bbdb895b50f40fb81a78ccc53f2de8eca6bb234facffdbf783ec96 chan #28: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
19908:2018-03-01T23:30:45.259Z lightningd(7188): 024d2a7636c331eb2b9216ee40405e105f0b887479517f10b6f3cd0cf87ab280fa chan #67: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
19919:2018-03-01T23:30:45.265Z lightningd(7188): 02eccebd9ed98f6d267080a58194dbe554a2b33d976eb95bb7c116d00fd64c4a13 chan #6: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
19930:2018-03-01T23:30:45.268Z lightningd(7188): 03b7ca940bc33b882dc1f1bee353a6cf205b1a7472d8ae24d45370a8f510c27d23 chan #5: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
19937:2018-03-01T23:30:45.273Z lightningd(7188): 02aeb693e2dfb2f890c65dcf27e6eb7d52dd3e4f7b74ce5fe646675793625f9b51 chan #3: Funding tx announce ready, but channel state CHANNELD_SHUTTING_DOWN owned by none
19943:2018-03-01T23:30:45.274Z lightningd(7188): FATAL SIGNAL 11 RECEIVED

20081:2018-03-01T23:31:05.746Z lightningd(15417): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77
...
40227:2018-03-02T00:52:23.997Z lightningd(15417): 0347469dea682bd59255a5e60eaed8079186f85b5b5ea6dfd24565212e474b30c6 chan #98: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
40230:2018-03-02T00:52:23.999Z lightningd(15417): FATAL SIGNAL 11 RECEIVED

40368:2018-03-02T00:52:44.465Z lightningd(12189): Server started with public key 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a, alias Plasma???Genesis (color #031c52) and lightningd v0.5.2-2016-11-21-2158-g9504a77
...
60979:2018-03-02T02:13:58.662Z lightningd(12189): 03204b29fe80bbdb895b50f40fb81a78ccc53f2de8eca6bb234facffdbf783ec96 chan #28: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
60998:2018-03-02T02:13:58.668Z lightningd(12189): 03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82 chan #107: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
61003:2018-03-02T02:13:58.669Z lightningd(12189): 02dcdd7e18b80dc7fd51af34e871901c13f574e4b82325dc74c0beeada5771e8e9 chan #88: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
61010:2018-03-02T02:13:58.672Z lightningd(12189): 02eccebd9ed98f6d267080a58194dbe554a2b33d976eb95bb7c116d00fd64c4a13 chan #6: Funding tx announce ready, but channel state CHANNELD_NORMAL owned by none
61013:2018-03-02T02:13:58.673Z lightningd(12189): FATAL SIGNAL 11 RECEIVED

[ZmnSCPxj]

Duplicates: #1114

"owned by none" seems to be normal if the counterparty is not connected, although @rustyrussell should probably be able to correct me there. I think the crash is when we notice that the funding tx confirmed, but the channel is not in AWAITING_LOCKIN state, and we want to report it but cannot due to this crash bug.

[ZmnSCPxj]

If you do not mind, could you run the lightningd under gdb and see what channel->owner and channel->owner->name are?

[cdecker]

Yes, "owned by none" simply means that we don't have a connection with the peer yet. Owner generally refers to the daemon in charge of the connection, so if there is no connection we can't inform the owner of state changes.

[Ulmo]

Ok, I'll restart it with gdb.

[ZmnSCPxj]

1164 is likely to have fixed this: this is a duplicate of #1114 which is fixed by #1164.

[cdecker]

@Ulmo can you confirm this was fixed?