Lightning: `gossipd` is unable to read gossip messages from `gossip_store`

[SimonVrouwe]

Issue and Steps to Reproduce

I just build and installed latest master "version" : "v0.7.0-248-g4985693" and restart my mainnet node (was f3c2345). It crashed on first start (see below), but 2th time was fine.

It is not really an issue, but I thought I report it.

2019-04-17T05:48:55.657Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Creating commit_sig signature 1943 30450221009844d8662bec47e55baf4575f5c998b9eff48b18ef699077653f67cfc1c4343202201dbf9154eab476f00162faf959b65704fb793388e02bf7e99568a628d399891401 for tx 02000000014507fe137482c2cd96603e1c67691449d3e15e3ea8b6983f8b6b9de06315c6ce00000000001781548002f7fa5b00000000002200204874864f76eb1a7eca36fc875d9c484029277203bb2387ddf0e986ec5732409747c089000000000016001404d57a85039ecbd5f80da283c57aa329b527f6abeef98f20 wscript 5221021e00c35a3e97deaf6ff417333f6890f7903035d9f89d2b3598c3c189775eb99321035689869536ba6a4cf1d285656e75cccf5ff600be519c7981317884e6819d4dda52ae key [redacted]
2019-04-17T05:48:55.657Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Telling master we're about to commit...
2019-04-17T05:48:55.657Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Sending master 1020
2019-04-17T05:48:55.657Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: UPDATE WIRE_CHANNEL_SENDING_COMMITSIG
2019-04-17T05:48:55.755Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: ... , awaiting 1120
2019-04-17T05:48:55.755Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Got it!
2019-04-17T05:48:55.755Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Sending commit_sig with 0 htlc sigs
2019-04-17T05:48:55.755Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: peer_out WIRE_COMMITMENT_SIGNED
2019-04-17T05:48:55.801Z lightningd(31536): Adding block 571971: 0000000000000000000d27f0d54baa76b7eb8413535565b7c2fd4e16ca5cd5bb
2019-04-17T05:48:56.298Z lightning_gossipd(31553): Received channel_announcement for channel 567242x1265x0
2019-04-17T05:48:56.298Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: peer_in WIRE_REVOKE_AND_ACK
2019-04-17T05:48:56.299Z lightning_gossipd(31553): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-04-17T05:48:56.300Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Received revoke_and_ack
2019-04-17T05:48:56.300Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Applying feerate 12654 to LOCAL
2019-04-17T05:48:56.300Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Commits outstanding after recv revoke_and_ack
lightning_gossipd: gossip_store: can't read hdr offset 2333, store len 2483: Success (version v0.7.0-248-g4985693)
2019-04-17T05:48:56.300Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Sending master 1022
2019-04-17T05:48:56.300Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: UPDATE WIRE_CHANNEL_GOT_REVOKE
2019-04-17T05:48:56.300Z lightningd(31536): 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: got revoke 1942: 0 changed
0x55d904115a54 send_backtrace
        common/daemon.c:40
0x55d90411a718 status_failed
        common/status.c:173
0x55d9041054f1 gossip_store_get
        gossipd/gossip_store.c:439
0x55d904108c9d routing_add_channel_announcement
        gossipd/routing.c:953
0x55d904109764 handle_pending_cannouncement
        gossipd/routing.c:1235
0x55d90411263e handle_txout_reply
        gossipd/gossipd.c:2643
0x55d904112bd5 recv_req
        gossipd/gossipd.c:2824
0x55d904115f5e handle_read
        common/daemon_conn.c:31
0x55d9041388e8 next_plan
        ccan/ccan/io/io.c:59
0x55d904139465 do_plan
        ccan/ccan/io/io.c:407
0x55d9041394a3 io_ready
        ccan/ccan/io/io.c:417
0x55d90413b4f2 io_loop
        ccan/ccan/io/poll.c:445
0x55d904112e8f main
        gossipd/gossipd.c:2934
0x7f8c71d522e0 ???
        ???:0
0x55d9040ffd39 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2019-04-17T05:48:56.433Z lightning_gossipd(31553): gossip_store: can't read hdr offset 2333, store len 2483: Success (version v0.7.0-248-g4985693)
2019-04-17T05:48:56.433Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: ... , awaiting 1122
2019-04-17T05:48:56.433Z lightning_gossipd(31553): backtrace: common/daemon.c:45 (send_backtrace) 0x55d904115aaa
2019-04-17T05:48:56.433Z lightning_gossipd(31553): backtrace: common/status.c:173 (status_failed) 0x55d90411a718
2019-04-17T05:48:56.433Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Got it!
2019-04-17T05:48:56.433Z lightning_gossipd(31553): backtrace: gossipd/gossip_store.c:439 (gossip_store_get) 0x55d9041054f1
2019-04-17T05:48:56.433Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: revoke_and_ack LOCAL: remote_per_commit = 0343a971f42f1a45bad1b6ee1f0a7421d750bcf7b0d332952896c311620df208eb, old_remote_per_commit = 03b5ddab0c41e0007445e4af82afb255e3563bb60c74dd2748069e41f2745e8e23
2019-04-17T05:48:56.433Z lightning_gossipd(31553): backtrace: gossipd/routing.c:953 (routing_add_channel_announcement) 0x55d904108c9d
2019-04-17T05:48:56.433Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Trying commit
2019-04-17T05:48:56.433Z lightning_gossipd(31553): backtrace: gossipd/routing.c:1235 (handle_pending_cannouncement) 0x55d904109764
2019-04-17T05:48:56.434Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Can't send commit: nothing to send
2019-04-17T05:48:56.434Z lightning_gossipd(31553): backtrace: gossipd/gossipd.c:2643 (handle_txout_reply) 0x55d90411263e
2019-04-17T05:48:56.434Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: peer_in WIRE_COMMITMENT_SIGNED
2019-04-17T05:48:56.435Z lightning_gossipd(31553): backtrace: gossipd/gossipd.c:2824 (recv_req) 0x55d904112bd5
2019-04-17T05:48:56.435Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Received commit
2019-04-17T05:48:56.436Z lightning_gossipd(31553): backtrace: common/daemon_conn.c:31 (handle_read) 0x55d904115f5e
2019-04-17T05:48:56.436Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Derived key [redacted] from basepoint [redacted], point 0298bbdb59bf61aee1e83574c0444328b9fd5d1a02cf6a4847dad83c40dfef08d2
2019-04-17T05:48:56.437Z lightning_gossipd(31553): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55d9041388e8
2019-04-17T05:48:56.437Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Received commit_sig with 0 htlc sigs
2019-04-17T05:48:56.437Z lightning_gossipd(31553): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55d904139465
2019-04-17T05:48:56.437Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Sending master 1021
2019-04-17T05:48:56.437Z lightning_gossipd(31553): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55d9041394a3
2019-04-17T05:48:56.437Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: UPDATE WIRE_CHANNEL_GOT_COMMITSIG
2019-04-17T05:48:56.437Z lightningd(31536): 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: got commitsig 1943: feerate 12654, 0 added, 0 fulfilled, 0 failed, 0 changed
2019-04-17T05:48:56.581Z lightningd(31536): Adding block 571972: 00000000000000000016419c141945fee0c742cb86c2c640c0a0eb72a140ebe2
2019-04-17T05:48:56.910Z lightning_gossipd(31553): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55d90413b4f2
2019-04-17T05:48:56.910Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: ... , awaiting 1121
2019-04-17T05:48:56.912Z lightning_gossipd(31553): backtrace: gossipd/gossipd.c:2934 (main) 0x55d904112e8f
2019-04-17T05:48:56.912Z lightning_hsmd(31551): Client: Received message 18 from client
2019-04-17T05:48:56.912Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Got it!
2019-04-17T05:48:56.912Z lightning_gossipd(31553): backtrace: (null):0 ((null)) 0x7f8c71d522e0
2019-04-17T05:48:56.913Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: Sending revoke_and_ack
2019-04-17T05:48:56.913Z lightning_gossipd(31553): backtrace: (null):0 ((null)) 0x55d9040ffd39
2019-04-17T05:48:56.913Z lightningd(31536): lightning_channeld-030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f chan #538: peer_out WIRE_REVOKE_AND_ACK
2019-04-17T05:48:56.913Z lightning_gossipd(31553): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-04-17T05:48:56.913Z lightning_gossipd(31553): STATUS_FAIL_INTERNAL_ERROR: gossip_store: can't read hdr offset 2333, store len 2483: Success
lightningd: lightning_gossipd failed (exit status 242), exiting.

[ghost]

I confirm (error, second time fine) but there is a more serious issue here. After every restart (of the same version) the gossip_store is emptied and has to be rebuild, which consumes a lot of cpu. @SimonVrouwe: can you confirm?

"version" : "v0.7.0-262-g0fc4241",

[m-schmoock]

I can also confirm this, didn't had the time to look into it yet, but the issue exist on master for at least one week.

[ScottTre]

Did you also see a increased memory usage from the gossip process like here: https://github.com/ElementsProject/lightning/issues/2574 ?

[ghost]

@ScottTre I don't see that (except while c-lightning is still building the gossip_store (~30 MB)). Have you tried to update to the latest version, delete the gossip_store by hand and restart?

[m-schmoock]

Hey, to track this down I attached a testnet gossip store that can reproduce the issue 100%.
Just copy it in your .lightning folder and start the daemon: gossip_store.zip

lightning_gossipd: gossip_store: can't read hdr offset 3521818, store len 3523408: Success (version v0.7.0-276-g62e1423)
0x56086348309c send_backtrace
    common/daemon.c:40
0x560863487d7b status_failed
    common/status.c:173
0x560863471fc2 gossip_store_get
    gossipd/gossip_store.c:439
0x56086347966a next_broadcast
    gossipd/broadcast.c:116
0x56086347c6d7 maybe_queue_gossip
    gossipd/gossipd.c:1218
0x56086347c781 dump_gossip
    gossipd/gossipd.c:1256
0x560863483638 daemon_conn_write_next
    common/daemon_conn.c:52
...

[SimonVrouwe]

Can confirm gossip_store is emptied after restart and also crashed my node:

2019-04-30T12:28:26.150Z lightning_gossipd(24346): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-04-30T12:28:26.151Z lightning_gossipd(24346): Received channel_update for channel 564195x1570x1/0 now ACTIVE (from pending update)
2019-04-30T12:28:26.151Z lightning_gossipd(24346): Received channel_update for channel 564195x1570x1/1 now ACTIVE (from pending update)
2019-04-30T12:28:26.151Z lightning_gossipd(24346): Updated pending announce with update 550742x846x1/0
2019-04-30T12:28:26.151Z lightning_gossipd(24346): Updated pending announce with update 550742x846x1/1
2019-04-30T12:28:26.151Z lightning_gossipd(24346): Received channel_announcement for channel 573803x1780x0
2019-04-30T12:28:26.151Z lightning_gossipd(24346): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-04-30T12:28:26.151Z lightning_gossipd(24346): gossip_store: bad checksum offset 55820, store len 4517764 (version v0.7.0-248-g4985693)
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: common/daemon.c:45 (send_backtrace) 0x555590ec6aaa
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: common/status.c:173 (status_failed) 0x555590ecb718
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: gossipd/gossip_store.c:455 (gossip_store_get) 0x555590eb65cb
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: gossipd/broadcast.c:116 (next_broadcast) 0x555590ebd03f
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: gossipd/gossipd.c:1218 (maybe_queue_gossip) 0x555590ec00a1
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: gossipd/gossipd.c:1256 (dump_gossip) 0x555590ec014b
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: common/daemon_conn.c:52 (daemon_conn_write_next) 0x555590ec6ff0
2019-04-30T12:28:26.151Z lightning_gossipd(24346): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x555590ee98e8
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: ccan/ccan/io/io.c:435 (io_do_always) 0x555590eea56e
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: ccan/ccan/io/poll.c:300 (handle_always) 0x555590eebfdc
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: ccan/ccan/io/poll.c:377 (io_loop) 0x555590eec2b4
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: gossipd/gossipd.c:2934 (main) 0x555590ec3e8f
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: (null):0 ((null)) 0x7f312ad452e0
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: (null):0 ((null)) 0x555590eb0d39
2019-04-30T12:28:26.152Z lightning_gossipd(24346): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-04-30T12:28:26.152Z lightning_gossipd(24346): STATUS_FAIL_INTERNAL_ERROR: gossip_store: bad checksum offset 55820, store len 4517764
lightningd: lightning_gossipd failed (exit status 242), exiting.

updating node to latest f99c461 and restoring gossip_store with a backup gives same result, but now crashes differently

2019-04-30T12:34:06.574Z lightning_gossipd(24859): Received node_announcement for node 02682020a37a3b5876b16bffbf0ac56d7f379695a5d273cf16ed0f35f18ab2dc98
2019-04-30T12:34:06.574Z lightning_gossipd(24859): Received channel_update for channel 567016x918x0/1 now ACTIVE (from pending update)
2019-04-30T12:34:06.574Z lightning_gossipd(24859): gossip_store: can't read hdr offset 15922775, store len 15923086: Success (version v0.7.0-279-gf99c461)
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: common/daemon.c:45 (send_backtrace) 0x55b79e744702
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: common/status.c:173 (status_failed) 0x55b79e749370
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossip_store.c:439 (gossip_store_get) 0x55b79e7334f1
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:458 (get_node_announcement) 0x55b79e73bfb8
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:492 (get_node_announcement_by_id) 0x55b79e73c18b
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:505 (node_announcement_redundant) 0x55b79e73c1db
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:544 (maybe_send_own_node_announce) 0x55b79e73c3e1
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:2647 (handle_txout_reply) 0x55b79e7412a2
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:2824 (recv_req) 0x55b79e74182d
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: common/daemon_conn.c:31 (handle_read) 0x55b79e744bb6
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55b79e767540
2019-04-30T12:34:06.574Z lightning_gossipd(24859): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55b79e7680bd
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55b79e7680fb
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55b79e76a14a
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: gossipd/gossipd.c:2934 (main) 0x55b79e741ae7
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: (null):0 ((null)) 0x7f65b3c142e0
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: (null):0 ((null)) 0x55b79e72dd39
2019-04-30T12:34:06.575Z lightning_gossipd(24859): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-04-30T12:34:06.575Z lightning_gossipd(24859): STATUS_FAIL_INTERNAL_ERROR: gossip_store: can't read hdr offset 15922775, store len 15923086: Success
lightning_connectd: Failed parsing msg gossipctl:  (version v0.7.0-279-gf99c461)
lightningd: lightning_gossipd failed (exit status 242), exiting.
0x5654f930d0d2 send_backtrace
        common/daemon.c:40
0x5654f9311994 status_failed
        common/status.c:173
0x5654f93035d8 get_gossipfd
        connectd/connectd.c:312
0x5654f930392f peer_connected
        connectd/connectd.c:419
0x5654f9306d6a peer_init_received
        connectd/peer_exchange_initmsg.c:89
0x5654f933253c next_plan
        ccan/ccan/io/io.c:59
0x5654f93330b9 do_plan
        ccan/ccan/io/io.c:407
0x5654f93330f7 io_ready
        ccan/ccan/io/io.c:417
0x5654f9335146 io_loop
        ccan/ccan/io/poll.c:445
0x5654f9306a99 main
        connectd/connectd.c:1530
0x7fc4a72a42e0 ???
        ???:0
0x5654f9300db9 ???
        ???:0
0xffffffffffffffff ???
        ???:0

[darosior]

I've been testing with @m-schmoock gossip_store and from 0.7 to cb297b0a1b97a39b67984cf6ffbd9b3a80ef348b I only get a

lightning_gossipd(30472): gossip_store: Bad node_announcement (1002008e01019442e1d1afaf19e9d510dc25c58b75dba5ee08758a650b4af82e335326e0a6cc70bd2d9fbc253c8c1a062b2307acc02b72e28a5ff42826e7d4857d8cb6e4626600005be0654f031201e62297a420a3878d0d8b7c4206553d354097da1a9e7c34158303d92235690312014841505059424147454c000000000000000000000000000000000000000000000000) truncating to 1

without a crash. But at 0608c363013538831d53f208ef8fb2d213ffd6b7 I get the crash (same as above). Thus it might be related to this commit.

Weird thing is it happens only with one of my test directory (I made plenty for each db version so that I can go back in commits), after some testing even with the same gossip_store file I cannot reproduce the error.

[rustyrussell]

Indeed. I've fixed this bug, which was trivial to find thanks to @m-schmoock gossip_store.

And of course, it was my fault in the first place :(

[rustyrussell]

FWIW, you can always simply delete or move ~/.lightning/gossip_store and restart. It will download from peers again.

[SimonVrouwe]

Unfortunate the problem with gossip_store being emptied at restart still seems the be there, now running bc5dbb6. I thought it was solved with #2609 because it didn't crash back then, but I forgot to actually check gossip_store file persistence.

My setup also has electrs server connected to the same bitcoind, this usually goes without problem, but this now probably causes the crash (see below). Temporarily disabling electrs during startup (at least) does not make c-lightning crash. I left the crash log below, but the main problem (I think) is emptying of the gossip_store file.

lightning_gossipd: FATAL SIGNAL 6 (version v0.7.0-310-gbc5dbb6)
2019-05-05T12:33:25.723Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
0x56372c4f26c3 send_backtrace
        common/daemon.c:40
0x56372c4f2769 crashdump
        common/daemon.c:53
0x7fc66f53305f ???
        ???:0
0x7fc66f532fff ???
        ???:0
0x7fc66f534429 ???
        ???:0
0x56372c4e8a8b add_broadcast
        gossipd/broadcast.c:53
0x56372c4e8ac5 insert_broadcast_nostore
        gossipd/broadcast.c:60
0x56372c4e8b98 insert_broadcast
        gossipd/broadcast.c:83
0x56372c4e5715 add_channel_announce_to_broadcast
        gossipd/routing.c:1339
0x56372c4e6a5d routing_add_channel_update
        gossipd/routing.c:1837
0x56372c4e709a handle_channel_update
        gossipd/routing.c:1989
2019-05-05T12:33:25.725Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
0x56372c4ea4db handle_channel_update_msg
        gossipd/gossipd.c:585
0x56372c4eca02 peer_msg_in
        gossipd/gossipd.c:1603
0x56372c4f2bcd handle_read
        common/daemon_conn.c:31
0x56372c5156ef next_plan
        ccan/ccan/io/io.c:59
0x56372c51626c do_plan
        ccan/ccan/io/io.c:407
0x56372c5162aa io_ready
        ccan/ccan/io/io.c:417
2019-05-05T12:33:25.726Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
0x56372c5182f9 io_loop
        ccan/ccan/io/poll.c:445
0x56372c4efafe main
        gossipd/gossipd.c:2934
0x7fc66f5202e0 ???
        ???:0
0x56372c4dbd39 ???
        ???:0
0xffffffffffffffff ???
        ???:0
lightning_gossipd: FATAL SIGNAL (version v0.7.0-310-gbc5dbb6)
0x56372c4f26c3 send_backtrace
        common/daemon.c:40
0x56372c4f7461 status_failed
        common/status.c:193
0x56372c4f76b3 status_backtrace_exit
        common/subdaemon.c:25
0x56372c4f2772 crashdump
        common/daemon.c:56
0x7fc66f53305f ???
        ???:0
0x7fc66f532fff ???
        ???:0
0x7fc66f534429 ???
        ???:0
0x56372c4e8a8b add_broadcast
        gossipd/broadcast.c:53
0x56372c4e8ac5 insert_broadcast_nostore
        gossipd/broadcast.c:60
0x56372c4e8b98 insert_broadcast
        gossipd/broadcast.c:83
0x56372c4e5715 add_channel_announce_to_broadcast
        gossipd/routing.c:1339
0x56372c4e6a5d routing_add_channel_update
        gossipd/routing.c:1837
0x56372c4e709a handle_channel_update
        gossipd/routing.c:1989
0x56372c4ea4db handle_channel_update_msg
        gossipd/gossipd.c:585
0x56372c4eca02 peer_msg_in
        gossipd/gossipd.c:1603
0x56372c4f2bcd handle_read
        common/daemon_conn.c:31
0x56372c5156ef next_plan
        ccan/ccan/io/io.c:59
0x56372c51626c do_plan
        ccan/ccan/io/io.c:407
0x56372c5162aa io_ready
        ccan/ccan/io/io.c:417
0x56372c5182f9 io_loop
        ccan/ccan/io/poll.c:445
0x56372c4efafe main
        gossipd/gossipd.c:2934
0x7fc66f5202e0 ???
        ???:0
0x56372c4dbd39 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2019-05-05T12:33:25.727Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-05-05T12:33:25.727Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
...
2019-05-05T12:33:25.778Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-05-05T12:33:25.778Z lightning_gossipd(14125): UPDATE WIRE_GOSSIP_GET_TXOUT
2019-05-05T12:33:25.778Z lightning_gossipd(14125): FATAL SIGNAL 6 (version v0.7.0-310-gbc5dbb6)
2019-05-05T12:33:25.778Z lightning_gossipd(14125): backtrace: common/daemon.c:45 (send_backtrace) 0x56372c4f2719
2019-05-05T12:33:25.778Z lightning_gossipd(14125): backtrace: common/daemon.c:53 (crashdump) 0x56372c4f2769
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f53305f
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f532fff
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f534429
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:53 (add_broadcast) 0x56372c4e8a8b
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:60 (insert_broadcast_nostore) 0x56372c4e8ac5
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:83 (insert_broadcast) 0x56372c4e8b98
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1339 (add_channel_announce_to_broadcast) 0x56372c4e5715
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1837 (routing_add_channel_update) 0x56372c4e6a5d
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1989 (handle_channel_update) 0x56372c4e709a
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:585 (handle_channel_update_msg) 0x56372c4ea4db
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:1603 (peer_msg_in) 0x56372c4eca02
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: common/daemon_conn.c:31 (handle_read) 0x56372c4f2bcd
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x56372c5156ef
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x56372c51626c
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x56372c5162aa
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x56372c5182f9
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:2934 (main) 0x56372c4efafe
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f5202e0
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x56372c4dbd39
2019-05-05T12:33:25.779Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-05-05T12:33:25.780Z lightning_gossipd(14125): FATAL SIGNAL (version v0.7.0-310-gbc5dbb6)
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: common/daemon.c:45 (send_backtrace) 0x56372c4f2719
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: common/status.c:193 (status_failed) 0x56372c4f7461
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: common/subdaemon.c:25 (status_backtrace_exit) 0x56372c4f76b3
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: common/daemon.c:56 (crashdump) 0x56372c4f2772
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f53305f
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f532fff
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f534429
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:53 (add_broadcast) 0x56372c4e8a8b
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:60 (insert_broadcast_nostore) 0x56372c4e8ac5
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/broadcast.c:83 (insert_broadcast) 0x56372c4e8b98
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1339 (add_channel_announce_to_broadcast) 0x56372c4e5715
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1837 (routing_add_channel_update) 0x56372c4e6a5d
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/routing.c:1989 (handle_channel_update) 0x56372c4e709a
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:585 (handle_channel_update_msg) 0x56372c4ea4db
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:1603 (peer_msg_in) 0x56372c4eca02
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: common/daemon_conn.c:31 (handle_read) 0x56372c4f2bcd
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x56372c5156ef
2019-05-05T12:33:25.780Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x56372c51626c
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x56372c5162aa
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x56372c5182f9
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: gossipd/gossipd.c:2934 (main) 0x56372c4efafe
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x7fc66f5202e0
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0x56372c4dbd39
2019-05-05T12:33:25.781Z lightning_gossipd(14125): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-05-05T12:33:25.781Z lightning_gossipd(14125): STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
lightningd: lightning_gossipd failed (exit status 242), exiting.

[cdecker]

This appears to be a separate issue. Could you report that as a new bug? It either is an out-of-memory error (unlikely), or we are attempting to add the same channel_update twice:

https://github.com/ElementsProject/lightning/blob/bc5dbb6a80f89210a25f98135db452ada5f14743/gossipd/broadcast.c#L52-L53

which fails due to this:

https://github.com/ElementsProject/lightning/blob/bc5dbb6a80f89210a25f98135db452ada5f14743/ccan/ccan/intmap/intmap.h#L156-L176

In other news @m-schmoock still encountered the issue with the offset. So we might end up reopening this issue.