Lnd: LND crashed (panic) during QueryRoutes / SendToRouteSync

Created on 12 Nov 2019  路  15Comments  路  Source: lightningnetwork/lnd

Background

I had intense code executions QueryRoutes / SendToRouteSync with big lists of ignored_pairs & ignored_nodes. One from LNDs (lnd-01) was crashed

Your environment

  • version of lnd - 0.8.0-beta

Steps to reproduce

May be intense rebalancing through QueryRoutes (with ignored_pairs & ignored_nodes) -> SendToRouteSync executions.

stacktrace:

2019-11-12 12:53:08.431 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:08.819 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.160 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.694 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.823 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.453 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.505 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.878 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.944 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.204 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.723 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.912 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.307 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.447 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.921 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.926 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.482 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.620 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.701 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.789 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:14.545 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
panic: runtime error: index out of range [20] with length 20

goroutine 8943415 [running]:
github.com/lightningnetwork/lnd/routing/route.(*Route).ToSphinxPath(0xc00648a140, 0x1da7740, 0xc010fbda00, 0x20)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/route/route.go:306 +0x53a
github.com/lightningnetwork/lnd/routing.generateSphinxPacket(0xc00648a140, 0xc0097def3c, 0x20, 0x20, 0xc01e500030, 0x0, 0xc00bede880, 0xf26542, 0xfc5f00, 0xc0000386f8, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1494 +0x4b
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).createNewPaymentAttempt(0xc0085c4f00, 0x40, 0x40, 0xc001034b80, 0xc01281b278, 0x52e342)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:262 +0x475
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).resumePayment(0xc0085c4f00, 0x0, 0x0, 0x0, 0x0, 0x1, 0xb4e4ac, 0xc000010120)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:52 +0xd0a
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).sendPayment(0xc0004d0140, 0x0, 0xc0097def00, 0x14d6860, 0xc007ab2b40, 0x0, 0x0, 0x0, 0x0, 0x1151b00, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1812 +0x1d7
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).SendToRoute(0xc0004d0140, 0x58ca5384bd9010e2, 0xc79742f1db16e50, 0x64bf8000ad875178, 0x7e996cf4e75018da, 0xc00648a140, 0x0, 0x0, 0x0, 0x0, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1724 +0x20f
github.com/lightningnetwork/lnd.(*rpcServer).dispatchPaymentIntent(0xc0001f8160, 0xc018f88420, 0x0, 0x0, 0x0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3172 +0xac
github.com/lightningnetwork/lnd.(*rpcServer).sendPaymentSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc01fefe200, 0x0, 0xc005b21700, 0x1514d40)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3412 +0x11f
github.com/lightningnetwork/lnd.(*rpcServer).SendToRouteSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a050, 0xc0001f8160, 0xd26625, 0xc001652360)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3387 +0x9c
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x0, 0x0, 0x10, 0xc001898000)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11745 +0x89
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1199ba0, 0xc015510001, 0xc00dd82180, 0xc005b21920)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 +0x114
github.com/lightningnetwork/lnd.errorLogUnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0xc00fa51ba0, 0xc01c86c010, 0xc015510000, 0xc00648a0a0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/log.go:248 +0x81
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1, 0x1, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0x9c
github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0x0, 0xc001c04100, 0xc005b21a10, 0x40e6a8)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/macaroons/service.go:124 +0x150
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00fa51b60, 0xc0117a3a80, 0x5aa858, 0x11344e0, 0xc007ab2750)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 +0x155
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler(0x11eec80, 0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a000, 0xc0014dc960, 0x14f5c40, 0xc007ab2750, 0xc01001a000, 0x85d)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11747 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0xc0014dcd50, 0x1d80960, 0x0, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/[email protected]/server.go:966 +0x46a
google.golang.org/grpc.(*Server).handleStream(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/[email protected]/server.go:1245 +0xd16
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00abd0d70, 0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100)
        /home/bitcoin/go/pkg/mod/google.golang.org/[email protected]/server.go:685 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/bitcoin/go/pkg/mod/google.golang.org/[email protected]/server.go:683 +0xa1
bug crash multi-hop onion routing
Source
picture LNBIG-COM

All 15 comments

After restart i have many like these messages. Is it very bad? Possible loss of funds in this case? Or will everything come back with time?

2019-11-12 13:07:20.640 [INF] LNWL: Done catching up block hashes
2019-11-12 13:07:20.640 [INF] LNWL: Finished rescan for 212915 addresses (synced to block 0000000000000000000e16bedab7553cdbf4fcb007c87ab2f9eec9acec66f0de, height 603458)
2019-11-12 13:07:21.537 [INF] LTND: Chain backend is fully synced (end_height=603458)!
2019-11-12 13:07:21.542 [INF] PRNF: PeerNotifier starting
2019-11-12 13:07:21.617 [INF] HSWC: Starting HTLC Switch
2019-11-12 13:07:21.617 [INF] NTFN: New block epoch subscription
2019-11-12 13:07:21.617 [INF] NTFN: New block epoch subscription
2019-11-12 13:07:22.333 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574243:574:0, HTLC ID = 27
2019-11-12 13:07:22.333 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574243:574:0, HTLC ID = 27
2019-11-12 13:07:22.485 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:672:1, HTLC ID = 416
2019-11-12 13:07:22.485 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:672:1, HTLC ID = 416
2019-11-12 13:07:22.497 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 566066:1882:0, HTLC ID = 80
2019-11-12 13:07:22.497 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 566066:1882:0, HTLC ID = 80
2019-11-12 13:07:22.503 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2702:0, HTLC ID = 107
2019-11-12 13:07:22.512 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2702:0, HTLC ID = 107
2019-11-12 13:07:22.989 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550423:10:0, HTLC ID = 396
2019-11-12 13:07:22.989 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550423:10:0, HTLC ID = 396
2019-11-12 13:07:23.019 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 586988:500:1, HTLC ID = 49
2019-11-12 13:07:23.019 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 586988:500:1, HTLC ID = 49
2019-11-12 13:07:24.423 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2728:0, HTLC ID = 11
2019-11-12 13:07:24.423 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2728:0, HTLC ID = 11
2019-11-12 13:07:25.521 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:586:0, HTLC ID = 0
2019-11-12 13:07:25.521 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:586:0, HTLC ID = 0
2019-11-12 13:07:25.564 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 108
2019-11-12 13:07:25.564 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 182
2019-11-12 13:07:25.564 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 108
2019-11-12 13:07:25.564 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 182
2019-11-12 13:07:26.403 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 586988:489:1, HTLC ID = 155
2019-11-12 13:07:26.403 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 586988:489:1, HTLC ID = 155
2019-11-12 13:07:26.439 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550739:2187:0, HTLC ID = 382
2019-11-12 13:07:26.439 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550739:2187:0, HTLC ID = 382
2019-11-12 13:07:26.469 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567233:1602:0, HTLC ID = 59
2019-11-12 13:07:26.469 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567233:1602:0, HTLC ID = 59
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 141
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 142
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 141
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 142
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 175
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 175
2019-11-12 13:07:27.299 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:669:1, HTLC ID = 123
2019-11-12 13:07:27.299 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:669:1, HTLC ID = 123
2019-11-12 13:07:27.328 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567206:952:1, HTLC ID = 145
2019-11-12 13:07:27.328 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567206:952:1, HTLC ID = 145
2019-11-12 13:07:28.909 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2284:0, HTLC ID = 237
2019-11-12 13:07:28.909 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2284:0, HTLC ID = 237
2019-11-12 13:07:28.928 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 70
2019-11-12 13:07:28.928 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 81
2019-11-12 13:07:28.928 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 70
2019-11-12 13:07:28.928 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 81
2019-11-12 13:07:28.965 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550333:1519:0, HTLC ID = 33
2019-11-12 13:07:28.965 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550333:1519:0, HTLC ID = 33
2019-11-12 13:07:28.997 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 13
2019-11-12 13:07:28.997 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 14
2019-11-12 13:07:28.997 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 13
2019-11-12 13:07:28.997 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 14
2019-11-12 13:07:29.212 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550330:685:0, HTLC ID = 220
2019-11-12 13:07:29.212 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550330:685:0, HTLC ID = 220
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 30
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 38
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 30
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 38
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 45
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 45
2019-11-12 13:07:29.665 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 558535:1606:1, HTLC ID = 21
2019-11-12 13:07:29.665 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 558535:1606:1, HTLC ID = 21
2019-11-12 13:07:30.105 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 88
2019-11-12 13:07:30.105 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 94
2019-11-12 13:07:30.105 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 88
2019-11-12 13:07:30.105 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 94
2019-11-12 13:07:31.389 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 210
2019-11-12 13:07:31.389 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 293
2019-11-12 13:07:31.389 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 210
2019-11-12 13:07:31.389 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 293
LNBIG-COM picture LNBIG-COM on 12 Nov 2019

panic: runtime error: index out of range [20] with length 20

Was a route with more than 20 hops provided?

After restart i have many like these messages. Is it very bad? Possible loss of funds in this case? Or will everything come back with time?

They can be ignored. We had a recent change that caused these to be logged without it actually being a fatal error, but we've yet to address that (see https://github.com/lightningnetwork/lnd/issues/3656).

wpaulino picture wpaulino on 13 Nov 2019

Yes, sounds like max hops exceeded. We need to fix that. From the code

// NumMaxHops is the maximum path length. This should be set to an estimate of
// the upper limit of the diameter of the node graph.
//
// TODO(roasbeef): adjust due to var-payloads?
const NumMaxHops = 20
joostjager picture joostjager on 13 Nov 2019
👍1

panic: runtime error: index out of range [20] with length 20

Was a route with more than 20 hops provided?

I no longer have detailed logs from my scripts about this incident. But may be...
But if this is so, this is not a reason to crash ;-)

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

Do you append hops to the result from QueryRoutes? Because the pure QueryRoutes result should never be longer than 20 hops. I think that you append your own node to complete the circular route?

Still no reason to crash :)

joostjager picture joostjager on 13 Nov 2019

Do you append hops to the result from QueryRoutes?

No

Because the pure QueryRoutes result should never be longer than 20 hops. I think that you append your own node to complete the circular route?

Still no reason to crash :)

After fail payment i add one failed direct way (parsed from error of SendToRouteSync - ShortChannelId) of edge to ignored_pairs but if one node does more fails i add this node to ignored_nodes. And the ignore_pairs consists before start all channels from LNBIG (excluding two channels that need to be rebalanced). Due to these big lists i think yes, it happens. I don't have other way to find paths from one LNBIG channel to other LNBIG channel through outside.

I think this bug should be fixed otherwise every time of rebalancing i will repair LND and write here new logs ;-)

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

The code there i caught this crash is here (NodeJS):

https://gist.github.com/LNBIG-COM/90f143c8ba61b848eb33dc4253a90425

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

I think you do append in that code?
route.hops.push

Side note: I see your recalculateRoute. An alternative for that is to use the new BuildRoute rpc

joostjager picture joostjager on 13 Nov 2019

I think you do append in that code?
route.hops.push

First, I do not calculate the complete route, but to the N-1 node, then I add the last hop (receiver). Without this, it is impossible to calculate a route for rebalancing - for example, when node A wants to find a route to node A (but to a different channel) - without this lnd will say that there are no routes. I wrote about this here: https://github.com/lightningnetwork/lnd/issues/3689

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

An alternative for that is to use the new BuildRoute rpc

May be. I don't know about this. This function is missed in lnd-0.8.0-beta

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

Yes, I understand that that is necessary. But I also think that pushes the route length to 21 hops. But issue is clear, I will think about a solution. There is also no way currently to limit the max route length with queryroutes. Otherwise you could set it to 19 and still be able to add the last hop manually.

joostjager picture joostjager on 13 Nov 2019

But I also think that pushes the route length to 21 hops.

Ohh, may be. Thanks for this info. I didn't think about this ;-)

Otherwise you could set it to 19 and still be able to add the last hop manually.

I will do! Thanks!

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

Currently, you unfortunately cannot set the max number of hops on a queryroutes call.

But at least you can check whether the route returned from queryroutes is 20 hops. If that is the case, don't fire SendToRoute if you don't want to crash the node

joostjager picture joostjager on 13 Nov 2019
👍1

The limitation is already there and while the flight is normal :)

LNBIG-COM picture LNBIG-COM on 13 Nov 2019

Related issue: https://github.com/lightningnetwork/lnd/issues/3725

joostjager picture joostjager on 14 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Richard87 picture Richard87  路  3Comments
stevenroose picture stevenroose  路  3Comments
whizz picture whizz  路  3Comments
alexbosworth picture alexbosworth  路  3Comments
pm47 picture pm47  路  3Comments