Lnd: lncli listchannels hangs

[Technifocal]

Background

Having upgrades from 0.7.1-beta to 0.9.1-beta (via v0.8.0-beta-rc1 to apply mitigation 11), listchannels just hangs. getinfo works as expected:

$ lncli unlock                                              
Input wallet password:

lnd successfully unlocked!

$ lncli getinfo
{
    "version": "0.9.1-beta commit=",
    "identity_pubkey": "redacted",
    "alias": "redacted",
    "color": "redacted",
    "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 1,
    "num_peers": 0,
    "block_height": 622688,
    "block_hash": "00000000000000000003471a47be7701223e5974d514de52fb6bd8bde8b8cd30",
    "best_header_timestamp": "1584998835",
    "synced_to_chain": true,
    "synced_to_graph": false,
    "testnet": false,
    "chains": [
        {
            "chain": "bitcoin",
            "network": "mainnet"
        }
    ],
    "uris": [
        "redacted@redacted:redacted"
    ],
    "features": {
        "0": {
            "name": "data-loss-protect",
            "is_required": true,
            "is_known": true
        },
        "5": {
            "name": "upfront-shutdown-script",
            "is_required": false,
            "is_known": true
        },
        "7": {
            "name": "gossip-queries",
            "is_required": false,
            "is_known": true
        },
        "9": {
            "name": "tlv-onion",
            "is_required": false,
            "is_known": true
        },
        "13": {
            "name": "static-remote-key",
            "is_required": false,
            "is_known": true
        },
        "15": {
            "name": "payment-addr",
            "is_required": false,
            "is_known": true
        },
        "17": {
            "name": "multi-path-payments",
            "is_required": false,
            "is_known": true
        }
    }
}

$ lncli listchannels
#Hangs here

Your environment

Question|Answer
-|-
version of lnd|0.9.1-beta commit=
which operating system (uname -a on *Nix)|Linux hostname 4.19.102-1-MANJARO #1 SMP Wed Feb 5 19:48:44 UTC 2020 x86_64 GNU/Linux
version of btcd, bitcoind, or other backend|Bitcoin Core version v0.19.1.0-g58ba7c314d552cea8cb024960a8504577aee586f
any other relevant environment details|Upgrade route described above. If there's anything else needed feel free to ask.

Steps to reproduce

With my database:
lnd -> lncli unlock -> lncli listchannels.

Unsure how to reproduce for other users

Expected behaviour

lncli listchannels provides a response

Actual behaviour

lncli listchannels hangs

Log

with lnd --debuglevel=trace, the only thing that appears in the lnd stdout after running lncli listchannels is 2020-03-23 21:42:30.592 [DBG] RPCS: [listchannels] fetched 1 channels from DB

[alexbosworth]

Did you have a lot of invoices?

[Technifocal]

@alexbosworth I don't think so:

$ lncli listinvoices --max_invoices 9999999 | jq '.invoices | length'                                                                                                                                          
210

[Roasbeef]

Are you able to obtain a goroutine or CPU profile using go tool pprof? Also to confirm, there's nothing _at all_ in the logs on trace other than the list channels debug RPC statement?

[Technifocal]

I am very confused. Even though I had restarted lnd multiple times before making this Github issue (even rebooted my PC), attempting to get this information for you again seems to have resulted in it working perfectly fine (and taking only 78ms to get a result from lncli listchannels, so it wasn't even that I didn't wait long enough) with a lot more information being logged by lnd (even without --debuglevel). I haven't to the extent of my knowledge changed anything with my setup.

I wish I could document the reason for why it occurred prior. Sorry.

EDIT: actually, I did do one thing, I moved ~/.lnd somewhere else, ran lnd, killed lnd, then moved it back to where it originally was. Maybe there was some disk related error(?), but the files at no point left the filesystem so :man_shrugging:

[PierreRochard]

I'm getting a hang too on lncli listchannels. This node has one (old) channel

Here's what I think is the relevant part from http://localhost:9736/debug/pprof/goroutine?debug=1

2 @ 0x1030c80 0x104070b 0x1a5f8fc 0x1b493de 0x1b4895c 0x1a08df9 0x1a9cdc4 0x1b77c81 0x1a9cd4c 0x194c400 0x1a9cf75 0x19f152b 0x191a6fa 0x191e676 0x192a07b 0x105e891
#   0x1a5f8fb   github.com/lightningnetwork/lnd/chanfitness.(*ChannelEventStore).GetLifespan+0x13b  github.com/lightningnetwork/lnd@/chanfitness/chaneventstore.go:351
#   0x1b493dd   github.com/lightningnetwork/lnd.createRPCOpenChannel+0x7bd              github.com/lightningnetwork/lnd@/rpcserver.go:3085
#   0x1b4895b   github.com/lightningnetwork/lnd.(*rpcServer).ListChannels+0x30b             github.com/lightningnetwork/lnd@/rpcserver.go:2981
#   0x1a08df8   github.com/lightningnetwork/lnd/lnrpc._Lightning_ListChannels_Handler.func1+0x88    github.com/lightningnetwork/lnd@/lnrpc/rpc.pb.go:13112
#   0x1a9cdc3   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x113     github.com/grpc-ecosystem/[email protected]/chain.go:31
#   0x1b77c80   github.com/lightningnetwork/lnd.errorLogUnaryServerInterceptor.func1+0x80       github.com/lightningnetwork/lnd@/log.go:151
#   0x1a9cd4b   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x9b      github.com/grpc-ecosystem/[email protected]/chain.go:34
#   0x194c3ff   github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1+0x14f github.com/lightningnetwork/lnd@/macaroons/service.go:124
#   0x1a9cf74   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1+0x154       github.com/grpc-ecosystem/[email protected]/chain.go:39
#   0x19f152a   github.com/lightningnetwork/lnd/lnrpc._Lightning_ListChannels_Handler+0x14a     github.com/lightningnetwork/lnd@/lnrpc/rpc.pb.go:13114
#   0x191a6f9   google.golang.org/grpc.(*Server).processUnaryRPC+0x469                  google.golang.org/[email protected]/server.go:971
#   0x191e675   google.golang.org/grpc.(*Server).handleStream+0xd15                 google.golang.org/[email protected]/server.go:1250
#   0x192a07a   google.golang.org/grpc.(*Server).serveStreams.func1.1+0xba              google.golang.org/[email protected]/server.go:690

[wpaulino]

@PierreRochard can you include the rest of the profile? The GetLifespan call is blocking due to some other event not shown here.

[PierreRochard]

```
goroutine profile: total 108
13 @ 0x1030c80 0x104070b 0x1519d1a 0x105e891

0x1519d19 github.com/lightningnetwork/lnd/queue.(*ConcurrentQueue).start.func1+0x319 github.com/lightningnetwork/lnd/[email protected]/queue.go:67

8 @ 0x1030c80 0x104070b 0x16cc1e1 0x105e891

0x16cc1e0 github.com/lightningnetwork/lnd/lnwallet.(*SigPool).poolWorker+0x190 github.com/lightningnetwork/lnd@/lnwallet/sigpool.go:184

6 @ 0x1030c80 0x104070b 0x1760366 0x105e891

0x1760365 github.com/lightningnetwork/lnd/chainntnfs/neutrinonotify.(*NeutrinoNotifier).RegisterBlockEpochNtfn.func1+0x145 github.com/lightningnetwork/lnd@/chainntnfs/neutrinonotify/neutrino.go:930

4 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c6eeb 0x10c6ecd 0x11d3a3f 0x11e6858 0x12fe620 0x10f3de4 0x12fe86c 0x12fcdc4 0x1301031 0x130103c 0x10f8cca 0x10a4ae7 0x1879d17 0x1879cca 0x187a581 0x189c836 0x1919103 0x1929f63 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c6eea internal/poll.(*pollDesc).waitRead+0x22a internal/poll/fd_poll_runtime.go:92

0x10c6ecc internal/poll.(*FD).Read+0x20c internal/poll/fd_unix.go:169

0x11d3a3e net.(*netFD).Read+0x4e net/fd_unix.go:202

0x11e6857 net.(*conn).Read+0x67 net/net.go:184

0x12fe61f crypto/tls.(*atLeastReader).Read+0x5f crypto/tls/conn.go:780

0x10f3de3 bytes.(*Buffer).ReadFrom+0xb3 bytes/buffer.go:204

0x12fe86b crypto/tls.(*Conn).readFromUntil+0xeb crypto/tls/conn.go:802

0x12fcdc3 crypto/tls.(*Conn).readRecordOrCCS+0x123 crypto/tls/conn.go:609

0x1301030 crypto/tls.(*Conn).readRecord+0x160 crypto/tls/conn.go:577

0x130103b crypto/tls.(*Conn).Read+0x16b crypto/tls/conn.go:1255

0x10f8cc9 bufio.(*Reader).Read+0x269 bufio/bufio.go:226

0x10a4ae6 io.ReadAtLeast+0x86 io/io.go:310

0x1879d16 io.ReadFull+0x86 io/io.go:329

0x1879cc9 golang.org/x/net/http2.readFrameHeader+0x39 golang.org/x/[email protected]/http2/frame.go:237

0x187a580 golang.org/x/net/http2.(*Framer).ReadFrame+0xa0 golang.org/x/[email protected]/http2/frame.go:492

0x189c835 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams+0x85 google.golang.org/[email protected]/internal/transport/http2_server.go:429

0x1919102 google.golang.org/grpc.(*Server).serveStreams+0xf2 google.golang.org/[email protected]/server.go:686

0x1929f62 google.golang.org/grpc.(*Server).handleRawConn.func1+0x42 google.golang.org/[email protected]/server.go:648

4 @ 0x1030c80 0x104070b 0x151c0c3 0x105e891

0x151c0c2 github.com/lightninglabs/neutrino/blockntfns.(*SubscriptionManager).NewSubscription.func1+0x142 github.com/lightninglabs/[email protected]/blockntfns/manager.go:223

4 @ 0x1030c80 0x104070b 0x188821a 0x1888a23 0x18aad5b 0x105e891

0x1888219 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x119 google.golang.org/[email protected]/internal/transport/controlbuf.go:317

0x1888a22 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x1e2 google.golang.org/[email protected]/internal/transport/controlbuf.go:435

0x18aad5a google.golang.org/grpc/internal/transport.newHTTP2Server.func2+0xca google.golang.org/[email protected]/internal/transport/http2_server.go:276

4 @ 0x1030c80 0x104070b 0x18a0be8 0x105e891

0x18a0be7 google.golang.org/grpc/internal/transport.(*http2Server).keepalive+0x217 google.golang.org/[email protected]/internal/transport/http2_server.go:921

3 @ 0x1030c80 0x104070b 0x1566029 0x1570814 0x105e891

0x1566028 github.com/lightninglabs/neutrino.rescan+0xef8 github.com/lightninglabs/[email protected]/rescan.go:651

0x1570813 github.com/lightninglabs/neutrino.(*Rescan).Start.func1+0xe3 github.com/lightninglabs/[email protected]/rescan.go:1281

2 @ 0x1030c80 0x104070b 0x1518e8c 0x105e891

0x1518e8b github.com/lightningnetwork/lnd/queue.(*GCQueue).queueManager+0x1cb github.com/lightningnetwork/lnd/[email protected]/gc_queue.go:139

2 @ 0x1030c80 0x104070b 0x17896c4 0x105e891

0x17896c3 github.com/lightningnetwork/lnd/subscribe.(*Server).subscriptionHandler+0x153 github.com/lightningnetwork/lnd@/subscribe/subscribe.go:167

2 @ 0x1030c80 0x104070b 0x1a5f8fc 0x1b493de 0x1b4895c 0x1a08df9 0x1a9cdc4 0x1b77c81 0x1a9cd4c 0x194c400 0x1a9cf75 0x19f152b 0x191a6fa 0x191e676 0x192a07b 0x105e891

0x1a5f8fb github.com/lightningnetwork/lnd/chanfitness.(*ChannelEventStore).GetLifespan+0x13b github.com/lightningnetwork/lnd@/chanfitness/chaneventstore.go:351

0x1b493dd github.com/lightningnetwork/lnd.createRPCOpenChannel+0x7bd github.com/lightningnetwork/lnd@/rpcserver.go:3085

0x1b4895b github.com/lightningnetwork/lnd.(*rpcServer).ListChannels+0x30b github.com/lightningnetwork/lnd@/rpcserver.go:2981

0x1a08df8 github.com/lightningnetwork/lnd/lnrpc._Lightning_ListChannels_Handler.func1+0x88 github.com/lightningnetwork/lnd@/lnrpc/rpc.pb.go:13112

0x1a9cdc3 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x113 github.com/grpc-ecosystem/[email protected]/chain.go:31

0x1b77c80 github.com/lightningnetwork/lnd.errorLogUnaryServerInterceptor.func1+0x80 github.com/lightningnetwork/lnd@/log.go:151

0x1a9cd4b github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x9b github.com/grpc-ecosystem/[email protected]/chain.go:34

0x194c3ff github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1+0x14f github.com/lightningnetwork/lnd@/macaroons/service.go:124

0x1a9cf74 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1+0x154 github.com/grpc-ecosystem/[email protected]/chain.go:39

0x19f152a github.com/lightningnetwork/lnd/lnrpc._Lightning_ListChannels_Handler+0x14a github.com/lightningnetwork/lnd@/lnrpc/rpc.pb.go:13114

0x191a6f9 google.golang.org/grpc.(*Server).processUnaryRPC+0x469 google.golang.org/[email protected]/server.go:971

0x191e675 google.golang.org/grpc.(*Server).handleStream+0xd15 google.golang.org/[email protected]/server.go:1250

0x192a07a google.golang.org/grpc.(*Server).serveStreams.func1.1+0xba google.golang.org/[email protected]/server.go:690

2 @ 0x1030c80 0x104070b 0x1a9b556 0x105e891

0x1a9b555 github.com/lightningnetwork/lnd/pool.(*Worker).requestHandler+0x115 github.com/lightningnetwork/lnd@/pool/worker.go:153

1 @ 0x102c732 0x10453e6 0x1a49e42 0x105e891

0x10453e5 os/signal.signal_recv+0x95 runtime/sigqueue.go:144

0x1a49e41 os/signal.loop+0x21 os/signal/signal_unix.go:23

1 @ 0x1030c80 0x10057f8 0x10057ce 0x10054bb 0x155dc14 0x155dbf0 0x105e891

0x155dc13 github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect+0x53 github.com/btcsuite/[email protected]/peer/peer.go:2200

0x155dbef github.com/lightninglabs/neutrino.(*ChainService).peerDoneHandler+0x2f github.com/lightninglabs/[email protected]/neutrino.go:1463

1 @ 0x1030c80 0x10057f8 0x10057ce 0x10054bb 0x190e296 0x105e891

0x190e295 google.golang.org/grpc.(*addrConn).resetTransport+0xa25 google.golang.org/[email protected]/clientconn.go:1066

1 @ 0x1030c80 0x1005b9a 0x10054bb 0x1a0be1c 0x105e891

0x1a0be1b github.com/lightningnetwork/lnd/lnrpc.RegisterLightningHandlerFromEndpoint.func1.1+0x4b github.com/lightningnetwork/lnd@/lnrpc/rpc.pb.gw.go:1070

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c6eeb 0x10c6ecd 0x11d3a3f 0x11e6858 0x10a4ae7 0x1202d28 0x1202cea 0x1203b40 0x15380c3 0x1539d79 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c6eea internal/poll.(*pollDesc).waitRead+0x22a internal/poll/fd_poll_runtime.go:92

0x10c6ecc internal/poll.(*FD).Read+0x20c internal/poll/fd_unix.go:169

0x11d3a3e net.(*netFD).Read+0x4e net/fd_unix.go:202

0x11e6857 net.(*conn).Read+0x67 net/net.go:184

0x10a4ae6 io.ReadAtLeast+0x86 io/io.go:310

0x1202d27 io.ReadFull+0x87 io/io.go:329

0x1202ce9 github.com/btcsuite/btcd/wire.readMessageHeader+0x49 github.com/btcsuite/[email protected]/wire/message.go:204

0x1203b3f github.com/btcsuite/btcd/wire.ReadMessageWithEncodingN+0x4f github.com/btcsuite/[email protected]/wire/message.go:344

0x15380c2 github.com/btcsuite/btcd/peer.(*Peer).readMessage+0xb2 github.com/btcsuite/[email protected]/peer/peer.go:1009

0x1539d78 github.com/btcsuite/btcd/peer.(*Peer).inHandler+0xb8 github.com/btcsuite/[email protected]/peer/peer.go:1334

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c6eeb 0x10c6ecd 0x11d3a3f 0x11e6858 0x12fe620 0x10f3de4 0x12fe86c 0x12fcdc4 0x1301031 0x130103c 0x10f8cca 0x10a4ae7 0x1879d17 0x1879cca 0x187a581 0x189844e 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c6eea internal/poll.(*pollDesc).waitRead+0x22a internal/poll/fd_poll_runtime.go:92

0x10c6ecc internal/poll.(*FD).Read+0x20c internal/poll/fd_unix.go:169

0x11d3a3e net.(*netFD).Read+0x4e net/fd_unix.go:202

0x11e6857 net.(*conn).Read+0x67 net/net.go:184

0x12fe61f crypto/tls.(*atLeastReader).Read+0x5f crypto/tls/conn.go:780

0x10f3de3 bytes.(*Buffer).ReadFrom+0xb3 bytes/buffer.go:204

0x12fe86b crypto/tls.(*Conn).readFromUntil+0xeb crypto/tls/conn.go:802

0x12fcdc3 crypto/tls.(*Conn).readRecordOrCCS+0x123 crypto/tls/conn.go:609

0x1301030 crypto/tls.(*Conn).readRecord+0x160 crypto/tls/conn.go:577

0x130103b crypto/tls.(*Conn).Read+0x16b crypto/tls/conn.go:1255

0x10f8cc9 bufio.(*Reader).Read+0x269 bufio/bufio.go:226

0x10a4ae6 io.ReadAtLeast+0x86 io/io.go:310

0x1879d16 io.ReadFull+0x86 io/io.go:329

0x1879cc9 golang.org/x/net/http2.readFrameHeader+0x39 golang.org/x/[email protected]/http2/frame.go:237

0x187a580 golang.org/x/net/http2.(*Framer).ReadFrame+0xa0 golang.org/x/[email protected]/http2/frame.go:492

0x189844d google.golang.org/grpc/internal/transport.(*http2Client).reader+0x17d google.golang.org/[email protected]/internal/transport/http2_client.go:1223

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c88d8 0x10c88b7 0x11d43b2 0x11f0042 0x11ee797 0x1329e07 0x14adc30 0x1b7ccf9 0x1b7ccaa 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c88d7 internal/poll.(*pollDesc).waitRead+0x1f7 internal/poll/fd_poll_runtime.go:92

0x10c88b6 internal/poll.(*FD).Accept+0x1d6 internal/poll/fd_unix.go:384

0x11d43b1 net.(*netFD).accept+0x41 net/fd_unix.go:238

0x11f0041 net.(*TCPListener).accept+0x31 net/tcpsock_posix.go:139

0x11ee796 net.(*TCPListener).Accept+0x46 net/tcpsock.go:261

0x1329e06 crypto/tls.(*listener).Accept+0x36 crypto/tls/tls.go:59

0x14adc2f net/http.(*Server).Serve+0x27f net/http/server.go:2896

0x1b7ccf8 net/http.Serve+0x118 net/http/server.go:2468

0x1b7cca9 github.com/lightningnetwork/lnd.(*rpcServer).Start.func3+0xc9 github.com/lightningnetwork/lnd@/rpcserver.go:796

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c88d8 0x10c88b7 0x11d43b2 0x11f0042 0x11ee797 0x14adc30 0x14ad957 0x1b7715f 0x1b7716a 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c88d7 internal/poll.(*pollDesc).waitRead+0x1f7 internal/poll/fd_poll_runtime.go:92

0x10c88b6 internal/poll.(*FD).Accept+0x1d6 internal/poll/fd_unix.go:384

0x11d43b1 net.(*netFD).accept+0x41 net/fd_unix.go:238

0x11f0041 net.(*TCPListener).accept+0x31 net/tcpsock_posix.go:139

0x11ee796 net.(*TCPListener).Accept+0x46 net/tcpsock.go:261

0x14adc2f net/http.(*Server).Serve+0x27f net/http/server.go:2896

0x14ad956 net/http.(*Server).ListenAndServe+0xb6 net/http/server.go:2825

0x1b7715e net/http.ListenAndServe+0x19e net/http/server.go:3081

0x1b77169 github.com/lightningnetwork/lnd.Main.func2+0x1a9 github.com/lightningnetwork/lnd@/lnd.go:202

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c88d8 0x10c88b7 0x11d43b2 0x11f0042 0x11ee797 0x17ffbb3 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c88d7 internal/poll.(*pollDesc).waitRead+0x1f7 internal/poll/fd_poll_runtime.go:92

0x10c88b6 internal/poll.(*FD).Accept+0x1d6 internal/poll/fd_unix.go:384

0x11d43b1 net.(*netFD).accept+0x41 net/fd_unix.go:238

0x11f0041 net.(*TCPListener).accept+0x31 net/tcpsock_posix.go:139

0x11ee796 net.(*TCPListener).Accept+0x46 net/tcpsock.go:261

0x17ffbb2 github.com/lightningnetwork/lnd/brontide.(*Listener).listen+0xc2 github.com/lightningnetwork/lnd@/brontide/listener.go:79

1 @ 0x1030c80 0x102b98a 0x102af55 0x10c5f15 0x10c88d8 0x10c88b7 0x11d43b2 0x11f0042 0x11ee797 0x1917d3e 0x1b7cb65 0x105e891

0x102af54 internal/poll.runtime_pollWait+0x54 runtime/netpoll.go:184

0x10c5f14 internal/poll.(*pollDesc).wait+0x44 internal/poll/fd_poll_runtime.go:87

0x10c88d7 internal/poll.(*pollDesc).waitRead+0x1f7 internal/poll/fd_poll_runtime.go:92

0x10c88b6 internal/poll.(*FD).Accept+0x1d6 internal/poll/fd_unix.go:384

0x11d43b1 net.(*netFD).accept+0x41 net/fd_unix.go:238

0x11f0041 net.(*TCPListener).accept+0x31 net/tcpsock_posix.go:139

0x11ee796 net.(*TCPListener).Accept+0x46 net/tcpsock.go:261

0x1917d3d google.golang.org/grpc.(*Server).Serve+0x22d google.golang.org/[email protected]/server.go:561

0x1b7cb64 github.com/lightningnetwork/lnd.(*rpcServer).Start.func1+0x104 github.com/lightningnetwork/lnd@/rpcserver.go:739

1 @ 0x1030c80 0x104070b 0x10a6207 0x10a6acc 0x10f8603 0x10f935d 0x10f9594 0x1637cdf 0x163b7a9 0x105e891

0x10a6206 io.(*pipe).Read+0xe6 io/pipe.go:50

0x10a6acb io.(*PipeReader).Read+0x4b io/pipe.go:127

0x10f8602 bufio.(*Reader).fill+0x102 bufio/bufio.go:100

0x10f935c bufio.(*Reader).ReadSlice+0x3c bufio/bufio.go:359

0x10f9593 bufio.(*Reader).ReadLine+0x33 bufio/bufio.go:388

0x1637cde github.com/jrick/logrotate/rotator.(*Rotator).Run+0x8e github.com/jrick/[email protected]/rotator/rotator.go:100

0x163b7a8 github.com/lightningnetwork/lnd/build.(*RotatingLogWriter).InitLogRotator.func1+0x48 github.com/lightningnetwork/lnd@/build/logrotator.go:80

1 @ 0x1030c80 0x104070b 0x12dfe2d 0x105e891

0x12dfe2c github.com/btcsuite/btcd/connmgr.(*ConnManager).connHandler+0x1ac github.com/btcsuite/[email protected]/connmgr/connmanager.go:241

1 @ 0x1030c80 0x104070b 0x151a86f 0x105e891

0x151a86e github.com/lightninglabs/neutrino/blockntfns.(*SubscriptionManager).subscriptionHandler+0x19e github.com/lightninglabs/[email protected]/blockntfns/manager.go:160

1 @ 0x1030c80 0x104070b 0x1524c3f 0x1528886 0x105e891

0x1524c3e github.com/lightninglabs/neutrino/query.(*worker).Run+0x26e github.com/lightninglabs/[email protected]/query/worker.go:95

0x1528885 github.com/lightninglabs/neutrino/query.(*WorkManager).workDispatcher.func2+0xd5 github.com/lightninglabs/[email protected]/query/workmanager.go:282

1 @ 0x1030c80 0x104070b 0x1526718 0x105e891

0x1526717 github.com/lightninglabs/neutrino/query.(*WorkManager).workDispatcher+0x697 github.com/lightninglabs/[email protected]/query/workmanager.go:255

1 @ 0x1030c80 0x104070b 0x152a5d1 0x105e891

0x152a5d0 github.com/btcsuite/btcd/addrmgr.(*AddrManager).addressHandler+0x110 github.com/btcsuite/[email protected]/addrmgr/addrmanager.go:346

1 @ 0x1030c80 0x104070b 0x1539388 0x105e891

0x1539387 github.com/btcsuite/btcd/peer.(*Peer).stallHandler+0x1f7 github.com/btcsuite/[email protected]/peer/peer.go:1198

1 @ 0x1030c80 0x104070b 0x153b755 0x105e891

0x153b754 github.com/btcsuite/btcd/peer.(*Peer).queueHandler+0x234 github.com/btcsuite/[email protected]/peer/peer.go:1574

1 @ 0x1030c80 0x104070b 0x153c6ba 0x105e891

0x153c6b9 github.com/btcsuite/btcd/peer.(*Peer).outHandler+0xe9 github.com/btcsuite/[email protected]/peer/peer.go:1708

1 @ 0x1030c80 0x104070b 0x153ccf5 0x105e891

0x153ccf4 github.com/btcsuite/btcd/peer.(*Peer).pingHandler+0x104 github.com/btcsuite/[email protected]/peer/peer.go:1782

1 @ 0x1030c80 0x104070b 0x15406ac 0x105e891

0x15406ab github.com/lightninglabs/neutrino/pushtx.(*Broadcaster).broadcastHandler+0x37b github.com/lightninglabs/[email protected]/pushtx/broadcaster.go:166

1 @ 0x1030c80 0x104070b 0x155129a 0x105e891

0x1551299 github.com/lightninglabs/neutrino.(*blockManager).blockHandler+0x149 github.com/lightninglabs/[email protected]/blockmanager.go:2083

1 @ 0x1030c80 0x104070b 0x155bb00 0x105e891

0x155baff github.com/lightninglabs/neutrino.(*ChainService).peerHandler+0x1ef github.com/lightninglabs/[email protected]/neutrino.go:1082

1 @ 0x1030c80 0x104070b 0x1561407 0x156d216 0x156e8d4 0x105e891

0x1561406 github.com/lightninglabs/neutrino.queryChainServicePeers+0x466 github.com/lightninglabs/[email protected]/query.go:416

0x156d215 github.com/lightninglabs/neutrino.NewChainService.func2+0x65 github.com/lightninglabs/[email protected]/neutrino.go:703

0x156e8d3 github.com/lightninglabs/neutrino.(*ChainService).GetCFilter.func1+0x1a3 github.com/lightninglabs/[email protected]/query.go:866

1 @ 0x1030c80 0x104070b 0x15635cc 0x1770b8a 0x179d3c2 0x179c766 0x1b81a39 0x107af63 0x1b5edd5 0x1b5ed6a 0x1b1d727 0x1b8b8bd 0x10308ae 0x105e891

0x15635cb github.com/lightninglabs/neutrino.(*ChainService).GetCFilter+0x4db github.com/lightninglabs/[email protected]/query.go:894

0x1770b89 github.com/lightningnetwork/lnd/routing/chainview.(*CfFilteredChainView).FilterBlock+0x199 github.com/lightningnetwork/lnd@/routing/chainview/neutrino.go:240

0x179d3c1 github.com/lightningnetwork/lnd/routing.(*ChannelRouter).syncGraphWithChain+0x851 github.com/lightningnetwork/lnd@/routing/router.go:687

0x179c765 github.com/lightningnetwork/lnd/routing.(*ChannelRouter).Start+0x5c5 github.com/lightningnetwork/lnd@/routing/router.go:502

0x1b81a38 github.com/lightningnetwork/lnd.(*server).Start.func1+0x518 github.com/lightningnetwork/lnd@/server.go:1300

0x107af62 sync.(*Once).doSlow+0xe2 sync/once.go:66

0x1b5edd4 sync.(*Once).Do+0x94 sync/once.go:57

0x1b5ed69 github.com/lightningnetwork/lnd.(*server).Start+0x29 github.com/lightningnetwork/lnd@/server.go:1224

0x1b1d726 github.com/lightningnetwork/lnd.Main+0x1ad6 github.com/lightningnetwork/lnd@/lnd.go:654

0x1b8b8bc main.main+0x2c github.com/lightningnetwork/lnd@/cmd/lnd/main.go:14

0x10308ad runtime.main+0x21d runtime/proc.go:203

1 @ 0x1030c80 0x104070b 0x1584277 0x105e891

0x1584276 github.com/btcsuite/btcwallet/chain.(*NeutrinoClient).notificationHandler+0x296 github.com/btcsuite/[email protected]/chain/neutrino.go:693

1 @ 0x1030c80 0x104070b 0x15995b0 0x105e891

0x15995af github.com/btcsuite/btcwallet/wallet.(*Wallet).handleChainNotifications+0x19f github.com/btcsuite/[email protected]/wallet/chainntfns.go:88

1 @ 0x1030c80 0x104070b 0x15a4a98 0x105e891

0x15a4a97 github.com/btcsuite/btcwallet/wallet.(*Wallet).rescanBatchHandler+0x197 github.com/btcsuite/[email protected]/wallet/rescan.go:116

1 @ 0x1030c80 0x104070b 0x15a565c 0x105e891

0x15a565b github.com/btcsuite/btcwallet/wallet.(*Wallet).rescanProgressHandler+0x13b github.com/btcsuite/[email protected]/wallet/rescan.go:211

1 @ 0x1030c80 0x104070b 0x15a5aee 0x105e891

0x15a5aed github.com/btcsuite/btcwallet/wallet.(*Wallet).rescanRPCHandler+0x13d github.com/btcsuite/[email protected]/wallet/rescan.go:249

1 @ 0x1030c80 0x104070b 0x15ab9f6 0x105e891

0x15ab9f5 github.com/btcsuite/btcwallet/wallet.(*Wallet).txCreator+0x105 github.com/btcsuite/[email protected]/wallet/wallet.go:1164

1 @ 0x1030c80 0x104070b 0x15ac155 0x105e891

0x15ac154 github.com/btcsuite/btcwallet/wallet.(*Wallet).walletLocker+0x2c4 github.com/btcsuite/[email protected]/wallet/wallet.go:1242

1 @ 0x1030c80 0x104070b 0x16ce3c5 0x105e891

0x16ce3c4 github.com/lightningnetwork/lnd/lnwallet.(*LightningWallet).requestHandler+0xb4 github.com/lightningnetwork/lnd@/lnwallet/wallet.go:387

1 @ 0x1030c80 0x104070b 0x16e238f 0x16eace8 0x105e891

0x16e238e github.com/lightningnetwork/lnd/sweep.(*UtxoSweeper).collector+0x28e github.com/lightningnetwork/lnd@/sweep/sweeper.go:457

0x16eace7 github.com/lightningnetwork/lnd/sweep.(*UtxoSweeper).Start.func1+0xb7 github.com/lightningnetwork/lnd@/sweep/sweeper.go:347

1 @ 0x1030c80 0x104070b 0x16f5425 0x105e891

0x16f5424 github.com/lightningnetwork/lnd/contractcourt.(*chainWatcher).closeObserver+0x1d4 github.com/lightningnetwork/lnd@/contractcourt/chain_watcher.go:408

1 @ 0x1030c80 0x104070b 0x1704e32 0x105e891

0x1704e31 github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).channelAttendant+0x2b1 github.com/lightningnetwork/lnd@/contractcourt/channel_arbitrator.go:1968

1 @ 0x1030c80 0x104070b 0x1728d9a 0x105e891

0x1728d99 github.com/lightningnetwork/lnd/htlcswitch.(*DecayedLog).garbageCollector+0x149 github.com/lightningnetwork/lnd@/htlcswitch/decayedlog.go:161

1 @ 0x1030c80 0x104070b 0x1744aa1 0x105e891

0x1744aa0 github.com/lightningnetwork/lnd/htlcswitch.(*Switch).htlcForwarder+0x460 github.com/lightningnetwork/lnd@/htlcswitch/switch.go:1498

1 @ 0x1030c80 0x104070b 0x175bc45 0x105e891

0x175bc44 github.com/lightningnetwork/lnd/chainntnfs/neutrinonotify.(*NeutrinoNotifier).notificationDispatcher+0x1f4 github.com/lightningnetwork/lnd@/chainntnfs/neutrinonotify/neutrino.go:282

1 @ 0x1030c80 0x104070b 0x177092d 0x105e891

0x177092c github.com/lightningnetwork/lnd/routing/chainview.(*CfFilteredChainView).chainFilterer+0x14c github.com/lightningnetwork/lnd@/routing/chainview/neutrino.go:200

1 @ 0x1030c80 0x104070b 0x1771e46 0x105e891

0x1771e45 github.com/lightningnetwork/lnd/routing/chainview.(*blockEventQueue).queueCoordinator+0x2f5 github.com/lightningnetwork/lnd@/routing/chainview/queue.go:117

1 @ 0x1030c80 0x104070b 0x17c4f05 0x105e891

0x17c4f04 github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).networkHandler+0x474 github.com/lightningnetwork/lnd@/discovery/gossiper.go:920

1 @ 0x1030c80 0x104070b 0x17d12a9 0x105e891

0x17d12a8 github.com/lightningnetwork/lnd/discovery.(*reliableSender).peerHandler+0x1d8 github.com/lightningnetwork/lnd@/discovery/reliable_sender.go:174

1 @ 0x1030c80 0x104070b 0x17d326a 0x105e891

0x17d3269 github.com/lightningnetwork/lnd/discovery.(*SyncManager).syncerHandler+0x2e9 github.com/lightningnetwork/lnd@/discovery/sync_manager.go:215

1 @ 0x1030c80 0x104070b 0x188821a 0x1888a23 0x18a99bb 0x105e891

0x1888219 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x119 google.golang.org/[email protected]/internal/transport/controlbuf.go:317

0x1888a22 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x1e2 google.golang.org/[email protected]/internal/transport/controlbuf.go:435

0x18a99ba google.golang.org/grpc/internal/transport.newHTTP2Client.func3+0x7a google.golang.org/[email protected]/internal/transport/http2_client.go:330

1 @ 0x1030c80 0x104070b 0x1906aaf 0x105e891

0x1906aae google.golang.org/grpc.(*ccBalancerWrapper).watcher+0x10e google.golang.org/[email protected]/balancer_conn_wrappers.go:122

1 @ 0x1030c80 0x104070b 0x1a4a316 0x105e891

0x1a4a315 github.com/lightningnetwork/lnd/signal.mainInterruptHandler+0x105 github.com/lightningnetwork/lnd@/signal/signal.go:71

1 @ 0x1030c80 0x104070b 0x1af0444 0x105e891

0x1af0443 github.com/lightningnetwork/lnd.(*breachArbiter).contractObserver+0x1a3 github.com/lightningnetwork/lnd@/breacharbiter.go:262

1 @ 0x1030c80 0x104070b 0x1b6f470 0x105e891

0x1b6f46f github.com/lightningnetwork/lnd.(*utxoNursery).incubator+0x14f github.com/lightningnetwork/lnd@/utxonursery.go:679

1 @ 0x1030c80 0x104070b 0x1b7576c 0x105e891

0x1b7576b github.com/lightningnetwork/lnd.(*channelNotifier).SubscribeChans.func1+0x13b github.com/lightningnetwork/lnd@/channel_notifier.go:68

1 @ 0x1030c80 0x1042278 0x104224e 0x1078ccd 0x1548c46 0x156c86d 0x105e891

0x104224d sync.runtime_notifyListWait+0xcd runtime/sema.go:510

0x1078ccc sync.(*Cond).Wait+0x9c sync/cond.go:56

0x1548c45 github.com/lightninglabs/neutrino.(*blockManager).cfHandler+0x1085 github.com/lightninglabs/[email protected]/blockmanager.go:688

0x156c86c github.com/lightninglabs/neutrino.(*blockManager).Start.func1+0x1dc github.com/lightninglabs/[email protected]/blockmanager.go:337

1 @ 0x1030c80 0x1042278 0x104224e 0x1078ccd 0x156b5d3 0x105e891

0x104224d sync.runtime_notifyListWait+0xcd runtime/sema.go:510

0x1078ccc sync.(*Cond).Wait+0x9c sync/cond.go:56

0x156b5d2 github.com/lightninglabs/neutrino.(*UtxoScanner).batchManager+0x102 github.com/lightninglabs/[email protected]/utxoscanner.go:220

1 @ 0x14a3350 0x105e891

0x14a3350 net/http.(*connReader).backgroundRead+0x0 net/http/server.go:676

1 @ 0x1719e35 0x1719c50 0x171687a 0x1a4665a 0x1a47071 0x14aa424 0x14ac2fd 0x14ad874 0x14a9215 0x105e891

0x1719e34 runtime/pprof.writeRuntimeProfile+0x94 runtime/pprof/pprof.go:708

0x1719c4f runtime/pprof.writeGoroutine+0x9f runtime/pprof/pprof.go:670

0x1716879 runtime/pprof.(*Profile).WriteTo+0x3d9 runtime/pprof/pprof.go:329

0x1a46659 net/http/pprof.handler.ServeHTTP+0x339 net/http/pprof/pprof.go:245

0x1a47070 net/http/pprof.Index+0x6f0 net/http/pprof/pprof.go:268

0x14aa423 net/http.HandlerFunc.ServeHTTP+0x43 net/http/server.go:2007

0x14ac2fc net/http.(*ServeMux).ServeHTTP+0x1bc net/http/server.go:2387

0x14ad873 net/http.serverHandler.ServeHTTP+0xa3 net/http/server.go:2802

0x14a9214 net/http.(*conn).serve+0x874 net/http/server.go:1890

```

[wpaulino]

Thanks @PierreRochard, it should be fixed by https://github.com/lightningnetwork/lnd/pull/4176. The reason the server hasn't started for you is because it's syncing the graph with the chain. This is a very intensive process for light clients, it may be worth running with the --routing.assumechanvalid flag available when installing lnd with make install tags=experimental.

[PierreRochard]

I see, thank you for the quick response!

[PierreRochard]

Indeed, it no longer hangs now that the graph is synced.