Lnd: panic: runtime error: invalid memory address or nil pointer dereferenc

Created on 19 May 2020 · 12Comments · Source: lightningnetwork/lnd

Background

After updating to c2516d935284f320cc2b9fe64fed0f512c517deb LND cannot start

Your environment

version of lnd
Version: 0.10.0-beta commit=v0.10.0-beta-161-gc2516d935284f320cc2b9fe64fed0f512c517deb, build=production, logging=default
which operating system (uname -a on *Nix)
Linux 4.19.97-v8+ #1294 SMP PREEMPT Thu Jan 30 13:27:08 GMT 2020 aarch64 GNU/Linux
version of btcd, bitcoind, or other backend
Bitcoin Core Daemon version v0.18.1

Steps to reproduce

See first description.

Expected behaviour

Should start

Actual behaviour

2020-05-19 09:36:36.605 [INF] ATPL: Autopilot Agent starting
2020-05-19 09:36:36.605 [DBG] ATPL: Manager started autopilot agent
2020-05-19 09:36:36.605 [DBG] ATPL: Applying external balance state update
2020-05-19 09:36:36.609 [DBG] ATPL: Pending channels: (map[autopilot.NodeID]autopilot.Channel) {
}

2020-05-19 09:36:37.676 [INF] SRVR: Established connection to: 02a04446caa81636d60d63b066f2814cbd3a6b5c258e3172cbdded7a16e2cfff4c@3.122.40.122:9735
2020-05-19 09:36:37.676 [INF] SRVR: Finalizing connection to 02a04446caa81636d60d63b066f2814cbd3a6b5c258e3172cbdded7a16e2cfff4c@3.122.40.122:9735, inbound=false
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x1326c]

goroutine 772 [running]:
github.com/lightningnetwork/lnd.(*peer).writeMessage.func1(0x2a660f8, 0x2c89cd0)
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/peer.go:1618 +0x134
github.com/lightningnetwork/lnd.(*peer).writeMessage(0x39d6300, 0xf820e0, 0x3b885f0, 0x8, 0xac1c20)
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/peer.go:1651 +0x108
github.com/lightningnetwork/lnd.(*peer).sendInitMsg(0x39d6300, 0xc76e6c, 0x10)
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/peer.go:2698 +0x78
github.com/lightningnetwork/lnd.(*peer).Start(0x39d6300, 0xdf4a28, 0x2a0a500)
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/peer.go:337 +0xb0
github.com/lightningnetwork/lnd.(*server).peerInitializer(0x2a0a500, 0x39d6300)
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/server.go:2887 +0xd0
created by github.com/lightningnetwork/lnd.(*server).peerConnected
        /home/lightning/gocode/src/github.com/lightningnetwork/lnd/server.go:2814 +0x3d4

Source

alevchuk

Most helpful comment

Now that we use Github actions (and have more builds), we can also start running CI against ARM machines.

Roasbeef on 21 May 2020

❤2

All 12 comments

I looked at the line the stack trace indicates and tried to reproduce this bug.
No luck so far.

How many peers do you have? Is this the first peer that is being connected to? Does the error persist, even if you restart?

To narrow this down, could you please try if the same error also happens on 2f84d1a8?

guggero on 20 May 2020

I'm receiving the same crash after compiling a new version on my system as well.

2020-05-20 12:43:09.500 [INF] BTCN: Server listening on [::]:9735
2020-05-20 12:43:09.501 [INF] SRVR: New inbound connection from <redacted>
2020-05-20 12:43:09.501 [INF] SRVR: Finalizing connection to <redacted>, inbound=true
2020-05-20 12:43:09.503 [INF] SRVR: New inbound connection from <redacted>
2020-05-20 12:43:09.503 [INF] SRVR: Finalizing connection to <redacted>, inbound=true
2020-05-20 12:43:09.503 [INF] SRVR: New inbound connection from <redacted>
2020-05-20 12:43:09.504 [INF] SRVR: Finalizing connection to <redacted>, inbound=true
2020-05-20 12:43:09.505 [INF] SRVR: New inbound connection from <redacted>
2020-05-20 12:43:09.505 [INF] SRVR: Finalizing connection to <redacted>, inbound=true
2020-05-20 12:43:09.505 [INF] SRVR: New inbound connection from <redacted>
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x12bf8]

goroutine 575 [running]:
github.com/lightningnetwork/lnd.(*peer).writeMessage.func1(0x371c080, 0x45c6c60)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:1618 +0x134
github.com/lightningnetwork/lnd.(*peer).writeMessage(0x53de100, 0xf4e068, 0x45d8028, 0x8, 0xa94c40)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:1651 +0x108
github.com/lightningnetwork/lnd.(*peer).sendInitMsg(0x53de100, 0xc3df0a, 0x10)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:2698 +0x78
github.com/lightningnetwork/lnd.(*peer).Start(0x53de100, 0xdbc40c, 0x36043c0)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:337 +0xb0
github.com/lightningnetwork/lnd.(*server).peerInitializer(0x36043c0, 0x53de100)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2887 +0xcc
created by github.com/lightningnetwork/lnd.(*server).peerConnected
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2814 +0x3d8
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x12bf8]

goroutine 574 [running]:
github.com/lightningnetwork/lnd.(*peer).writeMessage.func1(0x371c080, 0x369ba30)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:1618 +0x134
github.com/lightningnetwork/lnd.(*peer).writeMessage(0x53de000, 0xf4e068, 0x4ad1f30, 0x8, 0xa94c40)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:1651 +0x108
github.com/lightningnetwork/lnd.(*peer).sendInitMsg(0x53de000, 0xc3df0a, 0x10)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:2698 +0x78
github.com/lightningnetwork/lnd.(*peer).Start(0x53de000, 0xdbc40c, 0x36043c0)
        /root/gocode/src/github.com/lightningnetwork/lnd/peer.go:337 +0xb0
github.com/lightningnetwork/lnd.(*server).peerInitializer(0x36043c0, 0x53de000)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2887 +0xcc
created by github.com/lightningnetwork/lnd.(*server).peerConnected
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2814 +0x3d8

LND Version: Version: 0.10.0-beta commit=v0.10.0-beta-179-gca9c58aec9d6cbef87a7cb84d3b28fc67c540185, build=production, logging=default

Linux Version: Linux localhost 4.15.0-1061-raspi2 #65-Ubuntu SMP PREEMPT Thu Apr 9 18:13:20 UTC 2020 armv7l armv7l armv7l GNU/Linux

Bitcoin Core Version: 0.19.1

clonmac on 20 May 2020

👍1

@clonmac thanks for the report. I'm still trying to figure out what exactly the problem is.
So the same questions for you: Does the error persist, even if you restart?
To narrow this down, could you please try if the same error also happens on commit 2f84d1a?

guggero on 20 May 2020

@guggero it does look like on the first peer. I have 8 peers after reverting. Don't know if it's transient, I reverted immediately after I got the failure. 2f84d1a build works fine at this time.

A specific LND problem related to this is that log lines starting from "panic: runtime error:" do not get captured in ~/.lnd/logs/.../lnd.log

alevchuk on 20 May 2020

@guggero, OK. Thanks. Yes, the segfault would happen even after numerous restarts before reverting to my previous build. For me, it didn't look like it was crashing on the first peer connection. I just compiled from 2f84d1a and that is running successfully here as well for me.

clonmac on 20 May 2020

Thank you both for the info. I think this will help me narrow it down.

guggero on 20 May 2020

Looks like I'm getting the same or similar issue.

I'm running latest master which is [INF] LTND: Version: 0.10.0-beta commit=v0.10.0-beta-195-g64b8aa8f114142b4b1d1e8e1e751c846d1a7f422, build=production, logging=default

2020-05-20 23:26:59.987 [INF] SRVR: New inbound connection from 23.129.64.197:30523
2020-05-20 23:26:59.987 [INF] SRVR: Finalizing connection to 0268b84ff41b75477349c1b1a8ab77213294884f7816028bec6704b325718c89d0@23.129.64.197:30523, inbound=true
2020-05-20 23:26:59.988 [INF] SRVR: New inbound connection from 23.129.64.197:60507
2020-05-20 23:26:59.988 [INF] PEER: disconnecting 0268b84ff41b75477349c1b1a8ab77213294884f7816028bec6704b325718c89d0@23.129.64.197:30523, reason: server: disconnecting peer 0268b84ff41b75477349c1b1a8ab77213294884f7816028bec6704b325718c89d0@23.129.64.197:30523
2020-05-20 23:26:59.988 [INF] SRVR: New inbound connection from 23.129.64.209:29861
2020-05-20 23:26:59.988 [INF] SRVR: Finalizing connection to 03817e6563dc36585b4510ff5fa50c0fe9a763ef14aa0bb8d10ead54c99c2da41d@23.129.64.209:29861, inbound=true
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x125b4]

goroutine 4032 [running]:
runtime/internal/atomic.goXadd64(0x32d2014, 0xb, 0x0, 0xb, 0x0)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:103 +0x1c
github.com/lightningnetwork/lnd.(*peer).writeMessage.func1(0x1d96fd8, 0x2cf8b40)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:1618 +0x134
github.com/lightningnetwork/lnd.(*peer).writeMessage(0x32d2000, 0xf5fdd0, 0x399cac8, 0x8, 0xaa4f90)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:1651 +0x108
github.com/lightningnetwork/lnd.(*peer).sendInitMsg(0x32d2000, 0xc4f0c5, 0x10)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:2698 +0x78
github.com/lightningnetwork/lnd.(*peer).Start(0x32d2000, 0xdcd714, 0x1ad42c0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:337 +0xb0
github.com/lightningnetwork/lnd.(*server).peerInitializer(0x1ad42c0, 0x32d2000)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/server.go:2896 +0xcc
created by github.com/lightningnetwork/lnd.(*server).peerConnected
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/server.go:2823 +0x3d8
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x125b4]

goroutine 4033 [running]:
runtime/internal/atomic.goXadd64(0x32d2114, 0xb, 0x0, 0xb, 0x0)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:103 +0x1c
github.com/lightningnetwork/lnd.(*peer).writeMessage.func1(0x1d96fd8, 0x2bd0f90)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:1618 +0x134
github.com/lightningnetwork/lnd.(*peer).writeMessage(0x32d2100, 0xf5fdd0, 0x197f148, 0x8, 0xaa4f90)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:1651 +0x108
github.com/lightningnetwork/lnd.(*peer).sendInitMsg(0x32d2100, 0xc4f0c5, 0x10)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:2698 +0x78
github.com/lightningnetwork/lnd.(*peer).Start(0x32d2100, 0xdcd714, 0x1ad42c0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/peer.go:337 +0xb0
github.com/lightningnetwork/lnd.(*server).peerInitializer(0x1ad42c0, 0x32d2100)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/server.go:2896 +0xcc
created by github.com/lightningnetwork/lnd.(*server).peerConnected
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/server.go:2823 +0x3d8

sangaman on 21 May 2020

Thank you @sangaman, your stack trace gave me the hint I needed! It was not a pointer that was nil but a pointer that wasn't properly aligned in the struct. Fix is incoming.

guggero on 21 May 2020

👍1

Thanks for the fix! Was this a recent bug and is there a way to catch struct issues during testing?

alevchuk on 21 May 2020

👍1

@alevchuk yeah we can look into expanding the linter to use tools like: https://github.com/dominikh/go-tools/tree/master/cmd/structlayout-optimize

Roasbeef on 21 May 2020

👍1

Now that we use Github actions (and have more builds), we can also start running CI against ARM machines.

Roasbeef on 21 May 2020

❤2

Thanks for the fix! Was this a recent bug and is there a way to catch struct issues during testing?

I did a refactor in #4286 that caused the problem. So yes, very recent change.
I'll look into structlayout-optimize and running itests on ARM.

guggero on 25 May 2020

❤1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Running on macOS Sierra

hxsquid · 3Comments

Concurrency issue with `update_fee`

pm47 · 3Comments

Funds lost after counterparty force-closed channel

zyp · 4Comments

Resolved hashes in payments do not reflect back their settled state

alexbosworth · 3Comments

[mobile] grpc: no transport security set

ealymbaev · 3Comments