[bug]: lnd crashes with panic: runtime error: invalid memory address or nil pointer dereference

[greenart7c3]

Background

lnd crashes with panic: runtime error: invalid memory address or nil pointer dereference

Your environment

lnd 0.18.4-beta
Ubuntu
bitcoind

Steps to reproduce

Don't know how to reproduce this but its happening a lot today. The service starts and after a while a get this error

Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x497698]
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: goroutine 30519 [running]:
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: sync.(*RWMutex).Lock(0xc000273180?)
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]:         sync/rwmutex.go:146 +0x18
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: github.com/lightningnetwork/lnd/lnwallet/chainfee.(*filterManager).updateMedian(0xc0041e6f00, {0xc00237b500, 0x8, 0x8})
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]:         github.com/lightningnetwork/lnd/lnwallet/chainfee/filtermanager.go:208 +0x5f
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: github.com/lightningnetwork/lnd/lnwallet/chainfee.(*filterManager).fetchPeerFilters(0xc0041e6f00)
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]:         github.com/lightningnetwork/lnd/lnwallet/chainfee/filtermanager.go:87 +0xc5
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]: created by github.com/lightningnetwork/lnd/lnwallet/chainfee.(*filterManager).Start in goroutine 1
Dec 31 18:44:24 greenart7c3-desktop lnd[820440]:         github.com/lightningnetwork/lnd/lnwallet/chainfee/filtermanager.go:60 +0x65

Expected behaviour

Actual behaviour

[Crypt-iQ]

@greenart7c3 what Go version are you using?

[greenart7c3]

go version go1.23.1 linux/amd64

[Crypt-iQ]

@greenart7c3 did you compile lnd yourself or did you use some sort of installer? I ask because the crashing line number in sync.rwmutex.go seems a little off if the go version used was go1.23.1 but I could be missing something.

[greenart7c3]

I used the compiled version from the releases

[Crypt-iQ]

I looked it up and the compiled version uses go1.22.6 and it looks like the mutex is nil for some reason

[ellemouton]

@Crypt-iQ - is it not that the actual filterManager is nil? I dont think the mutex can be nil (it is not a pointer)

[Crypt-iQ]

@Crypt-iQ - is it not that the actual filterManager is nil? I dont think the mutex can be nil (it is not a pointer)

Because it's crashing inside of sync.RWMutex (https://cs.opensource.google/go/go/+/refs/tags/go1.22.6:src/sync/rwmutex.go;l=146) I think the pointer receiver rw *RWMutex is nil. I think that if the panic were due to filterManager being nil then we'd see the panic in filtermanager.go. But I'm just guessing. There may be some quirk in how a BitcoindEstimator / BtcdEstimator is initialized with a filterManager.

[ziggie1984]

@greenart7c3 is your LND permanently in restart mode, because of this error, or does it only happen occasionally ? Would you be able to run a customized LND version with more DEBUG logs to trace down your problem ?

[ziggie1984]

Can you try the following solutions:

• Compiling golang from source, which then uses your local golang 1.23

• running it docker with the golang version 1.22.6

• downgrading your golang of your current system to 1.22.6 if you still want to use the compiled version.

UPDATE:

=> This should not be a problem, golang binaries are independant from the local golang version, so this is not the problem.

[ziggie1984]

waiting for your confirmation before closing.

[greenart7c3]

@greenart7c3 is your LND permanently in restart mode, because of this error, or does it only happen occasionally ? Would you be able to run a customized LND version with more DEBUG logs to trace down your problem ?

It only happened in that day and never again. The only difference that day in the logs is that it was having some tor connection issues with some peers

[ziggie1984]

which bitcoind version are you running ?

[ziggie1984]

do you still have the logs available ?

[greenart7c3]

I don't have the logs anymore
Bitcoind version is 28

[lord-cow]

I'm getting possibly the same issue. Just upgraded bitcoind and lnd to:

bitcoin-28.0
0.18.99-beta

FreeBSD 13.4
go121-1.21.13

All built from source.

EDIT (guggero): Unrelated issue, created here: #9413