Go-ethereum: Error starting protocol stack: missing block number for head header hash

[adrianmcli]

System information

Geth version: 1.6.0
OS & Version: OSX

Expected behaviour

Be able to login and activate Geth interactive prompt.

Actual behaviour

I run $ geth and I get the following error:

INFO [05-05|18:39:26] Starting peer-to-peer node               instance=Geth/v1.6.0-stable-facc47cb/darwin-amd64/go1.8.1
INFO [05-05|18:39:26] Allocated cache and file handles         database=/Users/liadrian/Library/Ethereum/chaindata cache=128 handles=1024
Fatal: Error starting protocol stack: missing block number for head header hash

I'm trying to get some of the ether I stored on these old accounts, but it's been a long time.

[fjl]

How old is your database? It might make more sense to resync.

[adrianmcli]

@fjl Super old. Probably the week of the Frontier release.

Are there instructions on how to resync? I tried running:

geth --fast --cache 1024

And it gave me the same error.

[karalabe]

geth removedb, then resync

[01GOD]

@karalebe If involved directly with ethereum, that answer is worthless and such a sign that ethereum is a scam. The "official" ethereum tools such as mist and wallet and geth are cr@p!!! Such BS tools. Sync'd many times again and again. Then did geth removedb and then did a resync. Yet, although geth says it is done, mist still says: Fatal: Error starting protocol stack: missing block number for head header hash

Cr@p tools. And from people that are seemingly running a big scam.

[gusarov]

Sorry, but I have the same issue 4 times in a row... Tried removedb, synced till the fatal error (checksum missmatch) then (sometimes removed last ldb files that caused error), continue with full mode but eventually always got "missing block number for head header hash".
Etherium is obviously not a scam but looks like it experiencing serious difficulties. And how you handled this issue doesn't looks good. I'm unable to sync on new PC for a week already.

[artemii235]

@karalabe
Having same problem starting from yesterday, it looks really bad - I have tried versions 1.7.2 and 1.7.3 - both starting having this issue maximum in 24 hours after synchronization starts - on different hosts and different block numbers. In both cases geth first crashed with error:

ERROR[11-27|09:32:10] Receipt refereced missing                number=4630922 hash=720016…26b163 index=93
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x178 pc=0xb5db04]

goroutine 3970050 [running]:
github.com/ethereum/go-ethereum/internal/ethapi.(*PublicTransactionPoolAPI).GetTransactionReceipt(0xc420d4ea60, 0x257475d016d41376, 0x2179b088dee3849f, 0xcfd258b1be20effc, 0x3b8bd0e527390d01, 0x0, 0x0, 0x0)
    /go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/ethapi/api.go:1023 +0x2f4
reflect.Value.call(0xc421752060, 0xc4201bfcf0, 0x13, 0xf4457d, 0x4, 0xc43d52d260, 0x2, 0x2, 0xc42223cb98, 0xc43d52d260, ...)
    /usr/local/go/src/reflect/value.go:434 +0x905
reflect.Value.Call(0xc421752060, 0xc4201bfcf0, 0x13, 0xc43d52d260, 0x2, 0x2, 0x1, 0x1, 0x19)
    /usr/local/go/src/reflect/value.go:302 +0xa4
github.com/ethereum/go-ethereum/rpc.(*Server).handle(0xc420f029a0, 0x1854040, 0xc4227c27e0, 0x1858d20, 0xc422826280, 0xc443a8d800, 0xc45180b1c0, 0x185d2a0, 0x824b53)
    /go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:311 +0x6ce
github.com/ethereum/go-ethereum/rpc.(*Server).exec(0xc420f029a0, 0x1854040, 0xc4227c27e0, 0x1858d20, 0xc422826280, 0xc443a8d800)
    /go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:333 +0x1ba
github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest.func2(0xc422180320, 0xc420f029a0, 0xc4201bc220, 0x1858d20, 0xc422826280, 0xc45c38d898, 0x1, 0x1, 0xc420b14e00)
    /go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:206 +0xad
created by github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest
    /go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:201 +0x2ab

Can this be fixed without full node resync? Is it possible to make this not happen at all? We need stable solution for our Dapp and these database corruptions are not acceptable - we have lost both main and reserve geth nodes due to this.

[artemii235]

Also this issue has been already mentioned here 27 days ago: https://github.com/ethereum/go-ethereum/issues/15408.

[hlogeon]

Any ideas on how it can be fixed? At least temporary solutions? Drop database and resync doesn't help

[hotelsieben]

Same problem. Updated to version 1.7.3. After starting node.log shows:
INFO [11-28|10:21:41] Starting peer-to-peer node instance=Geth/v1.7.2-stable-1db4ecdc/darwin-amd64/go1.9
INFO [11-28|10:21:41] Allocated cache and file handles database=/Users/.../Library/Ethereum/chaindata cache=1024 handles=1024
WARN [11-28|10:21:45] Upgrading database to use lookup entries
Fatal: Error starting protocol stack: missing block number for head header hash.
How to fix this issue?

[JackPickering]

Why has this issue been closed?

Is it possible to re-open the issue, it was closed by someone who has provided inadequate information on how to solve this problem, this is not a fix nor a reason why it has failed and a full node re-sync for most people takes hours if not days/weeks.

[karalabe]

There are quite a few issues convoluted into this single report.

• Accessing some strange receipt resulted in a crash.
• Possibly due to an earlier crash, some data got missing from the db.

We should handle both cases a lot more gracefully.

[JackPickering]

Okay, thank you for re-opening the case!

So what steps do we do from here if we do get this error, if data has gone missing due to a previous crash, is the only way to get such data back from a full node re-sync?

[hlogeon]

So what steps do we do from here if we do get this error, if data has gone missing due to a previous crash, is the only way to get such data back from a full node re-sync?

According to my experience, full node re-sync doesn't fix the problem. It just crashes again after re-sync. So you can fall into infinite resyncing loop.

[cfe1772e]

I have the same problem. It was with 1.7.2, but with 1.7.3 I have this bug really often (for 2 days it was 4 times).

The problem is on Linux server. On Linux machine I use geth with my app via web3 library. There I have 3 subscruptions:

• web3.eth.subscribe("pendingTransactions")
• web3.eth.subscribe("newBlockHeaders")
• web3.eth.subscribe("logs")

Also I have geth on macOS machine without any subscriptions. And geth on macOS works fine.

Here are stack trace of panic:

mber=4641074 hash=709c7d…fd0835                                                                                          [41/1916]
ERROR[11-29|01:42:53] Transaction referenced missing           number=4641074 hash=a58274…53b630 index=0
ERROR[11-29|01:42:53] Receipt refereced missing                number=4641074 hash=a58274…53b630 index=3
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x178 pc=0xb5d814]

goroutine 147394032 [running]:
github.com/ethereum/go-ethereum/internal/ethapi.(*PublicTransactionPoolAPI).GetTransactionReceipt(0xc422528bc0, 0xd54db036356110c1
, 0x80426e21f6420323, 0x556e3716a6784f26, 0x649863e8abd889e8, 0x0, 0x0, 0x0)
        /build/ethereum-ZSPmrv/ethereum-1.7.3+build11486+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/internal/etha
pi/api.go:1023 +0x2f4
reflect.Value.call(0xc420107860, 0xc421e7b248, 0x13, 0xf446fd, 0x4, 0xc421e79470, 0x2, 0x2, 0xc431177518, 0xc421e79470, ...)
        /usr/lib/go-1.9/src/reflect/value.go:434 +0x906
reflect.Value.Call(0xc420107860, 0xc421e7b248, 0x13, 0xc421e79470, 0x2, 0x2, 0x1, 0x1, 0xc4311775a0)
        /usr/lib/go-1.9/src/reflect/value.go:302 +0xa4
github.com/ethereum/go-ethereum/rpc.(*Server).handle(0xc421b67aa0, 0x185a3e0, 0xc593876480, 0x185f0c0, 0xc58682f450, 0xc50d348960,
 0xc427db5380, 0xc506c29500, 0x824c83)
        /build/ethereum-ZSPmrv/ethereum-1.7.3+build11486+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go
:311 +0x6ce
github.com/ethereum/go-ethereum/rpc.(*Server).exec(0xc421b67aa0, 0x185a3e0, 0xc593876480, 0x185f0c0, 0xc58682f450, 0xc50d348960)
        /build/ethereum-ZSPmrv/ethereum-1.7.3+build11486+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go
:333 +0x1ba
github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest.func2(0xc48dc470b0, 0xc421b67aa0, 0xc4bcdeeb90, 0x185f0c0, 0xc58682f450
, 0xc54aa91258, 0x1, 0x1, 0xba8cdfdcb99a5100)
        /build/ethereum-ZSPmrv/ethereum-1.7.3+build11486+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go
:206 +0xad
created by github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest
        /build/ethereum-ZSPmrv/ethereum-1.7.3+build11486+xenial/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go
:201 +0x2ab
root@xxx ~ # geth
INFO [11-29|01:42:53] Starting peer-to-peer node               instance=Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9
INFO [11-29|01:42:53] Allocated cache and file handles         database=/root/.ethereum/geth/chaindata cache=128 handles=1024
Fatal: Error starting protocol stack: missing block number for head header hash

And yes, only geth removedb can fix it :(

[JackPickering]

Just to add to the above, I run geth on macOS El Capitan and it fails for me.

[apukum]

Hello guys !! Do we have any solution to fix these issues. I am also facing the issue since last 2 wks. Need a reliable solution. Ethereum team please shed some light

[pipermerriam]

I just ran into this issue on a node that has never served a single RPC request. Node was fast synced 5-6 days ago and running with the primary purpose of being a --lightserv node. Here is the terminal output for the crash.

INFO [12-22|16:11:56] Imported new chain segment               blocks=1 txs=293  mgas=7.984  elapsed=1.986s    mgasps=4.020   number=4777572 hash=9c125f…4a18fe
INFO [12-22|16:12:05] Imported new chain segment               blocks=1 txs=202  mgas=7.980  elapsed=1.672s    mgasps=4.771   number=4777573 hash=745cd9…da1af3
INFO [12-22|16:12:29] Imported new chain segment               blocks=1 txs=114  mgas=7.984  elapsed=1.511s    mgasps=5.282   number=4777574 hash=047277…81db79
INFO [12-22|16:12:46] Imported new chain segment               blocks=1 txs=233  mgas=8.000  elapsed=1.902s    mgasps=4.204   number=4777575 hash=b49934…cdafad
INFO [12-22|16:12:54] Imported new chain segment               blocks=1 txs=237  mgas=7.992  elapsed=1.610s    mgasps=4.962   number=4777576 hash=09a806…3670bf
INFO [12-22|16:13:06] Imported new chain segment               blocks=1 txs=254  mgas=7.986  elapsed=1.415s    mgasps=5.643   number=4777577 hash=313912…a0b970
WARN [12-22|16:13:09] No etherbase set and no accounts found as default
INFO [12-22|16:13:09] Starting peer-to-peer node               instance=Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9
INFO [12-22|16:13:09] Allocated cache and file handles         database=/root/.ethereum/geth/chaindata cache=2048 handles=1024
Fatal: Error starting protocol stack: missing block number for head header hash

[jmozah]

I too ran in the same issue yesterday while my node was syncing.
I did a "geth removedb" and restarted sync.
but got the same issue again, this time with a stack trace

INFO [01-02|12:55:03] Imported new block receipts              count=179  elapsed=24.906ms  bytes=838638  number=2951337 hash=cf7426…c1b4e9 ignored=0
INFO [01-02|12:55:04] Imported new block headers               count=192  elapsed=36.152ms  number=2972790 hash=481fb3…774b2c ignored=0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa5f3c8]

goroutine 2724 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders.func1(0xc427343cb8, 0xc42006d1e0, 0x49dfce)
        /home/ubuntu/go/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1171 +0x2b8
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc42006d1e0, 0x2d5c77, 0xc4407ff4c0, 0x18436c0, 0xc4201bb3d0)
        /home/ubuntu/go/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1196 +0x1680
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0x8, 0xffe2e0)
        /home/ubuntu/go/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:477 +0x40
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc43a46d320, 0xc43a4d0660, 0xc42025aae0)
        /home/ubuntu/go/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:500 +0x51
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        /home/ubuntu/go/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:500 +0xc7
INFO [01-02|12:55:05] Starting peer-to-peer node               instance=Geth/v1.7.3-stable/linux-amd64/go1.9.2
INFO [01-02|12:55:05] Allocated cache and file handles         database=/nodes/datadir/mainnet/ethereum/geth/chaindata cache=128 handles=1024
Fatal: Error starting protocol stack: missing block number for head header hash

[tnguyen69]

After starting P2P node #1 successfully, I opened another terminal to start another node as the command below

geth --port 3002 --datadir node2

Then i got an error but don't know how to resolve. Hope would have a hint.

INFO [01-03|17:27:32] Starting P2P networking
INFO [01-03|17:27:34] UDP listener up                          self=enode://4a60
ce808fe1632bf061c57005c711a779d4ebccc1927fd5ec8548b7af7330e0742b450b3da59a35ea6d
[email protected]:3002
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0xb88a07]

goroutine 103 [running]:
github.com/ethereum/go-ethereum/eth/filters.(*EventSystem).eventLoop(0xc04228424
0)
        C:/gopath/src/github.com/ethereum/go-ethereum/eth/filters/filter_system.
go:417 +0x337
created by github.com/ethereum/go-ethereum/eth/filters.NewEventSystem
        C:/gopath/src/github.com/ethereum/go-ethereum/eth/filters/filter_system.
go:112 +0x10b

[smalaichami]

you will have to --ipcdisable when trying to run multiple nodes in the same server or machine..

[karalabe]

@smalaichami You can also set different IPC paths for them explicitly.

That being said, it seems unrelated that the filter system would crash.

[tnguyen69]

Thanks @smalaichami. It works well now with the option -ipcdisable.

@karalabe: i intend to run with the option --ipcpath, however, i cannot allocate geth.ipc under node2 folder. Take a look into the terminal of running node1, i see "INFO [01-04|10:29:15] IPC endpoint opened: \.\pipe\geth.ipc". Could you help how to allocate the IPC paths?

[questionator]

Completed sync on the previous version, ethereum wallet was ok and receiving blocks.
Downloaded 1.8.0 and started it in the command prompt and waited for the rest of the blocks.
Started ethereum wallet and now i get this:

Fatal: Error starting protocol stack: missing block number for head header hash

Nice, i just found my password...

[Snerken]

I'll add my 10 cents worth to this conversation if I may. I'm working my way through an Ethereum course and had downloaded and installed the most recent versions of Ethereum Wallet and Mist. Fired up Mist this morning and all was good--even added a new account on Rinkenby network. Then I thought maybe I should be using Ethereum Wallet instead and that's when it went all pear-shaped:

Error message from Ethereum Wallet:

Node type: geth
Network: rinkeby
Platform: linux (Architecture x64)

...INFO [02-27|08:16:10] Maximum peer count ETH=25 LES=0 total=25
INFO [02-27|08:16:10] Starting peer-to-peer node instance=Geth/v1.8.1-stable-1e67410e/linux-amd64/go1.9.4
INFO [02-27|08:16:10] Allocated cache and file handles database=/home/snerx/.ethereum/rinkeby/geth/chaindata cache=768 handles=1024
INFO [02-27|08:16:10] Persisted trie from memory database nodes=355 size=65.27kB time=1.283468ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
Fatal: Error starting protocol stack: missing block number for head header hash

Error message from Mist:

Node type: geth
Network: rinkeby
Platform: linux (Architecture x64)

...INFO [02-27|08:18:19] Starting peer-to-peer node instance=Geth/v1.7.2-stable-1db4ecdc/linux-386/go1.9
INFO [02-27|08:18:19] Allocated cache and file handles database=/home/snerx/.ethereum/rinkeby/geth/chaindata cache=1024 handles=1024
Fatal: Error starting protocol stack: missing block number for head header hash

Found the solution how to update the missing block numbers:

$ mist --syncmode=light

Too easy...

[Antiarchitect]

Having this issue time to time on 1.8.3. I have Kubernetes cluster and sometimes I need to stop geth. According to #16131 even 1.8.2 should properly understand SIGTERM. But after restart I get missing block number for head header hash

[tanhuiya]

same issue here with geth 1.8.6

Fatal: Error starting protocol stack: missing block number for head header hash

[skarn01]

same prob on 1.8.19. starting message change sometime to "Genesis not found in chain" then revert to the "head header hash" one