Go-ethereum: fatal error: Concurrent map iteration and map write on super fast chain.

[reckter]

System information

Geth version:
Version: 1.8.3-unstable
Git Commit: 6deb98636e240cc51f1bf81ae57286682018379c
Architecture: amd64
Protocol Versions: [63 62]
Go Version: go1.9.2
Operating System: linux
GOPATH=
GOROOT=/usr/local/go
OS & Version: Linux/OSX (tested on both, crashed on both)
note:
this is from reckter/go-ethereum (please excuse the fast-net folder being commited, this was never intended to be released in any way, it's purely for testing huge amounts of block creation in a timely manner)

I modified Geth to always use a fixed difficulty and ignore the usual block time. This enabled sup second block time for testing. (250 blocks ps being my record)

I then use javascript with web3 to deploy a contract and interact with it. Sadly shortly (3-20 minutes) after starting the chain it crashes.

Expected behaviour

No Crash, mining the blocks and including transactions as usual.

Actual behaviour

After a while geth crashes with a go panic with the fatal error: " concurrent map iteration and map write".
This problem can be delayed, by slowing down the block speed and transaction pushing to the chain.

Steps to reproduce the behaviour

check out reckter/go-ethereum, use the following as init:

{
        "config": {
            "chainId": 148,
            "homesteadBlock": 0,
            "eip155Block": 0,
            "eip158Block": 0,
            "ethash": {
                "FixedDifficulty": "14000"
            }
        },
        "difficulty": "14000",
        "gasLimit": "207199254740991",
        "nonce": "0x0000000000000001",
        "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "alloc": {}
    }

and then interact heavily using web3 with the chain. After a while, it crashes.
Note that only mining the chain without actually sending transaction works fine at all speeds.

Backtrace

```
fatal error: concurrent map iteration and map write

goroutine 399472 [running]:
runtime.throw(0xf9d0a8, 0x26)
/usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc46a071f68 sp=0xc46a071f48 pc=0x441315
runtime.mapiternext(0xc46a072298)
/usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc46a072000 sp=0xc46a071f68 pc=0x41f5b1
github.com/ethereum/go-ethereum/core/state.(StateDB).Copy(0xc50ab28e10, 0x0)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:477 +0x388 fp=0xc46a072308 sp=0xc46a072000 pc=0x71c088
github.com/ethereum/go-ethereum/miner.(worker).pending(0xc420336000, 0x0, 0x0)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:185 +0x158 fp=0xc46a072380 sp=0xc46a072308 pc=0xa65ee8
github.com/ethereum/go-ethereum/miner.(Miner).Pending(0xc4201b8960, 0xc46a0723f8, 0x43fbe9)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/miner.go:167 +0x2f fp=0xc46a0723a8 sp=0xc46a072380 pc=0xa630af
github.com/ethereum/go-ethereum/eth.(EthApiBackend).StateAndHeaderByNumber(0xc42031ce80, 0x18462e0, 0xc472134300, 0xfffffffffffffffe, 0x1b97c40, 0x0, 0x0, 0x5bd8)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/api_backend.go:87 +0x12e fp=0xc46a072408 sp=0xc46a0723a8 pc=0xac52be
github.com/ethereum/go-ethereum/internal/ethapi.(PublicBlockChainAPI).doCall(0xc4384ee790, 0x18462e0, 0xc472134300, 0xd31fe63fb1a465ff, 0x84bd886564458524, 0x102466ec, 0xc42b2eb260, 0x3b9fc16, 0x0, 0x0, ...)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/ethapi/api.go:617 +0x163 fp=0xc46a074f40 sp=0xc46a072408 pc=0xa77cb3
github.com/ethereum/go-ethereum/internal/ethapi.(PublicBlockChainAPI).EstimateGas.func1(0x3b9fc16, 0x1)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/ethapi/api.go:708 +0xbe fp=0xc46a077850 sp=0xc46a074f40 pc=0xa819de
github.com/ethereum/go-ethereum/internal/ethapi.(PublicBlockChainAPI).EstimateGas(0xc4384ee790, 0x18462e0, 0xc472134300, 0xd31fe63fb1a465ff, 0x84bd886564458524, 0x102466ec, 0xc42b2eb260, 0x3b9fc16, 0x0, 0x0, ...)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/ethapi/api.go:717 +0xf9 fp=0xc46a0778e0 sp=0xc46a077850 pc=0xa78709
runtime.call256(0xc420f0d290, 0xc420f1af18, 0xc4200a8370, 0x98000000b0)
/usr/local/go/src/runtime/asm_amd64.s:512 +0x55 fp=0xc46a0779f0 sp=0xc46a0778e0 pc=0x4701e5
reflect.Value.call(0xc420348680, 0xc420f1af18, 0x13, 0xf79b94, 0x4, 0xc46ad6c000, 0x3, 0x4, 0xc4bfa47510, 0xc46ad6c000, ...)
/usr/local/go/src/reflect/value.go:434 +0x905 fp=0xc46a077cc8 sp=0xc46a0779f0 pc=0x4d0b05
reflect.Value.Call(0xc420348680, 0xc420f1af18, 0x13, 0xc46ad6c000, 0x3, 0x4, 0x1, 0x1, 0xc420f9d020)
/usr/local/go/src/reflect/value.go:302 +0xa4 fp=0xc46a077d30 sp=0xc46a077cc8 pc=0x4d00e4
github.com/ethereum/go-ethereum/rpc.(Server).handle(0xc420f3fd40, 0x18462e0, 0xc472134300, 0x184af80, 0xc4201b83c0, 0xc421181200, 0x1009d58, 0x558b5678b19e1e63, 0x838823)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:311 +0x6ce fp=0xc46a077ed8 sp=0xc46a077d30 pc=0x831e5e
github.com/ethereum/go-ethereum/rpc.(Server).exec(0xc420f3fd40, 0x18462e0, 0xc472134300, 0x184af80, 0xc4201b83c0, 0xc421181200)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:333 +0x1ba fp=0xc46a077f48 sp=0xc46a077ed8 pc=0x832a5a
github.com/ethereum/go-ethereum/rpc.(Server).serveRequest.func2(0xc4b672a370, 0xc420f3fd40, 0xc5219960a0, 0x184af80, 0xc4201b83c0, 0xc4201c61c8, 0x1, 0x1, 0xc46f678c00)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:206 +0xad fp=0xc46a077f98 sp=0xc46a077f48 pc=0x83888d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc46a077fa0 sp=0xc46a077f98 pc=0x472881
created by github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:201 +0x2ab

goroutine 1 [chan receive, 10 minutes]:
github.com/ethereum/go-ethereum/node.(Node).Wait(0xc4201fa780)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:539 +0x7f
main.geth(0xc42023a840, 0x1007df0, 0xb2d05e00)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:219 +0x51
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0xe088a0, 0x10092c0, 0xc42023a840, 0xc42023a840, 0xc4204b9f40)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xd2
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(App).Run(0xc420258000, 0xc4200100a0, 0xa, 0xa, 0x0, 0x0)
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x635
main.main()
/home/reckter/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:207 +0x55

[...] Can deliver full stacktrace as needed, but as it is over 1000 lines long, I opted to cut out a lot here
````

[karalabe]

Good catch, will look into ti, thanks!

[3h-william]

hi guys , The same problems happened in my test network , is there any update ?

[610265158]

hi guys, i got the same problem.

[zone-a]

i got the same problem too

[xindebug]

`fatal error: concurrent map iteration and map write

goroutine 102778 [running]:
runtime.throw(0xfdd57f, 0x26)
/usr/lib/golang/src/runtime/panic.go:596 +0x95 fp=0xc422d659a0 sp=0xc422d65980
runtime.mapiternext(0xc422d65cd8)
/usr/lib/golang/src/runtime/hashmap.go:737 +0x7ee fp=0xc422d65a50 sp=0xc422d659a0
github.com/ethereum/go-ethereum/core/state.(StateDB).Copy(0xc42b0130e0, 0x0)
/root/eth/go-ethereum-1.8.2/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:477 +0x4fd fp=0xc422d65d48 sp=0xc422d65a50
github.com/ethereum/go-ethereum/miner.(worker).pending(0xc420092a00, 0x0, 0x0)`

1.8.2-stable

[zone-a]

I am not familiar with the code and try to modify it based on the crash information. Fortunately, it is useful.

VERSION:
1.8.9-unstable-f6bc65fc

line 315：

self.dbwMu.Lock()
stat, err := self.chain.WriteBlockWithState(block, work.receipts, work.state)
self.dbwMu.Unlock()

line 525：

self.dbwMu.Lock()
self.snapshotState = self.current.state.Copy()
self.dbwMu.Unlock()

[xindebug]

@karalabe we need you

[xindebug]

self.dbwMu undefined in miner/worker.go

@zone-a can you post more info? such as the whole file. tks!

[benbjohnson]

I pushed a fix: https://github.com/ethereum/go-ethereum/pull/16754

[reckter]

I tested this with the fix of #16754 And it seems to work now.
I did not let it run for multiple hours, as my pc gets quite unusable after a while running this setup, but it happily runs this scenario for about 40 minutes now. Before the fix it would never reach more then 15.

Thanks @benbjohnson!

[karalabe]

Fixed a while back Please open a new issue if you still see this.