Go-ethereum: panic: ethash_full_new IO or memory error

Created on 2 Aug 2015 · 28Comments · Source: ethereum/go-ethereum

geth crashes when I try to start mining. I'm running geth in a ubuntu 14.04 virtualbox vm with win7 host. Had no problems with account creation, genesis block generation, orrunning geth without mining.

When I run geth attach and then miner.start(1), I get panic: ethash_full_new IO or memory error immediately during initial DAG generation. ubuntu has 3GB memory and is geth crashing with over 2GB available.

Crash dump

I0801 18:51:29.903104    6114 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 11.644154ms. #18843 [0d13249c / 0d13249c]
I0801 18:51:29.977351    6114 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 6.943205ms. #18842 [5e05c444 / 5e05c444]
I0801 18:51:39.060575    6114 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 7.128135ms. #18843 [0d13249c / 0d13249c]
I0801 18:51:39.773635    6114 backend.go:642] Automatic pregeneration of ethash DAG ON (ethash dir: /home/ryan/.ethash)
I0801 18:51:39.773837    6114 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0801 18:51:39.777348    6114 worker.go:503] commit new work on block 18844 with 0 txs & 0 uncles. Took 3.340334ms
I0801 18:51:39.780887    6114 ethash.go:202] Generating DAG for epoch 0 (0000000000000000000000000000000000000000000000000000000000000000)
I0801 18:51:39.782935    6114 backend.go:649] checking DAG (ethash dir: /home/ryan/.ethash)
I0801 18:51:39.783706    6114 backend.go:663] Pregenerating DAG for epoch 1 (full-R23-290decd9548b62a8)
I0801 18:51:39.784114    6114 ethash.go:202] Generating DAG for epoch 1 (290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563)
I0801 18:52:06.069736    6114 ethash.go:230] Still generating DAG: 0%
panic: ethash_full_new IO or memory error
// See crash dump link

Source

ryanberckmans

Most helpful comment

Switching from a 32-bit to 64-bit geth binary worked for me (Windows 10).

Miretz on 26 Mar 2017

👍2

All 28 comments

@LefterisJP ping

obscuren on 2 Aug 2015

It is generating two DAGs at the same time. @ryanberckmans please run geth with --autodag=false.

fjl on 2 Aug 2015

@fjl Crashes with geth --autodag=false

$ geth --autodag=false
I0801 22:15:43.592695    3022 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/blockchain
I0801 22:15:43.596753    3022 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/state
I0801 22:15:43.601285    3022 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/extra
I0801 22:15:43.605462    3022 backend.go:290] Protocol Versions: [61 60], Network Id: 1
I0801 22:15:43.606165    3022 backend.go:319] Blockchain DB Version: 3
I0801 22:15:43.606810    3022 chain_manager.go:232] Last block (#19555) 7c16f4404ed8d968d5a204ae40398866dbb8e6be4304a52729824c415af8e3ef TD=10143245621657733
I0801 22:15:43.631603    3022 cmd.go:122] Starting Geth/v1.0.0/linux/go1.4.2
I0801 22:15:43.631764    3022 server.go:311] Starting Server
I0801 22:15:43.643622    3022 nat.go:111] mapped network port udp:30303 -> 30303 (ethereum discovery) using NAT-PMP(192.168.1.1)
I0801 22:15:43.651175    3022 udp.go:205] Listening, enode://ff3bc2c1a70820da489528f22c229a1ef6ed50b1c4371dcca465fe2c130728f153fa81712f2e5531a401e8b97f146f28a88593304e9ee5de5e8401a44e7ab402@24.185.187.91:30303
I0801 22:15:43.651327    3022 backend.go:555] Server started
I0801 22:15:43.651560    3022 server.go:548] Listening on [::]:30303
I0801 22:15:43.651702    3022 ipc_unix.go:76] IPC service started (/home/ryan/.ethereum/geth.ipc)
I0801 22:15:43.664793    3022 nat.go:111] mapped network port tcp:30303 -> 30303 (ethereum p2p) using NAT-PMP(192.168.1.1)
I0801 22:15:50.991001    3022 downloader.go:278] Block synchronisation started
I0801 22:15:54.963343    3022 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.374710458s. #19555 [7c16f440 / 7c16f440]
I0801 22:15:55.003165    3022 chain_manager.go:650] imported 4 block(s) (0 queued 0 ignored) including 0 txs in 39.71475ms. #19559 [b9126596 / adee6647]
I0801 22:16:03.391918    3022 backend.go:642] Automatic pregeneration of ethash DAG ON (ethash dir: /home/ryan/.ethash)
I0801 22:16:03.392058    3022 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0801 22:16:03.395731    3022 worker.go:503] commit new work on block 19560 with 0 txs & 0 uncles. Took 3.592969ms
I0801 22:16:03.398369    3022 backend.go:649] checking DAG (ethash dir: /home/ryan/.ethash)
I0801 22:16:03.399619    3022 ethash.go:202] Generating DAG for epoch 0 (0000000000000000000000000000000000000000000000000000000000000000)
I0801 22:16:03.399938    3022 backend.go:663] Pregenerating DAG for epoch 1 (full-R23-290decd9548b62a8)
I0801 22:16:03.401224    3022 ethash.go:202] Generating DAG for epoch 1 (290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563)
I0801 22:16:23.018005    3022 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 22.814486ms. #19560 [f3336789 / f3336789]
I0801 22:16:23.022405    3022 worker.go:503] commit new work on block 19561 with 0 txs & 0 uncles. Took 731.742µs
I0801 22:16:23.101884    3022 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.451811ms. #19559 [adee6647 / adee6647]
I0801 22:16:23.762388    3022 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 4.647882ms. #19560 [f3336789 / f3336789]
I0801 22:16:26.611881    3022 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 21.879919ms. #19561 [e2fcd725 / e2fcd725]
I0801 22:16:26.613927    3022 worker.go:503] commit new work on block 19562 with 0 txs & 0 uncles. Took 803.843µs
I0801 22:16:26.696073    3022 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 11.45731ms. #19560 [f3336789 / f3336789]
I0801 22:16:27.475728    3022 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 4.131715ms. #19561 [e2fcd725 / e2fcd725]
I0801 22:16:30.933568    3022 ethash.go:230] Still generating DAG: 0%
panic: ethash_full_new IO or memory error

ryanberckmans on 2 Aug 2015

Hm, it still appears to be generating a DAG. Could you pre-generate the DAG with geth makedag 0?

obscuren on 2 Aug 2015

Seems like an out of memory error while generating the DAG. That would indeed happen if you have insufficient memory and try to mmap memory for 2 DAGs.

LefterisJP on 2 Aug 2015

The vm has 3GB of memory. Same crash after geth makedag 0. Pardon that I'm new and am not sure what to try.

I0802 10:08:53.888756    8292 backend.go:642] Automatic pregeneration of ethash DAG ON (ethash dir: /home/ryan/.ethash)
I0802 10:08:53.889132    8292 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0802 10:08:53.892872    8292 worker.go:503] commit new work on block 22199 with 0 txs & 0 uncles. Took 3.273909ms
I0802 10:08:53.893987    8292 backend.go:649] checking DAG (ethash dir: /home/ryan/.ethash)
I0802 10:08:53.894489    8292 backend.go:663] Pregenerating DAG for epoch 1 (full-R23-290decd9548b62a8)
I0802 10:08:53.895023    8292 ethash.go:202] Generating DAG for epoch 1 (290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563)
I0802 10:08:53.896855    8292 ethash.go:202] Generating DAG for epoch 0 (0000000000000000000000000000000000000000000000000000000000000000)
I0802 10:08:57.831586    8292 ethash.go:219] Done generating DAG for epoch 0, it took 3.934716229s
I0802 10:09:00.391328    8292 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 13.112483ms. #22199 [eb4f3064 / eb4f3064]
I0802 10:09:00.392416    8292 worker.go:503] commit new work on block 22200 with 0 txs & 0 uncles. Took 663.043µs
I0802 10:09:00.956174    8292 chain_manager.go:650] imported 0 block(s) (0 queued 2 ignored) including 0 txs in 7.990281ms. #22199 [4439942f / eb4f3064]
I0802 10:09:12.859426    8292 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 24.514535ms. #22200 [5e2a4b8c / 5e2a4b8c]
I0802 10:09:12.861420    8292 worker.go:503] commit new work on block 22201 with 0 txs & 0 uncles. Took 1.479858ms
I0802 10:09:13.256982    8292 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.639258ms. #22199 [eb4f3064 / eb4f3064]
I0802 10:09:13.333206    8292 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.902378ms. #22200 [5e2a4b8c / 5e2a4b8c]
panic: ethash_full_new IO or memory error

ryanberckmans on 2 Aug 2015

After geth makedag 0 ~/.ethash, I ran geth --autodag=true which generated the epoch 1 dag.

$ ls -lh ~/.ethash
total 2.1G
-rw-rw-r-- 1 ryan ryan 1.0G Aug  2 09:57 full-R23-0000000000000000
-rw-rw-r-- 1 ryan ryan 1.1G Aug  2 10:45 full-R23-290decd9548b62a8

Now it crashes in two different ways.

With autodag=true, fatal error: unexpected signal during runtime execution [signal 0xb code=0x1 addr=0x1 pc=0x8050fc8]

$ geth --autodag=true
I0802 10:46:52.492183    8827 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/blockchain
I0802 10:46:52.500881    8827 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/state
I0802 10:46:52.510511    8827 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/extra
I0802 10:46:52.515429    8827 backend.go:290] Protocol Versions: [61 60], Network Id: 1
I0802 10:46:52.515972    8827 backend.go:319] Blockchain DB Version: 3
I0802 10:46:52.516831    8827 chain_manager.go:232] Last block (#22347) 162aaf76f857eae8acde6795dac2a43de5f979635db9344e430447597d301b35 TD=12934789019999065
I0802 10:46:52.572103    8827 cmd.go:122] Starting Geth/v1.0.0/linux/go1.4.2
I0802 10:46:52.572461    8827 server.go:311] Starting Server
I0802 10:46:52.580700    8827 udp.go:205] Listening, enode://ff3bc2c1a70820da489528f22c229a1ef6ed50b1c4371dcca465fe2c130728f153fa81712f2e5531a401e8b97f146f28a88593304e9ee5de5e8401a44e7ab402@24.185.187.91:30303
I0802 10:46:52.580855    8827 backend.go:555] Server started
I0802 10:46:52.581251    8827 ipc_unix.go:76] IPC service started (/home/ryan/.ethereum/geth.ipc)
I0802 10:46:52.581333    8827 server.go:548] Listening on [::]:30303
I0802 10:46:52.581569    8827 backend.go:642] Automatic pregeneration of ethash DAG ON (ethash dir: /home/ryan/.ethash)
I0802 10:46:52.583841    8827 backend.go:649] checking DAG (ethash dir: /home/ryan/.ethash)
I0802 10:46:52.583927    8827 backend.go:663] Pregenerating DAG for epoch 1 (full-R23-290decd9548b62a8)
I0802 10:46:52.583966    8827 ethash.go:202] Generating DAG for epoch 1 (290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563)
I0802 10:46:52.609514    8827 nat.go:111] mapped network port udp:30303 -> 30303 (ethereum discovery) using NAT-PMP(192.168.1.1)
I0802 10:46:52.623930    8827 nat.go:111] mapped network port tcp:30303 -> 30303 (ethereum p2p) using NAT-PMP(192.168.1.1)
I0802 10:46:55.915024    8827 ethash.go:219] Done generating DAG for epoch 1, it took 3.331066659s
I0802 10:47:00.493974    8827 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0802 10:47:00.496314    8827 worker.go:503] commit new work on block 22348 with 0 txs & 0 uncles. Took 2.243011ms
I0802 10:47:00.500606    8827 ethash.go:202] Generating DAG for epoch 0 (0000000000000000000000000000000000000000000000000000000000000000)
I0802 10:47:00.694733    8827 downloader.go:278] Block synchronisation started
I0802 10:47:00.694881    8827 miner.go:73] Mining operation aborted due to sync operation
I0802 10:47:02.651229    8827 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0802 10:47:02.652320    8827 worker.go:503] commit new work on block 22348 with 0 txs & 0 uncles. Took 768.193µs
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x1 pc=0x8050fc8]

With autodag=false, same memory error

$ geth --autodag=false
I0802 10:51:13.279519    8870 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/blockchain
I0802 10:51:13.283818    8870 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/state
I0802 10:51:13.293489    8870 database.go:74] Alloted 16MB cache to /home/ryan/.ethereum/extra
I0802 10:51:13.304396    8870 backend.go:290] Protocol Versions: [61 60], Network Id: 1
I0802 10:51:13.305199    8870 backend.go:319] Blockchain DB Version: 3
I0802 10:51:13.305895    8870 chain_manager.go:232] Last block (#22347) 162aaf76f857eae8acde6795dac2a43de5f979635db9344e430447597d301b35 TD=12934789019999065
I0802 10:51:13.338457    8870 cmd.go:122] Starting Geth/v1.0.0/linux/go1.4.2
I0802 10:51:13.338583    8870 server.go:311] Starting Server
I0802 10:51:13.362074    8870 udp.go:205] Listening, enode://ff3bc2c1a70820da489528f22c229a1ef6ed50b1c4371dcca465fe2c130728f153fa81712f2e5531a401e8b97f146f28a88593304e9ee5de5e8401a44e7ab402@24.185.187.91:30303
I0802 10:51:13.362208    8870 backend.go:555] Server started
I0802 10:51:13.362521    8870 ipc_unix.go:76] IPC service started (/home/ryan/.ethereum/geth.ipc)
I0802 10:51:13.366608    8870 nat.go:111] mapped network port udp:30303 -> 30303 (ethereum discovery) using NAT-PMP(192.168.1.1)
I0802 10:51:13.470117    8870 server.go:548] Listening on [::]:30303
I0802 10:51:13.528275    8870 nat.go:111] mapped network port tcp:30303 -> 30303 (ethereum p2p) using NAT-PMP(192.168.1.1)
I0802 10:51:23.471349    8870 downloader.go:278] Block synchronisation started
I0802 10:51:27.113384    8870 chain_manager.go:650] imported 1 block(s) (0 queued 1 ignored) including 0 txs in 3.336199734s. #22348 [162aaf76 / 84fa545d]
I0802 10:51:27.184468    8870 chain_manager.go:650] imported 20 block(s) (0 queued 0 ignored) including 0 txs in 69.450034ms. #22368 [ecc08d9a / ee8023c4]
I0802 10:51:32.332280    8870 backend.go:642] Automatic pregeneration of ethash DAG ON (ethash dir: /home/ryan/.ethash)
I0802 10:51:32.332336    8870 miner.go:117] Starting mining operation (CPU=1 TOT=2)
I0802 10:51:32.332991    8870 worker.go:503] commit new work on block 22369 with 0 txs & 0 uncles. Took 608.097µs
I0802 10:51:32.333590    8870 ethash.go:202] Generating DAG for epoch 0 (0000000000000000000000000000000000000000000000000000000000000000)
I0802 10:51:32.338808    8870 backend.go:649] checking DAG (ethash dir: /home/ryan/.ethash)
I0802 10:51:32.339003    8870 backend.go:663] Pregenerating DAG for epoch 1 (full-R23-290decd9548b62a8)
I0802 10:51:32.339114    8870 ethash.go:202] Generating DAG for epoch 1 (290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563)
I0802 10:51:33.582611    8870 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 14.500586ms. #22369 [b04e5a52 / b04e5a52]
I0802 10:51:33.584778    8870 worker.go:503] commit new work on block 22370 with 0 txs & 0 uncles. Took 1.267065ms
I0802 10:51:33.815626    8870 chain_manager.go:650] imported 1 block(s) (0 queued 0 ignored) including 0 txs in 27.089553ms. #22368 [a361e1df / a361e1df]
I0802 10:51:33.846959    8870 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 5.373598ms. #22368 [ee8023c4 / ee8023c4]
I0802 10:51:34.054053    8870 chain_manager.go:650] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.84279ms. #22369 [b04e5a52 / b04e5a52]
I0802 10:51:35.886896    8870 ethash.go:219] Done generating DAG for epoch 1, it took 3.54775711s
panic: ethash_full_new IO or memory error

ryanberckmans on 2 Aug 2015

@LefterisJP some new info ^. Should I give up and try another platform?

ryanberckmans on 2 Aug 2015

I am one of the C++ developer so I can't speak about the Go design. IO or memory error is quite generic. We also set the errno inside the C code so you could actually get a bit more information if here the errno is also read and printed out.

LefterisJP on 3 Aug 2015

We should change https://github.com/ethereum/ethash/blob/master/ethash.go#L208 to

var err error
d.ptr, err = C.ethash_full_new_internal(
        C.CString(d.dir),
        hashToH256(seedHash),
        dagSize,
        cache,
        (C.ethash_callback_t)(unsafe.Pointer(C.ethashGoCallback_cgo)),
)

obscuren on 3 Aug 2015

EDIT - still can't mine

Updated to ethereum 1.0.1+722SNAPSHOT20150808134323trusty-0ubuntu1, 32bit ubuntu 14.04 virtualbox guest on windows 7 64bit host.

geth --autodag=true will pregenerate dag for epoch 2. miner.start(), --mine then tries to generate dag for epoch 1 and panic: ethash_full_new IO or memory error.

Sadly, I was able to mine for awhile with miner.start(1) and can't reproduce that temporary success.

@obscuren can you suggest stuff to try? I don't think this is a memory issue, the vm has tons of memory. Even pregenerating the dags doesn't help - mining attempts to verify them or something and it crashes. The fatal msg says "IO or memory" error - maybe it is related to IO? I'm using the default ~/.ethash and have disk space available. Should I abandon geth on 32bit vm and try another platform? Thanks

ryanberckmans on 8 Aug 2015

I am also experiencing the same "panic: ethash_full_new IO or memory error" when I try to mine with geth on Raspberry Pi 2 with 4GB of swap file. Is there anything that can be done to resolve this?

vladin79 on 19 Aug 2015

Maybe related to #1686

tgerring on 23 Sep 2015

Same problem for me on Raspberry Pi

SebastianBoyd on 5 Nov 2015

I don´t have a solution but a similar problem. I ran some tests and the outcome is that it is _not_ (at least in my case) a memory issue but rather a filesystem issue.

I am running geth from within a docker container.
/ethdata/ethash is a mounted volume (if I don´t mount the volume with docker, then the issue does not occur but the DAG is not persisted)

If I run geth makedag 0 /ethdata/ethash I get the same error message than this issue.
If I run geth makedag 0 /ethdata/ (note that the path is now no longer a mounted volume), then it works fine.

chevdor on 30 Apr 2016

Hi,

I have the same problem when i launch 2 nodes at the same time on the same machine (16G!!!).

I am positively sure that it's the 2 nodes thing, because when i shut either, then i can mine with the running one. However, when both nodes are running the first node starts mining, but the other crashes.

avi613 on 14 Dec 2016

same problem on my raspberry pie ... has anyone found a solution ?

tuytuy on 15 Feb 2017

👍2

Same issue having the two nodes , trying to run miner getting memory issue. Can we specify memory before starting miner ?

MashB on 20 Feb 2017

can confirm this on ubuntu 16.04 LTS, machine with 32G memory, single node, using --testnet.

fuckyou-walledgarden on 26 Feb 2017

For me this turned out to be a memory issue. Freeing up extra space did the trick.

AN8991 on 21 Mar 2017

👍1

And the rabbit hole ends here...

PartTimeLegend on 23 Mar 2017

👍1

Switching from a 32-bit to 64-bit geth binary worked for me (Windows 10).

Miretz on 26 Mar 2017

👍2

Same issue on Raspberry Pie 3

transducer on 25 May 2017

Switching to 64-bit geth worked for me. Windows 7.

sinhashish on 23 Jun 2017

👍1

Still seeing this error,
Anyone get mine working on the Raspberry pie 3?

rajpatta on 8 Jul 2017

I'm experimenting with a Raspberry Pi3 cluster running a private network. Encountering this issue also when attempting to mine. All worked fine with a set of small (500MB RAM) VMs running 64bit Ubuntu on 64bit host. Assuming this has something to do with the 32 bit memory address space rather than the physical quantity of RAM available(?).

Edit: for some basic testing on a private network I found that adding the geth --dev flag seems to get over this problem.