Go-ethereum: Node can't keep up with network, out of sync

[MitchK]

We are running a Geth node on AWS with good internet connection and fast storage. However, it takes us 1-3 minutes for each block to receive and we are getting more and more out of sync with the network.

We are using ropsten testnet.
We are using fast download and also increased the cache.

System information

Version: 1.7.1-stable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8
Operating System: linux
GOPATH=/truerec-geth
GOROOT=/usr/local/go

Expected behaviour

Node should sync up with Ethereum. Block times should be under a minute

Actual behaviour

Node gets more and more out of sync, Block times are 1-3 minutes.

Steps to reproduce the behaviour

We start our node like this on a Ubuntu machine:

  geth \
    --ipcdisable \
    --rpc \
    --rpcaddr "0.0.0.0" \
    --rpcport 8545 \
    --testnet \
    --datadir "./.ethereum" \
    --port 30303 \
    --rpcapi eth,web3,net \
    --syncmode fast \
    --cache=2048

[LefterisJP]

Similar behaviour for me around the latest ropsten blocks. Can't seem to catch up with ropsten anymore.
Using geth 1.7.1

https://gitter.im/ethereum/ropsten?at=59dbf50ef7299e8f53c9497c

[jackyxie]

I have the same problem in mainnet.

[lyricalpolymath]

same problem mainnet

[samlavery]

what hardware configurations are you guys running? I just lost 1/4 of my geth's to this, it should be something fixable

[MitchK]

We are still having these issues and is becoming productive show-stopper for us... We are running geth 1.7.3 in Docker on AWS in Docker with EBS. Re-sync even with fast-sync does not help. Once synced up, Geth becomes slower and slower and gets more and more behind (is: 4597593, should: 4635065)

We basically start geth inside Docker with:

geth \
  --cache 2048 \
  --networkid 1 \
  --ipcdisable \
  --rpc \
  --rpcaddr 0.0.0.0 \
  --rpcport 8545 \
  --datadir /data \
  --rpcapi "eth,web3" \
  --verbosity 3 \
  --syncmode fast

[mathi123]

Same trouble here! Sync keeps falling behind, elapsed time is bigger than block mining time.

[kiahmed]

anyone found a solution for this yet?

[mathi123]

I did the following:

• switch to a stable version of geth
• attach SSD drive instead of normal disk

sync is very fast now (fast sync in +- 5-6h)

[MitchK]

I still had the issue with a full sync, even with SSD and --cache 2048 setting with Geth 1.7.3. I am connected to around 6 nodes.

I finally fixed the issues by switching to Parity, since Geth also seems to eat a lot of memory. With Parity, I'm also able to connect to more nodes with auto discovery for some reason. Parity seems more mature/stable to me than go-ethereum.

[kiahmed]

isnt it the same parity that got $130-300 mil LOST? Honestly are you any pair promoter some sort ? no offence

[Proteles]

So my options are run Geth and not be able to sync node, which is the equivalent to no running geth at all. Constantly being 10,000+ blocks behind on Geth is as useful as not running a node at all. OR, run Parity (which actually updates and works) with huge security flaws and a worthless team who knowingly doesn't give a crap about said security flaws.

The truth is that it is pretty pathetic that the Ethereum team can't produce a viable piece of software. This whole "you need an SSD to run it" is garbage. You shouldn't need to replace your hardware. Geth should actually not suck. I run Geth on a dedicated, new install Windows 10 machine, and at best I have 2 peers.. usually 1 or 0 peers. I try to sync my blockchain, but the closer it gets to being fully synced the slower it runs until I get about 10,000-5,000 block behind current block, and then it falls back to 10,000-20,000 blocks. How is it that Geth is such a garbage application? Running 1.7.3-stable-4bb3c89d.

I've been trying nonstop for 2 weeks now, and blockchain still won't full sync. No matter what I try it won't fully sync. Honestly and sadly, in spite of their complete disregard for peoples' account security, Parity sounds like a better solution for keeping your node in sync, but the worst idea possible for keeping your money.

eth.syncing
{
currentBlock: 4658748,
highestBlock: 4665813,
knownStates: 0,
pulledStates: 0,
startingBlock: 4654092

[kiahmed]

I hear you bro..I wonder what they doing with all these billions poring into the ecosystem? This should come attention to Vitalik and core team

[PV-Patrick]

I'm experiencing the same issue... Our wallet is falling further and further behind. Has anyone tried maybe hard coding nodes? Really out of ideas... haha.

[MitchK]

isnt it the same parity that got $130-300 mil LOST? Honestly are you any pair promoter some sort ? no offence

true, that's them... all I'm saying it is synchronizing and it also has some more features than Geth. I'm still interested of getting to the root cause of this...

We had luck doing a full sync with Geth, however, it takes quite some time...

[almindor]

Same issue here on Etherwall's server with slower HDD, node keeps falling behind being obviously IO bound. I've got an update scheduled for SSD dishing additional ~25 EUR a month to keep this thing from falling over.

I can't even switch to parity as they don't have websocket support yet!
UPDATE: just noticed ws landed some time ago in parity, going to try that

[jtakalai]

@kiahmed @Proteles it's not really fair to criticize Parity client because of Parity wallet smart contract oversight. You don't have to use any of their smart contracts if you only want the Ethereum client.

My geth is behind some 70000 blocks now, and I'm seriously considering backing it up with Parity instance(s)...

[elie222]

Still no update on this. This seems like such an important item to fix.

[karalabe]

Closing this issue as it's old and many many fixes and optimizations have been merged since. The disk io was reduced enormously over the last years since this bug was opened. As for HDD, no, you still can't run Ethereum on an HDD and never will be able to, independent of client. We have a PR that permits splitting the chain database between a smaller dataset needing SSD and a larger one needing HDD. https://github.com/ethereum/go-ethereum/pull/17814 That might help a bit more in the future once its merged.