Go-ethereum: Exremely slow sync ETH 1.9.8

[nikitasius]

System information

Geth version: 1.9.8
OS & Version: Debian 9x64

Expected behaviour

like this

Actual behaviour

Already more than 12h syncing.

Steps to reproduce the behaviour

1) Wipe old blockchain
2) Start geth with maxpeers 64 and cache 4096
3) let is sync

This is what i see now:

INFO [11-28|12:22:34.492] Imported new state entries               count=1657 elapsed=20.691ms  processed=249122383 pending=34439  retry=0   duplicate=4921 unexpected=27062
{
  currentBlock: 9015309,
  highestBlock: 9015434,
  knownStates: 249156822,
  pulledStates: 249122383,
  startingBlock: 9015035
}

INFO [11-28|12:25:09.563] Imported new state entries               count=1152 elapsed=12.879ms  processed=250164680 pending=54912  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:09.620] Imported new state entries               count=1152 elapsed=12.627ms  processed=250165832 pending=53961  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:09.917] Imported new state entries               count=1152 elapsed=13.185ms  processed=250166984 pending=53077  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:10.060] Imported new state entries               count=1152 elapsed=14.574ms  processed=250168136 pending=52305  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:10.100] Imported new state entries               count=768  elapsed=8.739ms   processed=250168904 pending=52166  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:10.221] Imported new state entries               count=929  elapsed=5.992ms   processed=250169833 pending=54170  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:10.452] Imported new state entries               count=1152 elapsed=10.728ms  processed=250170985 pending=54639  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:10.615] Imported new state entries               count=1152 elapsed=13.659ms  processed=250172137 pending=53830  retry=0   duplicate=5321 unexpected=29612
> eth.syncing
{
  currentBlock: 9015374,
  highestBlock: 9015443,
  knownStates: 250225967,
  pulledStates: 250172137,
  startingBlock: 9015035
}
> INFO [11-28|12:25:11.641] Imported new state entries               count=768  elapsed=8.227ms   processed=250172905 pending=53546  retry=0   duplicate=5321 unexpected=29612
INFO [11-28|12:25:11.685] Imported new state entries               count=1213 elapsed=12.715ms  processed=250174118 pending=52856  retry=0   duplicate=5321 unexpected=29612

I use fast sync. Before it synced with nice. I stopped, i removed nice CPU limiter, i gave all:

• upto 48 cores
• upto 10Gbit
• raid NVMe

it uses:

• --syncmode=fast
• --cache 4096
• --maxpeers 64
• actual folder size now is 166Gb

[nikitasius]

this is SICK. it's still syncing. It started at 20:30 UTC yesterday.
17 UTC NOW.

[nikitasius]

i increased cache to 8192, peers to 256, still syncing

[holiman]

Cache and cpu are not so important, you're syncing the state now. 300M+ entries, downloaded in chunks of about 300 at a time. So lots of tiny requests. Your network latency, speed and number of peers factor in on how fast it will be

[nikitasius]

synced (i stopped to change config):

{
  currentBlock: 9017661,
  highestBlock: 9017668,
  knownStates: 401300820,
  pulledStates: 401300820,
  startingBlock: 9017661
}

Well.. you need to rework something in ETH cause it becomes more and more hard to sync after wiping data. Server on 10 Gbit.

[passionofvc]

hi @nikitasius

I have been suffering from various ETH synchronizations.
It was solved by adding PEER manually.
goto https://gist.github.com/rfikki/ to get peer node and run admin.addPeer
or goto https://ethernodes.org/nodes to scrabble some node for your syncing.

[ang-st]

same here started almost 22H ago running Geth/v1.9.8-stable-d62e9b28/linux-amd64/go1.13.4

> eth.syncing
{
  currentBlock: 9021140,
  highestBlock: 9021232,
  knownStates: 229163418,
  pulledStates: 228959501,
  startingBlock: 9020788
}

And node can't cope with the few last blocks and states remaining

[holiman]

@ang-st blocks are fine, but as you can see from @nikitasius data above, you need 400M state entries, you're only at 229M yet. Increasing the number of peers might help. but it's the RTT of 400M/300 that is the bottleneck with the current sync algorithm

[tribycol]

Hi Nikitaisius

The sync process is done in two stages. one is the Blocks and other is the States. both takes time and if you are using HDD; it will be longer than SDD.
Increase the peer to 100 and the cache by adding --cache "4096" --maxpeers "100" to the console
try to reboot the computer every night. it also help.

[ang-st]

increased to --maxpeers 255 and --cache to 6000 still syncing.
lesson learned : only do geth mandatory updates and protect your blocks data like the jewel of the crown

[holiman]

You do know that you can totally update geth without nuking the existing data, right?

[ang-st]

I do, unfortunately one of my devops did not.
hence 48h and an unfinished fast sync ...

[nikitasius]

Hi Nikitaisius

The sync process is done in two stages. one is the Blocks and other is the States. both takes time and if you are using HDD; it will be longer than SDD.
Increase the peer to 100 and the cache by adding --cache "4096" --maxpeers "100" to the console
try to reboot the computer every night. it also help.

please read well, i use enterprise NVMes in raid, i use 10Gbit connection, 96 Gb ram and 24 cores (2x CPU 12 cores each, so 48 threads with HT).

I finished sync with 16Gb cache and 512 peers.

The problem: actual way how ETH syncing the data.

Actually node works (after being synced) with cache 4Gb and 64 peers only. But it was pain in the ass to sync it.

[tribycol]

Hi Nikitasius.

I wonder if you could please share us your general setting of GETH, I am getting all the transactions underpriced and I could not fix it. I am trying to do solo mining.

thanks in advance
these are my settings

[Eth.Miner]
Etherbase = "my wallet address"
GasFloor = 1000000
GasCeil = 999999999999
GasPrice =1000000
Recommit = 3000000000
Noverify = false

[Eth.Ethash]
CacheDir = "ethash"
CachesInMem = 2
CachesOnDisk = 3
DatasetDir = "/home/caw/.ethash"
DatasetsInMem = 1
DatasetsOnDisk = 2
PowMode = 0

[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 1
PriceBump = 10
AccountSlots = 16
GlobalSlots = 4096
AccountQueue = 64
GlobalQueue = 1024
Lifetime = 10800000000000

[nikitasius]

@tribycol hi, i shared my settings arleady in a 1st post.

[ang-st]

3 days later... I would advise users to skip this version

> eth.syncing
{
  currentBlock: 9045483,
  highestBlock: 9045590,
  knownStates: 413368871,
  pulledStates: 413366585,
  startingBlock: 9045418
}

[boy-good]

@nikitasius Hey,boy!I used 1.9.8version is also synchronisation slow!! Always a few dozen blocks behind.T_T

[nikitasius]

@boy-good i used 1.9.8 too lol (in 1st post).

i increased to 16Gb cache, 512 peers and let it sync. Server have 10Gbit in nice DC.

[boy-good]

@nikitasius How did you solve it? I switched to version 1.9.7 or the same, maybe related to the version v7 of the database?

[ang-st]

Finally i resorted to brute force mode :
spin an aws m5ad.4xlarge use v1.9.7 pass it --maxpeers=2048 --cache=32768.
Sync was done in less than 4h ... I rsynced back to my prod node.

Synced 🎉

[nikitasius]

@boy-good https://github.com/ethereum/go-ethereum/issues/20408#issuecomment-560156558

[boy-good]

@ boy-good ＃20408（评论）

My God, I don't have that high profile Machine. T_T
I feel like I'll never be able to synchronize！

[karalabe]

Closing as works as intended (albeit slow). We're working on a new sync algorithm, but it will take a while, there's a lot of moving components.