Yes .. same exact effect
Command is: geth --syncmode=fast --cache=4096 console

ghost on 28 Jun 2017

Please try geth v1.6.6.

fjl on 29 Jun 2017

in my case v1.6.6 does not fix it

n0-m4d on 30 Jun 2017

👍4

same here, described detailed status here:
https://github.com/ethereum/go-ethereum/issues/14571

pebwindkraft on 2 Jul 2017

The same for me for --testnet (ropsten) on Mac OS. The geth version is 1.6.6.
I'm running:
geth --testnet --syncmode "fast" --rpc --rpcapi db,eth,net,web3,personal --cache=1024 --rpcport 8545 --rpcaddr 127.0.0.1 --rpccorsdomain "*" --bootnodes "enode://20c9ad97c081d63397d7b685a412227a40e23c8bdc6688c6f37e97cfbc22d2b4d1db1510d8f61e6a8866ad7f0e17c02b14182d37ea7c3c8b9c2683aeb6b733a1@52.169.14.227:30303,enode://6ce05930c72abc632c58e2e4324f7c7ea478cec0ed4fa2528982cf34483094e9cbc9216e7aa349691242576d552a2a56aaeae426c5303ded677ce455ba1acd9d@13.84.180.240:30303"
and when it comes near to latest blocks (at https://ropsten.etherscan.io/) it continuously "imports new state entries". If I restart the cmd above it fetches most recent blocks but never reach latest ones.

egortsaryk9 on 4 Jul 2017

same here... the geth is 1.6.6, running geth --testnet --fast --cache=1024

porfavorite on 8 Jul 2017

I am in same state after a week using geth --fast --cache=1024. Anyone knows what should I do right now?

adhicl on 11 Jul 2017

Same situation on testnet. OSX, geth 1.6.7-stable-ab5646c5. Started with --fast and --cache 1500

thecopy on 17 Jul 2017

What I could understand from this geth fast mode problem is that:

You must use Quad core processor with 4 gb or more RAM
You must use SSD instead of HDD
Your internet connection must be at least 2mbps or more and reliable.
If all above are checked then you should try geth. Using geth in full mode you would most probably synced in a week or two max. In fast mode it depends on your luck but you are most probably synced in 2-3 days or never.

sonulrk on 17 Jul 2017

👍3 😕2

Encountering similar issues. Geth --fast sync does not sync, let alone fast.

Using 1.6.7-stable on Ubuntu, and it gets within 100 blocks and then endlessly imports state entries.

clowestab on 23 Aug 2017

👍12

@clowestab did you ever get it to sync?

alfkors on 1 Sep 2017

Having same issue. Also using ubuntu, and geth endlessly syncs.

kashkalik on 3 Sep 2017

I am having the same issue. Has anyone been able to solve this problem?

mboehler on 7 Sep 2017

Same problem here, I run geth with 1024 cache and fast syncing three days ago and, after reaching the last block number one day ago, it had never stopped the "Imported new state entries" state.

During this "state entries" phase, however, my balance changed from 0 to the correct value and the block number reported changed from 0 to the a number near the highestBlock value.

This is geth version on my ubuntu machine:

Geth
Version: 1.6.7-stable
Git Commit: ab5646c532292b51e319f290afccf6a44f874372
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8.1
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.8

and this is the result now every half second:
...
INFO [09-08|10:05:34] Imported new state entries count=11 flushed=7 elapsed=216.807ms processed=25170515 pending=1852 retry=1 duplicate=5293 unexpected=6261
INFO [09-08|10:05:34] Imported new state entries count=2 flushed=5 elapsed=521.989µs processed=25170517 pending=1847 retry=0 duplicate=5293 unexpected=6261
INFO [09-08|10:05:34] Imported new state entries count=7 flushed=1 elapsed=263.738ms processed=25170524 pending=1862 retry=2 duplicate=5293 unexpected=6261
INFO [09-08|10:05:34] Imported new state entries count=2 flushed=4 elapsed=6.934ms processed=25170526 pending=1859 retry=0 duplicate=5293 unexpected=6261
INFO [09-08|10:05:34] Imported new state entries count=1 flushed=0 elapsed=27.828ms processed=25170527 pending=1861 retry=1 duplicate=5293 unexpected=6261
INFO [09-08|10:05:34] Imported new state entries count=5 flushed=5 elapsed=19.440ms processed=25170532 pending=1860 retry=0 duplicate=5293 unexpected=6261
...

this is the result of eth.syncing and other geth tools:

eth.syncing
{
currentBlock: 4249131,
highestBlock: 4250814,
knownStates: 25172364,
pulledStates: 25170517,
startingBlock: 0
}

net.peerCount
25

eth.blockNumber
4244762

The ether balance of my wallet is not 0 and is reported correcly and updated to the last transaction made one day ago.

How long is this state supposed to last?

brennino on 8 Sep 2017

Another information, my .ethereum folder size is currently 41 GB, maybe a little too large for a right fast sync.

brennino on 8 Sep 2017

I think never.

sonulrk on 8 Sep 2017

UPDATE: I stopped geth with CTRL-D and reopen it. Now it seems that the "Imported new state entries" phase halted and geth is working correctly updating only new blocks.
It seems that the problem is that fast sync continue forever to download states and is not aware that the blockchain has already in a right, consistent state.

So, for now until the issue is solved this is my advise:

Start geth fast sync
wait until the command eth.syncing report a currentBlock near the highestBlock
After that, every few hours run the command: eth.blockNumber ... firstly it returns probably 0 but continue to wait.
When eth.blockNumber returns a value different from 0 and near the currentBlock value close geth with (CTRL-D or with the command "exit" so it can close correclty and under control) and wait until the program closes in the right way and your operating system shell comes back
Reopen geth with fast option. You will see the warning "Blockchain not empty, fast sync disabled"... this is the correct behavior, is telling you that fast sync has been finished.
Now the "Imported new state entries" messages disappear and you just see this messages every few seconds:
INFO [09-08|21:12:06] Imported new chain segment blocks=1 txs=131 mgas=5.379 elapsed=13.855s mgasps=0.388 number=4251422 hash=697652…d85ce8

Now your problem has been solved and probably geth avoid to download a lot of states, reducing the hard disk space taken by geth too.

This is valid until the issue will be solved and geth will become aware when the blockchain is correctly synced.

This is my experience, maybe work, maybe not. For me it worked.
Hope this helps,
Marco

brennino on 8 Sep 2017

👍8

Congrats but I had run it for more than 8 hours and eth.blockNumber had shown 6 always. I have to change to parity to sync the blockchain.
Edit: Fast sync worked at first run only on consecutive run you have to run in full mode or geth automatically give warning "Blockchain not empty, fast sync disable" and continue with full mode.

sonulrk on 8 Sep 2017

I think you have just to wait until eth.blockNumber shows a number near currentBlock before close it and start it again.
I forget to tell to remove old blockchains before starting the task I told before.
Yes, fast sync can be used with an empty blockchain and only the first time.
The command for clear the whole blockchain is "geth removedb" on the operating system shell, it removes everything has been downloaded before.
After that you are able to start a fast sync again from an empty blockchain and follow the procedure I told in my prev post hoping it works.

I'm not a geth developer, I just use it so I can't solve the problem or tell you what it does internally or why the command returns you "6" and what you have to do, but it seems that it downloads a lot of states and, when it finds the head state it's able to build the full blockchain. For me this happened when geth.syncing showed a "knownStates" near : 20.000.000 but it can happen before or after.

During my test, after fast sync finished to download all the blocks headers, it takes more than 24 hours more for having eth.blockNumber = 4244762 . I run geth on a server in with a band of 100 Mb/s.

When it showed to me "0" I let it doing the work and after 24 hours I see the command returns 4244762. I haven't tried to run the command in the middle so I don't know if the command returns other numbers before reaching the last block.

I have never used parity but is seems good and use less disk space than geth so it worth a try.
Maybe some geth dev can make things more clear.

brennino on 9 Sep 2017

👍1

We believe this is fixed on the master branch. Fast sync takes a while (especially with the mainnet), but will terminate eventually.

fjl on 10 Sep 2017

@brennino
My eth.blockNumber shows 0 after almost several days sync. I am wondering whether the fast sync will fail if I stop and restart the sync process in middle for several times.

vincentvc on 12 Sep 2017

@vincentvc fast sync in geth only works when the database is empty thus you get one chance to fast sync and then it will be the full sync after that _[STATED IN ERROR THE FOLLOWING- THIS IS INCORRECT AS POINTED OUT BY FJL: thus yes if you stop and restart anytime before that first fast sync finishes you won't do a fast sync from that point.]_ My experience with the scenario listed here was two fold. I used the latest build off of main and I made sure I had my database on an SSD. doesn't seem like the SSD vs HDD thing would matter so much but in my experience, until I put it on the SSD I could never get that first sync to finish up - not to say that is true for everyone just my experience.

manicprogrammer on 12 Sep 2017

👍1

I'm having the same problem, continuous imported state I'm currently trying as @brennino says and will come back with result later... currently 350k states processed

here some info :

> eth.syncing
{
  currentBlock: 4269853,
  highestBlock: 4270000,
  knownStates: 357664,
  pulledStates: 348163,
  startingBlock: 4268019
}

net.peerCount 10

eth.blockNumber 0
update : almost 24h later here's the number
blockNumber : 0

eth.syncing
{
currentBlock: 4270728,
highestBlock: 4270793,
knownStates: 6879452,
pulledStates: 6875584,
startingBlock: 4268019
}

imported state still going, gonna check back tomorow...

skarn01 on 13 Sep 2017

hi @skarn01 I don't know if this happen only to me but when I start fast sync with an empty block chain starting block always shows 0. You can see my eth.syncing on my previous post that I report here:

eth.syncing
{
currentBlock: 4249131,
highestBlock: 4250814,
knownStates: 25172364,
pulledStates: 25170517,
startingBlock: 0
}

Maybe something wrong happen during fast sync or you close geth before eth.blockNumber says a number near last block. Blockchain sync is really time consuming and you haven't to stop fast sync until finishes or eth.blockNumber != 0 and near highestBlock.

What can I tell for helping you... it seems you are not starting fast sync from an empty block chain so if I'm in you I will start again from the beginning.

If you don't want to start again (and I can understand you, I went crazy for days before having some results...) you have this opportunities:

1 - If you want just to make a transaction because ethereum are falling in price now and you are in panic you can use light sync for syncing, make your transaction and, when you stop praying and shout in your room for a price peak, try with calm to sync your blockchain with fast sync again.
2 - wait two days more with your current situation and see if something changes.

About point number 1, I think light sync is not an experimental feature any more (but maybe someone else can confirm)... and I succeed to make a transaction with a light sync blockchain without problems.
If you want to start again with point number 2 later, you can close geth (because your situation is already corrupted) and rename your blockchiain directory .ethereum/geth. If you are ok instead for clear your current blockchain just use geth removedb on the operating system shell.
After that, for starting light sync (different from fast sync!), I have started geth with the command:
geth --light --cache=1024
and wait just 2 or 3 hour for download a 600Mb light blockchain. After that you can make your transaction. Again this is just my experience for helping people, no responsabilities if you lose your ethereum.
Hope it helps
Marco

brennino on 15 Sep 2017

hi @brennino, you're right that strange that my starting block is not 0 as i haven't stop the geth daemon...

what i want to do is develop my own services using the chain ( got experience creating an ICO for my boss, now i got some interest in that technology ^^ ) so i don't worry for money i currently put here, there's currently none, ahah.

i'll try the --light option after the geth removedb. --light still give me possibility to work with the chain and see full block after the first sync?

Thank you and I'll come back with update on my situation.

skarn01 on 15 Sep 2017

I have tried -light option to check if it is experimental or not. I had used Geth 1.70 and in output it said light mode is experimental feature.

sonulrk on 15 Sep 2017

fast sync in geth only works when the database is empty thus you get one chance to fast sync and then it will be the full sync after that thus yes if you stop and restart anytime before that first fast sync finishes you won't do a fast sync from that point.

That's not quite true. Fast sync will continue after restart.

fjl on 15 Sep 2017

👍5

The original issue should be solved in the 1.7.0 release. If you fast sync with the release, it should stop eventually.

fjl on 15 Sep 2017

@fjl is right in v1.7.0 fast mode continue from the last receipt after restart. I have started sync with geth 1.7.0 from scratch. This is current system:
Hardware: i5 2400, 16gb, hdd
(deliberately use hdd instread of ssd to check the results)
Softwares: Windows x64, geth 1.7.0
Network: 512kbps FTTH
Command: geth --syncmode "fast" cache 1024

Let's see what happens.

Edit: geth is fully synced but minimum network speed should be 1 mbps. For initial 20 days it was running on 512kbps but it was not enough. So I increase the network speed to 1mbps and in 8 days it synced from 2.1 million to latest block. I am closing this issue because the initial problem of unbounded importing state entries is resolved in version 1.7.0. @nabeelamjad and @fjl are correct in their approach and information.

sonulrk on 16 Sep 2017

@sonulrk any update? I have the same problem as you. Current state:

eth.syncing
{
currentBlock: 4282458,
highestBlock: 4282683,
knownStates: 18160211,
pulledStates: 18122951,
startingBlock: 0
}
Running for about 3 hours now

adikancherla on 17 Sep 2017

I'm experiencing very similar problem using Geth1.7 & Mist 0.9.0 on Win10 Home
However for me eth.syncing returns False and blockNumber returns 0

Started running Geth 1.7 yesterday afternoon after doing removedb and it completed sometime this morning, but wouldn't run the app (Mist)

I'm newbie just chipping in, (not a comprehensive posting .... but if problem persists I'll post more)

NicholasB54 on 17 Sep 2017

Seeing the same problem on Geth 1.7.0

> eth.blockNumber
0
> eth.syncing
{
  currentBlock: 4285965,
  highestBlock: 4286240,
  knownStates: 512791,
  pulledStates: 506078,
  startingBlock: 4273567
}

vogelito on 18 Sep 2017

👍1

--light work well, but i want a fast sync has i can get full block, so i've try another time from scratch and geth got killed at 3.8M block i dont know why, i'm gonna upgrade to 1.7.0 and post what happen with my next try...

update : same prob, now syncing say false, blocknumber say 0, and i've got 13.5 M state processed and still going, I gonna post feedback later...

using geth 1.7.0

update2 : just been able to do a sync without problem (got some out of memory error, so i've close everything else while it was syncing) but still can't finish to sync... that one never stoped but the starting block doesn't show blocknumber 0...

eth.syncing
{
currentBlock: 4312845,
highestBlock: 4313035,
knownStates: 12114361,
pulledStates: 12101760,
startingBlock: 4312790
}
eth.blockNumber
0

i'm gonna resign to use light mode for now, in hope everything I need will be there.

skarn01 on 18 Sep 2017

to give you feedback.

i've update to geth 1.7.0, adjust my setting so there was no memory error ( got too many thing on my server, closed some of them) and followed @brennino procedure.

After a while, it was near completion (about 100-200 block away) and started to import states only. eth.sync stoped showing startblock:0 and was sometime restarting to sync the last hundred of block needed. At about 28M states processed, it stop importing states and started saying "new chain segment". So i've let it go until it finally got the last block (sync was sometime temporary stoping and took some minute to see that it was late and started again to sync, switching eth.sync from stats to false to stats...). At the last block, after eth.sync was saying false, and blocknumber was still going up and showing the same blocknumber as etherscan, i've stop geth and restart it.

Now everything seem fine. Took about 2-3days to do a fast sync. Approx half a day to 1 day for the first sync, 1day to import state, and some hour to finish syncing.

Now i hope my computer won't blow for the next few month, I don't want to start syncing again, ahahah.

Good luck to everyone else

skarn01 on 27 Sep 2017

this eventually worked, but it was painful for sure.

vogelito on 29 Sep 2017

Maybe you can try the light mode on the next time you need to start it from scratch!

https://github.com/ethereum/mist/issues/3097 🎉 and discussion here: https://github.com/ethereum/go-ethereum/issues/15001

wtfiwtz on 29 Sep 2017

👍1

I was away for a while and decided to do a fast sync to save some disk space by starting fresh, but ran pretty much in the same issue as described in this thread, here's what I did to solve it (using Geth 1.7.0):

Ensure you have an empty data directory, otherwise run your usual geth command followed with removedb at the end once to remove your DB.

Run geth --cache 2048 --datadir "D:/my/data/dir" (remove the -datadir argument if you want the default directory, the use of fast command wasn't required in 1.7 as it automatically assumes it)
When blockchain is nearing the highestBlock (about 100-150 blocks away) it will start importing a lot of state entries. Please let these complete, as of today this will be around 31 million state entries ( they go really fast).
Once all state entries are completed it will automatically start dropping stale peers, when this happens close geth entirely.
Repen geth using geth --syncmode "full" --cache 2048 --datadir "D:/my/data/dir" (syncmode is probably irrelevant)
Let it run for a while, it will reindex the chain and eventually auto-disable fast mode and switch to full mode. Took about 10 minutes for me during this stage.

Using the above settings, an SSD and a 220 Mbps download speed internet it took me roughly 1.5 hours to synchronize the blockchain from 0% to 99%, then the state entries took a good 3-4 hours before it finally finished. I think the keypoint is to let the state entries run until its completion, don't cut it off early. There's over 30 million state entries so it will take a while. Total blockchain size: ~26 GB from 0 to block 4351952 using the above steps.

I realize there's a light mode available, but it just doesn't cut it for me. I've tried it and it is very slow when submitting transactions (doesn't get picked up for a while or not at all when looping through 10-20 sendTransaction calls at once using web3, whereas the fast/full client picks them up instantly and I can see them on Etherscan).

Hope this helps.

nabeelamjad on 10 Oct 2017

👍17 ❤1

I suppose it is a wise choice you turn your geth syncmode to "full" mode.

Roaid on 3 Nov 2017

Just completed an import (--syncmode "fast" --cache=2048). "Imported new chain segment went" up to 14.7M in the process. Took about 6 hours. 38.4GB on disk.

maverikou on 7 Dec 2017

sigh, all that is required is just some simple explanation of what 'fast' is doing(by the team), I was puzzled by what happened as well(and all those weird value when importing states, like ignored, duplicate what are they)

It seems what fast sync really does is :

download block headers all the way to almost as close as latest(4.7M as of Dec 15 2017)
download all states(but no verification) all the way to the latest(or may be close to)
since each block can contain multiple transactions(?) the # of state which is effectively a database transaction log can be much larger I think 48M as of Dec 15 2017
the 'processed=' is referring to the state record during the import state stage. So have to wait till this reaches 48M(in my case) and can be a gauge to guess when it will be done

Is this guess/interpretation correct ?

garyng2000 on 15 Dec 2017

👍5

still not working

ghost on 21 Dec 2017

Did it ever finish?

xstalpha on 28 Dec 2017

I can't get it to work either. I went into the geth directory and manually deleted the chain store folder using windows explorer. I then ran the command: "%APPDATA%Ethereum WalletbinariesGethunpackedgeth.exe" geth --syncmode light

And all I currently see in CMD is just "imported new block headers" and "imported new block entries". Is this working correctly?

HeerHub on 28 Dec 2017

you would need to wait till you see import new state entries up until 50M(at least) before it starts the 'normal' full block sync for the remaining thousand or so.
Given today's size, expect at least 2-3 days. Parity may be faster but never had an opportunity to do that to completion.

garyng2000 on 28 Dec 2017

for light mode you should see the number to about 4.8M(as of today) and it is basically done

try 'geth attach' then 'eth.syncing' to check for the status

garyng2000 on 28 Dec 2017

will update if it syncs

{
currentBlock: 4815023,
highestBlock: 4815321,
knownStates: 36384431,
pulledStates: 36381874,
startingBlock: 4810860

xstalpha on 29 Dec 2017

still have a while, the latest state is at least 50M and your's is only 36M

garyng2000 on 29 Dec 2017

Yup. I ran Geth on Windows a few months ago and have let the node run since, but am migrating the node to a Linux machine. I keep seeing people asking this question repeatedly so if it does sync at ~50m I'll update and let everyone know it does actually work lol

xstalpha on 29 Dec 2017

I just created a new node in 'fast' starting yesterday and it is almost done now. again just some reference point for those who may need it:

As of Dec 28 2017:
latest block was 4.8M, transaction state about 58M.
it took me 26 hours or so to have it sync up and it is in the final stage(already switched to full node equivalent as that is the design of fast sync), about 300 block left
the average disk usage is about 10-20MBps during the importing state stage and about 1-2Mbps in terms of network bandwidth
I have to kill it once as it stalled for about an hour during the early importing state stage but it does know where to pick up again(so still sort of fast sync even after the restart). I think this is an improvement in 1.73
the total disk space used is 49GB

hope this help others in the future

edit:
and it is fully sync now so this is now a node I can use for submitting transaction etc. about 5 minutes more when I first created this comment

garyng2000 on 29 Dec 2017

I left geth 1.73 running for about 15 hours
geth --rpc --syncmode "fast" --cache 3096 --maxpeers 50
I was about 500 blocks away from highest block and knownStates was ~49M
geth stalled for about 20 min, so I restarted. knownStates reset to 0 and is now going extremely slow. But disk usage is the same as before restarting.

Windows 10
SSD
16GB RAM
100Mbit connection

wwallace on 29 Dec 2017

so it would be at least 10M state to import. these sync always depend on what peer you have connected to. it was pretty bad yesterday but after restart, it proceeded with much higher rate. I was expecting at least another 2 days based on what I saw yesterday but some how, it got quicker over the night, no idea why.

garyng2000 on 29 Dec 2017

geth 1.8
still having this issue.
after reboot geth shows this error

WARN [01-05|02:54:25] Ancestor below allowance                 peer=2586937320363579 number=4015416 hash=abc963…9b3dc5 allowance=4765509
WARN [01-05|02:54:25] Synchronisation failed, dropping peer    peer=2586937320363579 err="retrieved ancestor is invalid"

now fast sync is dropping peers and not syncing. ~100 blocks to finish but it never gets synced.

winsvega on 5 Jan 2018

this is the nature of p2p networking, you are at the mercy of the peers and network condition. usually when you wait enough, it would work. ctrl-c and restart is fine too(which hopefully can get you a better peer). another alternative is find a good peer and add it manually.

cecilia-to on 5 Jan 2018

Is there a place to see the current max knownStates?

Shipwreck on 13 Jan 2018

👍1

not simple. it basically depends on the transactions. at this moment do a rough estimate which is 15-20 per block. so may be 70-80M(close to 6M block now). but going forward it will be just more per block since they up the gas limit over time.

cecilia-to on 13 Jan 2018

So I am at about 8 hours syncing on 57 Mbps with 16GB DDR4 / SSD / 1080Ti / 8700K and geth --fast --cache=4024. My eth.syncing says:

  currentBlock: 4900069,
  highestBlock: 4900328,
  knownStates: 51148480,
  pulledStates: 51137757,
  startingBlock: 0

currentBlock hasn't moved for hours. 14 peers, eth.blockNumber 0, just cycling through new state entires:

INFO [01-13|10:00:32] Imported new state entries               count=803  elapsed=3.654ms   processed=51095193 pending=8217  retry=0   duplicate=4005 unexpected=12212
INFO [01-13|10:00:33] Imported new state entries               count=834  elapsed=1.759ms   processed=51096027 pending=13439 retry=0   duplicate=4005 unexpected=12212

Is this normal? Where can I find a breakdown of those values and their behavior? What's the point of 'pending' if it is constantly changing... ?

Shipwreck on 13 Jan 2018

it is normal and if it is constantly changing(processed=51096027 ), you are good. just wait it will finish then switched to normal mode

cecilia-to on 13 Jan 2018

Appreciate your time, thank you very much.

Do you recommend exploring Parity? I see it recommended over Geth but I've only personally been exposed through it through negative news from hacks and am wary.

Shipwreck on 13 Jan 2018

Parity the contracts they wrote have bugs, but Parity the program itself is fine. However, one thing I noticed is that it seems to suck up lots of CPU when in the initial sync phase. I don't know how it behave during normal stage. It does have advantage of constantly pruning thus keep the storage space checked. geth needs lots of space after it enter normal stage. about 100-200G a month at the current rate. I need to have a standby machine to do the 'fast' periodically. As fast is the only way to 'compress'

cecilia-to on 13 Jan 2018

I pulled the plug on my sync, going to look into Parity. Can't justify the space and effort to keep a real full node if this is what it looks like. The machine would have to be running constantly, right? Constantly syncing, and at some point probably not even being able to keep up? Are there known metrics for the network growing faster than D/L speeds can sync?

I thought I would have a HD that I booted to and got up to speed any time I wanted to work with ETH but it doesn't seem like that's the case with an actual full node without pruning or warp or some such optimisation.

Thanks again for your time, appreciate your info.

Shipwreck on 13 Jan 2018

I don't feel secure using MEW, my Ledger has more on it than I am comfortable having in one place and they are back-ordered for new units, so I thought having an official client would be the next best thing... since Mist doesn't seem to fit my needs, Parity is pretty close to being official so I guess I'm headed there.

Is my logic sound?

Shipwreck on 13 Jan 2018

if all that you care is the wallet address, you don't even need the node. just generate the wallet address, and send whatever coin to that address and be done with it. Basically treat this as 'I don't use it much' address. When you do need to pull things out, do it on an island machine(including a VM with no external network connection), sign the transaction the send the raw transaction using MEW or whatever to any node(infura whatever).

cecilia-to on 13 Jan 2018

Hi Guys,

i used this to check my current block which needs to sync

currentBlock: 4506167,
highestBlock: 4934648,
knownStates: 866589,
pulledStates: 848856,
startingBlock: 4426024
}

I use this to mine
geth --datadir ./.ethereum/devnet/ --dev --mine --minerthreads 4 etherbase 0

I use this to connect the running client to the miner

eth --datadir ./.ethereum/devnet --dev attach ipc:./.ethereum/devnet/geth.ipc
Welcome to the Geth JavaScript console!

instance: Geth/v1.7.2-stable-1db4ecdc/linux-amd64/go1.9
coinbase: 0x2........................
at block: 330038 (Fri, 19 Jan 2018 13:40:44 CET)
datadir: /home/smartcontract/.ethereum/devnet
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 shh:1.0 txpool:1.0 web3:1.0

web3.eth.syncing
false
web3.fromWei(eth.getBalance(eth.coinbase),"ether")
1445287.5
web3.eth.blockNumber
330039

When I want to send a transactions I have the following code in that console
th.sendTransaction({from:eth.coinbase, to:eth.accounts[1], value: web3.toWei(0.05, "ether")}) Please unlock account d1ade25ccd3d550a7eb532ac759cac7be09c2719. Passphrase: Account is now unlocked for this session. '0xeeb66b211e7d9be55232ed70c2ebb1bcc5d5fd9ed01d876fac5cff45b5bf8bf4'.

At the end I do not see the tranasction on Etherscan nor in the wallet that I am sending to. Also balance remain the same.
HOW CAN THIS BE HAPPENING OMG:)

background: I had a not synced chaindata, some did a removedb on geth then importing all the blocks again, since the database was corrupted with message such as "Failing peer messages"? Really ethereum still need to improve.

jalal1985 on 19 Jan 2018

I'm syncing because when I tried deploying with truffle, I got network state unknown. Reading around I found I needed to have eth.blockNumber be a positive integer. Currently it's 0 and my state is syncing.

> eth.syncing
{
  currentBlock: 5186010,
  highestBlock: 5186166,
  knownStates: 32389802,
  pulledStates: 32378516,
  startingBlock: 5185518

when can I expect this to finish? What are these states?

quantumproducer on 3 Mar 2018

At this point (9 March 2018) it takes almost 8 days to sync the Ethereum blockchain with a setup like so:

i7 processor, 8 cores, 64GB RAM, 100Mbps Internet connection, SSD drive - ideally with RAID

The fastest way to sync is to start clean by removing your existing database:
geth removedb
and then doing a fast sync with large cache size (i.e., 8GB if you have enough RAM, like 64GB of RAM):
geth --syncmode "fast" --cache 8192 console

There are 3 distinct stages in the sync process:

Importing blocks (see Etherchain.org, 5,269,000+, 10+ hours)
After you get close to the current block, geth automatically switches to full mode and continues on. So there is no need to stop and restart.
Importing state entries (96M+, 1+ days)
Importing chain segments (same # as blocks, 6+ days), for example:
INFO [03-10|14:07:44] Imported new chain segment blocks=139 txs=776 mgas=430.595 elapsed=8.039s mgasps=53.560 number=1854160 hash=9d134e…e0adc0 cache=15.85mB

Your chaindata folder will be 220+ GB after you have fully sync'ed.

You can tell that you are fully sync'ed by getting chain segment messages of only 1 block, which is identical to the current block number:
INFO [03-16|21:23:04] Imported new chain segment blocks=1 txs=341 mgas=7.990 elapsed=435.942ms mgasps=18.329 number=5269405 hash=939ae8…96e825 cache=143.76mB

Good luck!

tinschel on 8 Mar 2018

👍6 ❤4

See this issue for a detailed description of what is going on under the hood - https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-370732526

wtfiwtz on 11 Mar 2018

@quantumproducer:

See @tinschel 's comment above and also see https://github.com/ethereum/go-ethereum/issues/16147#issuecomment-371878483

cies on 21 Mar 2018

@tinschel you said:

You can tell that you are fully sync'ed by getting chain segment messages of only 1 block, which is identical to the current block number

Wouldn't it be super nice if eth.syncing would return false only when this point is reached? And give some sort of human understandable numbers back detailing the progress it is making, until it gets there?

I see so many questions on this point, users are totally confused by eth.syncing's behaviour.

cies on 21 Mar 2018

👍1

it sure would be, @cies !

tinschel on 21 Mar 2018

This is the dumbest shit ever.

jel111 on 29 Mar 2018

👍1

I synced and took some measurements: http://www.freekpaans.nl/2018/04/anatomy-geth-fast-sync/

@tinschel I don't see a lot of time being taken by "Importing chain segments". In fact it only seems to be doing that for "new" blocks. Any idea?

FreekPaans on 3 Apr 2018

👍1

That is awesome analysis @FreekPaans... I pondered back in December 2017 about whether it is to do with a particular method called requestTTL... https://github.com/ethereum/mist/issues/2749#issuecomment-354282608

Basically you can be stalled waiting for one of the peers to be giving you the information you need.

Also, parts of the network may be starved of blocks or the trie state that you are seeking. Effectively, if you are not connected to enough peers with the latest data, then you will waiting on the "network" to respond with the information you need in order to continue with the chain validation.

There was some good analysis I found here:

Hope this helps... I'm yet to figure out the exact reason why it stalls.

wtfiwtz on 3 Apr 2018

Yes, I also have the feeling it's due to blocks/state trie data not being available at the peers I'm connected to, but no clue how I'd verify that. I'm doing a full sync now, has been fetching just the blocks for over 11hrs as we speak, still about 30% to go.

FreekPaans on 3 Apr 2018

@FreekPaans: I'm surprised your first single-block “Imported new chain segment” log message came about so fast. At what point in your analysis did you see the importing of chain segments starting? For me the importing of chain segments lasted 6 days before my eth.synching actually showed false.

Maybe your higher speed had to do with the increased number of peers (25) that you were seeing. I did not unblock port 30303 and although I also used --maxpeers 25 (which is the default) I only ever saw 8 peers max.

tinschel on 3 Apr 2018

@tinschel yes, with 30303 closed it stuck on about 8 peers with me as well. Not familiar with the P2P protocol so don't know why.

The first Imported new chain segment came when the state trie finished importing, and it only had 4 entries count > 1.

FreekPaans on 3 Apr 2018

Took me about 1-2 hours to fully sync Rinkeby using geth --rinkeby (2,057,556 blocks and about 10,760,209 state entries).

PhABC on 5 Apr 2018

I now also completed a geth full sync: http://www.freekpaans.nl/2018/04/anatomy-of-a-geth-full-sync/. tl;dr: took a little over 9 days, vs under 8 hours for fast sync.

FreekPaans on 13 Apr 2018

👍1

@FreekPaans, yes, the 9 days are more in line with my sync; I didn't realize that your original sync didn't go all the way; if you only do fast sync, others are not able to use you as a node for blockchain synchronization; so if everyone just used fast sync, the blockchain couldn't exist.

tinschel on 13 Apr 2018

@tinschel
that sounds very bad for Ethereum. The size of it grows really fast(I think in the range of 100G a month) and not that many people have such space for full sync. How can it scale for the intended usage(rule the world) ?

garyng2000 on 13 Apr 2018

@garyng2000, that's a little bit of an exaggeration; I've been running the full sync now for a month and it has only grown from 220GB to 241GB today

tinschel on 13 Apr 2018

@tinschel
would that be affected by when you started(recent price drop can have an effect on # of transactions) ? I remembered we had a 'fast sync' node that was growing really fast but I am not managing it so not sure how large it is now. But even using your figure, it is still growing at an alarming rate. Basically, not something for a typical 'personal' node and it basically break the intended large P2P nature. even for mining, we are seeing that most of them are done by very few pools, basically centralized in a sense.

garyng2000 on 13 Apr 2018

if you only do fast sync, others are not able to use you as a node for blockchain synchronization; so if everyone just used fast sync, the blockchain couldn't exist.

Nope, that's incorrect. A fast-synced node helps out, it's got a lot of state others can download. A les-client otoh is maybe what you were thinking about.

holiman on 13 Apr 2018

@garyng2000, no effect on the start date. it's possible that different months have more transactions and state and therefore the blockchain size grows more rapidly during those months. There is no difference between the block chain size in fast sync mode and full mode because after fast sync completes, it automatically switches to full mode. In other words, fast sync may get you to the end of the blocks very quickly, but you won't have the full blockchain with all state until you complete the full sync.

So fast sync is good if all you want to do is maintain a wallet or start mining. Full sync is necessary so that you can become a full node within the Ethereum blockchain.

tinschel on 14 Apr 2018

that is strange, I just asked our guy maintaining the node. The last fast sync(from scratch) was about a month ago and it ended for about 60G in size(all updated state downloaded) and now is already 140G. not sure what is going on there. Though from an operational perspective, I see no difference between full and fast(which is just a trimmed full). can still do all the operation, unlike light which is basically useless for our usage.

garyng2000 on 14 Apr 2018

I am experiencing the same issue when using geth --fast --cache 1048.

Imported new state entries

 eth.syncing
{
  currentBlock: 5535921,
  highestBlock: 5536021,
  knownStates: 274,
  pulledStates: 273,
  startingBlock: 5535921
}

res-Q on 1 May 2018

👍1

This is driving me crazy. One week to sync! 😳

bneiluj on 24 May 2018

Read, tried! And nothing helps

lk29 on 28 May 2018

If you use a hard disk geth will not work. You need SSD, or switch to parity for syncing.

quantumproducer on 28 May 2018

Just give some information:

I started the syncing from May 18, It just ended minutes ago.

BlockNumber: 5740903
pulledStates: 191247645

TooBug on 6 Jun 2018

@TooBug Hello,
Do you get "false" when you type this command : eth.syncing
Can you give me the number of the KnowStates/pulledStates you at ?

Othman21 on 16 Jun 2018

@Othman21 Oth

Yes, When it's finished, eth.syncing returns false.

The number is given above. KnownStates is quite similar to pulledStates.

TooBug on 17 Jun 2018

@TooBug Can you tell me the exact processed number that you see right now ?
for exemple i'm at processed=193996209

Othman21 on 17 Jun 2018

FWIW I was able to get geth to fully sync by waiting until eth.blockNumber is near the numbers in eth.syncing and then restarting geth. I was able to do this at ~160m states. After restarting geth, it took about 20 min to catch up to the blockchain and now eth.syncing is false and the only output now is 'imported new chain segment' every time a new block is found.

jdowning100 on 17 Jun 2018

here is an output of mine taken on 16JUN2018 at 6:18am PTD:

>eth.syncing { currentBlock: 5799102, highestBlock: 5799188, knownStates: 96279752, pulledStates: 96279752, startingBlock: 5799001 }

tinschel on 18 Jun 2018

@tinschel here is an output of mine taken on 19JUN2018 at 7:32am PTD:
> eth.syncing { currentBlock: 5815191, highestBlock: 5815294, knownStates: 198143487, pulledStates: 198143486, startingBlock: 5815181 }

Othman21 on 19 Jun 2018

Just to clarify... fastsync is now default... and I may quit and restart geth with fast enabled?

Anyway I'm currently at:

> eth.syncing
{
  currentBlock: 5827681,
  highestBlock: 5827747,
  knownStates: 96988858,
  pulledStates: 96988508,
  startingBlock: 5824114
}

> eth.blockNumber
0

@tinschel @Othman21 How is this possible? 100% more state entries in 3 days?

codepushr on 21 Jun 2018

@Othman21 This is nuts! 29 days and 200m state entries? Crazy...

codepushr on 21 Jun 2018

@codepushr yes you can, i run this command :
./geth --cache=25000 --rpc --maxpeers 25 --fast

I have not finished syncing yet, today it will be 29 days

eth.syncing

{
currentBlock: 5829510,
highestBlock: 5829582,
knownStates: 203714648,
pulledStates: 203714648,
startingBlock: 5829510
}

eth.blockNumber
0

Othman21 on 21 Jun 2018

@codepushr i know
I'm looking for someone who has finished syncing to tell me the current knownStates that he has to get an idea if I'm far or near.

Othman21 on 21 Jun 2018

It's so bad that there is no mechanism to determine the absolute progress...

codepushr on 21 Jun 2018

👍1

@codepushr : yes, fast sync is enabled by default so you don't have to specify it on command line, BUT once you start sync'ing and you restart geth, it will automatically DISABLE fast sync; so best not to restart

tinschel on 22 Jun 2018

@tinschel Really? So even when restarting with --fast or --syncmode=fast it will boot up in full sync and I would need to remove the db and restart from scratch to get a fast sync?

codepushr on 23 Jun 2018

I'm fairly certain that it is not the case... you are in fast sync until you hit the pivot point. Then eth.blockNumber will be a positive value (not zero). From that point onwards, you are in full sync mode.

wtfiwtz on 23 Jun 2018

Here are the details of the pivot implementation: https://github.com/ethereum/go-ethereum/pull/14460 and https://github.com/ethereum/go-ethereum/commit/0042f13d47700987e93e413be549b312e81854ac

wtfiwtz on 23 Jun 2018

I would just not bother specifying --fast on command line or removing your DB and restarting in the hope that it will be faster. just run geth without the option and keep going! If you have to restart geth don't specify the option. geth will use fast by default if it can. otherwise it will give you a message that fast is disabled.

tinschel on 23 Jun 2018

@codepushr @tinschel any news ?

eth.blockNumber
0
eth.syncing
{
currentBlock: 5852375,
highestBlock: 5852457,
knownStates: 213724149,
pulledStates: 213706822,
startingBlock: 5852248
}

Othman21 on 25 Jun 2018

@Othman21 Currently at 121m states... I'll need another week to catch up lol.

codepushr on 25 Jun 2018

🎉1

you unfortunately don't see the known states anymore once synchronization is complete. But my currentBlock is at 5853416 as of 25JUN2018 1218 Pacific Time.

tinschel on 25 Jun 2018

is there anyone who has finished syncing to tell us the current knowstates he has?

eth.syncing
{
currentBlock: 5865970,
highestBlock: 5866080,
knownStates: 218653449,
pulledStates: 218633252,
startingBlock: 5865761
}
eth.blockNumber
0

Othman21 on 28 Jun 2018

here is where I'm currently at after a 1 hour break:
> eth.syncing { currentBlock: 5866670, highestBlock: 5866837, knownStates: 96279752, pulledStates: 96279752, startingBlock: 5866459 }

tinschel on 28 Jun 2018

@tinschel are you already synced and just paused it for an hour to demonstrate? if so, how come your states are less than 1/2 of @Othman21 's?

codepushr on 28 Jun 2018

I don't think pulledStates can be used for progress indication, since I just reconnected a node after a while and that one didn't get beyond 115m states.

FreekPaans on 28 Jun 2018

@codepushr yes I'm sync'ed and just paused for an hour. that's when I sent the latest eth.syncing. the states are a mystery to me. like @FreekPaans said, they are not a good indicator for sync completion.

tinschel on 28 Jun 2018

@FreekPaans did you finish the sync ?
I have not finished syncing yet, today it will be 37 days.

eth.syncing
{
  currentBlock: 5876059,
  highestBlock: 5876129,
  knownStates: 223140030,
  pulledStates: 223136789,
  startingBlock: 5875977
}
eth.blockNumber
0

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/simfs     524288000 126645968 397642032  25% /

@codepushr run it in background
sudo nano /etc/systemd/system/eth.service

[Unit]
Description=eth for Pool
After=network-online.target

[Service]
ExecStart=/home/eth-geth/build/bin/geth --cache=25000 --rpc  --maxpeers 25 --fast
Restart=always
RestartSec=3
User=root

[Install]
WantedBy=multi-user.target

sudo systemctl enable eth
sudo systemctl start eth
sudo systemctl status eth

Othman21 on 29 Jun 2018

👍2

@othman21 yes, did you see my blogposts?
http://www.freekpaans.nl/2018/04/anatomy-geth-fast-sync/
http://www.freekpaans.nl/2018/04/anatomy-of-a-geth-full-sync/

This time I synced from a node that was about 1 month stale, took 10 days.
On Fri, 29 Jun 2018 at 19:50, Othman21 notifications@github.com wrote:

@FreekPaans https://github.com/FreekPaans did you finish the sync ?
I have not finished syncing yet, today it will be 37 days.

eth.syncing
{
currentBlock: 5876059,
highestBlock: 5876129,
knownStates: 223140030,
pulledStates: 223136789,
startingBlock: 5875977
}
eth.blockNumber
0

@codepushr https://github.com/codepushr run it in background
sudo nano /etc/systemd/system/eth.service
`[Unit]
Description=eth for Pool
After=network-online.target

[Service]
ExecStart=/home/eth-geth/build/bin/geth --cache=15000 --rpc --maxpeers 25
--fast
Restart=always
RestartSec=3
User=root

[Install]
WantedBy=multi-user.targetsudo systemctl enable ethsudo systemctl start
ethsudo systemctl status eth`

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/14647#issuecomment-401426522,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADfb6UOvIyow9xag4aVQ-H8j44j_S0x1ks5uBmjrgaJpZM4N9Zz8
.

FreekPaans on 29 Jun 2018

👍4

@FreekPaans
Yes I saw them, the geth gets only the last block and sometimes some Knowstates and it's very "slow"
I just changed my command line like yours
geth --maxpeers 25 --cache 25000 --verbosity 4 --syncmode full
Can you please do a screenshot on the lines that appears in your terminal to get an idea
You have a vps more powerful than mine (x3) lol

Othman21 on 30 Jun 2018

@Othman21 see attached. This is when I restart a node that was last fully synced a couple of hours ago.
geth-detail.log

FreekPaans on 2 Jul 2018

❤1

I have 2 servers to sync. A for a week, B for 2 weeks. Here are their states:
Machine A:

> eth.syncing
{
  currentBlock: 6358077,
  highestBlock: 6358143,
  knownStates: 167673098,
  pulledStates: 167673097,
  startingBlock: 6357998
}
> eth.blockNumber
0
> net.peerCount
21

Machine B:

> eth.syncing
{
  currentBlock: 6358040,
  highestBlock: 6358157,
  knownStates: 208173888,
  pulledStates: 208173888,
  startingBlock: 6358040
}
> eth.blockNumber
0
> net.peerCount
2

I succeeded before, the "state entries" log ends at 150 millions I recall.
Now state entries more than 160 millions and 200 millions. I wonder when it will ends...

dreamingodd on 19 Sep 2018

@dreamingodd Do you have any updates (e.g. at which number the states were finishing)?

kaythxbye on 28 Sep 2018

@dreamingodd , I've been sync'ed for several months now; when I restart geth, it has to re-sync just the last few blocks. Here is what I get when I do that today (9/28/2018 at 11:41am PT):
> eth.syncing { currentBlock: 6416641, highestBlock: 6416662, knownStates: 96279752, pulledStates: 96279752, startingBlock: 6416641 }

tinschel on 28 Sep 2018

Whats the current upper bound of --testnet in fast sync mode, it's been almost a week, 8hrs per day:

{
  currentBlock: 4479565,
  highestBlock: 4479638,
  knownStates: 54592672,
  pulledStates: 54572219,
  startingBlock: 4474887
}

M-Shahbaz on 22 Nov 2018

I recently completed a fast sync, starting from scratch. My node processed 254,811,206 state entries before finishing. Here was the last Imported new state entries line:

INFO [12-07|09:41:58.173] Imported new state entries count=325 elapsed=9.931ms processed=254811206 pending=0 retry=0 duplicate=14306 unexpected=44810

And here is some context around that log line:

INFO [12-07|09:41:52.982] Imported new state entries               count=499  elapsed=7.055ms   processed=254810260 pending=771   retry=0    duplicate=14306 unexpected=44810
INFO [12-07|09:41:57.012] Imported new state entries               count=621  elapsed=4.587ms   processed=254810881 pending=796   retry=0    duplicate=14306 unexpected=44810
INFO [12-07|09:41:58.173] Imported new state entries               count=325  elapsed=9.931ms   processed=254811206 pending=0     retry=0    duplicate=14306 unexpected=44810
INFO [12-07|09:41:58.177] Imported new block receipts              count=1    elapsed=3.971ms   number=6841800 hash=c4db3b…040b25 age=27m9s    size=116.08kB
INFO [12-07|09:41:58.178] Committed new head block                 number=6841800 hash=c4db3b…040b25
INFO [12-07|09:42:07.044] Imported new chain segment               blocks=10 txs=927 mgas=65.008 elapsed=8.864s    mgasps=7.334 number=6841810 hash=2698fa…cebe66 age=25m46s   cache=7.55mB
INFO [12-07|09:42:15.582] Imported new chain segment               blocks=12 txs=1384 mgas=88.370 elapsed=8.537s    mgasps=10.351 number=6841822 hash=a9064c…432e78 age=23m32s   cache=17.34mB
INFO [12-07|09:42:24.527] Imported new chain segment               blocks=12 txs=1440 mgas=74.632 elapsed=8.944s    mgasps=8.344  number=6841834 hash=c80849…f12f03 age=20m21s   cache=27.75mB
INFO [12-07|09:42:32.561] Imported new chain segment               blocks=14 txs=1246 mgas=83.007 elapsed=8.034s    mgasps=10.331 number=6841848 hash=2cbbfe…d173c0 age=18m19s   cache=37.18mB
INFO [12-07|09:42:41.012] Imported new chain segment               blocks=14 txs=1655 mgas=80.438 elapsed=8.451s    mgasps=9.518  number=6841862 hash=5d3bc2…bd7289 age=14m19s   cache=47.72mB
INFO [12-07|09:42:49.356] Imported new chain segment               blocks=10 txs=1557 mgas=63.153 elapsed=8.344s    mgasps=7.569  number=6841872 hash=1c11a8…ba7cac age=11m44s   cache=56.52mB
INFO [12-07|09:42:57.637] Imported new chain segment               blocks=15 txs=1616 mgas=85.097 elapsed=8.280s    mgasps=10.276 number=6841887 hash=1ac03b…622d3e age=7m53s    cache=67.46mB
INFO [12-07|09:43:06.485] Imported new chain segment               blocks=13 txs=1657 mgas=79.934 elapsed=8.847s    mgasps=9.034  number=6841900 hash=2300d2…e3076a age=4m38s    cache=77.99mB
INFO [12-07|09:43:10.533] Imported new chain segment               blocks=8  txs=793  mgas=32.020 elapsed=4.048s    mgasps=7.909  number=6841908 hash=8dd97c…cb2afe age=3m9s     cache=82.81mB
INFO [12-07|09:43:10.535] Imported new block headers               count=1    elapsed=1m2.045s  number=6841909 hash=8bb9a3…27c074 age=3m4s
INFO [12-07|09:43:10.544] Imported new block headers               count=1    elapsed=8.759ms   number=6841910 hash=d3f22b…1d80b0 age=2m24s
INFO [12-07|09:43:10.554] Imported new block headers               count=2    elapsed=10.131ms  number=6841912 hash=67b75c…918ac2 age=1m10s
WARN [12-07|09:43:10.610] Discarded bad propagated block           number=6841904 hash=b7deda…28b3dc
INFO [12-07|09:43:10.683] Imported new block headers               count=1    elapsed=7.534ms   number=6841913 hash=58f0f9…a9a201
INFO [12-07|09:43:11.590] Imported new chain segment               blocks=1  txs=165  mgas=7.985  elapsed=808.053ms mgasps=9.882  number=6841909 hash=8bb9a3…27c074 age=3m5s     cache=83.99mB
INFO [12-07|09:43:14.869] Imported new chain segment               blocks=4  txs=755  mgas=29.935 elapsed=3.278s    mgasps=9.131  number=6841913 hash=58f0f9…a9a201 cache=88.67mB
INFO [12-07|09:43:14.869] Fast sync complete, auto disabling
INFO [12-07|09:43:36.504] Imported new chain segment               blocks=1  txs=48   mgas=1.493  elapsed=198.174ms mgasps=7.535  number=6841914 hash=1be461…c61103 cache=88.98mB
INFO [12-07|09:44:16.096] Imported new chain segment               blocks=1  txs=192  mgas=7.996  elapsed=558.982ms mgasps=14.304 number=6841915 hash=e9f46b…989562 cache=90.27mB

jeremyschlatter on 7 Dec 2018

👍8 ❤1

@jeremyschlatter what kind of system do you have and how long did it take to sync?

And should this issue be re-opened? It seems like people are still getting "stuck" during a fast sync waiting for states and confused because of the lack of a progress bar. I understand building a progress bar for this in a decentralized and secure way is not simple (or maybe even possible), but then we need better messaging.

WyseNynja on 8 Dec 2018

👍1

I did one recently, about same number of states, took 5 days on a 4 core/8
gb + SSD

On Sat, 8 Dec 2018 at 19:37, Bryan Stitt notifications@github.com wrote:

@jeremyschlatter https://github.com/jeremyschlatter what kind of system
do you have and how long did it take to sync?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/14647#issuecomment-445480304,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADfb6YAroP8d1bjPjCEwNuHxfC0w1bODks5u3AbPgaJpZM4N9Zz8
.

FreekPaans on 8 Dec 2018

@WyseNynja ~4 days on an n1-standard-4 + SSD on Google Cloud Platform

jeremyschlatter on 9 Dec 2018

I have a 12 core 32GB ram machine and after ~2 days or so I'm at 163mm state entries done which appears to be ~64% of the way through so yeah about 4 days (pretty sure it gets slower the further it progresses) on my machine. Gave it cache=7000

JohnAllen on 9 Dec 2018

Just finished fast sync on AWS EC2 i3.xlarge (with attached NVMe SSD).
It took about 17 hours, and the final eth.syncing.pulledStates was about 252307641.

zhouer on 10 Dec 2018

👍1

Interesting, I am currently at pulledStates: 261648186.

kaythxbye on 10 Dec 2018

For those interested: I just finished a sync to the testnet (Ropsten) in about 23 hours on an n1-standard-4 Google Cloud VM with an SSD (100GB), default configuration has 4 vCPUs and 15 GB memory. I ran geth with --cache=4096. Pulled states was 80M+ shortly before finishing.

smeis on 10 Dec 2018

@kaythxbye have you finally sync'd?

{
currentBlock: 6897129,
highestBlock: 6897235,
knownStates: 266058499,
pulledStates: 266043876,
startingBlock: 6897063
}

This node is synching since 2+ weeks. Recently upgraded to v1.8.20-stable-24d727b6

got3nks on 16 Dec 2018

It took me around 3 days and 4 hours to synchronize a ETH Geth fast node:

Settings: --syncmode fast --gcmode full --cache 4096
Start: 2018-12-15 10:00 UTC
End: 2018-12-18 14:08 UTC
State entries: 260172278
Disk usage: 149G
Hardware: Virtual machine, 4 cores, 8 GB memory, 300 GB SSD

grittibaenz on 19 Dec 2018

👍1

As of today .Took about a week to finally synced.

Testet(Ropsten)

Disk Using : about40G.
KnowStates:92 million more of less.
Hardware:AWS cloud service machine,2 Processor , 2 GB memory 100GB SSD.

xiaomingloveqinger on 13 Jan 2019

👍3

Just finished syncing mainnet at block 7089163. Total time about 36 hours on a quad core with 32GB RAM, fast connection, 256GB SSD. Disk usage ~154GB.

Final "Imported new state entries": 268473324

INFO [01-18|21:27:10.217] Imported new state entries               count=1081 elapsed=12.427ms  processed=268473324 pending=0      retry=0    duplicate=24770 unexpected=101202

AronVanAmmers on 18 Jan 2019

👍3

february 6 2019

sync taken5 days

2 vCPU
8 GB RAM
SSD drive

disk usage: 160G
highestBlock: 7178900
knownStates: 276777218

Ivan31415 on 6 Feb 2019

👍3

HARDWARE:
HP ProBook470 G2/windows7
Core i5-5200 @ 2.2Ghz
8GB
Samsung T5 USB 3.0 SSD 250G

Geth Running on VirtuaMachine:
ubuntu server on Vmware player
3 processor 3GB System memory
2GB geth cache option
default fast sync option

it's taken about 1-week

block 7191540
knownStates are about 283500000
disk usage : 167G

ui-hyo on 8 Feb 2019

👍4

Just finished now, took around 4 days on a Laptop with

Intel Core i5-8250U
Samsung MZ-V7E500BW 970 EVO.

Disk Usage: 162G

processed=284332222 pending=0     retry=0    duplicate=11504 unexpected=67689

xerg0n on 18 Feb 2019

👍5

My statistics:

2019-05-06 01:00:32 avg: 1827 max: 1938 min: 1378 states/s  remain: 136604075 states     4 peers    eta@ 20:46:28.165828
2019-05-06 01:00:37 avg: 1864 max: 1938 min: 1378 states/s  remain: 136595500 states     3 peers    eta@ 20:21:14.951050
2019-05-06 01:00:42 avg: 1791 max: 1938 min: 1378 states/s  remain: 136583359 states     3 peers    eta@ 21:11:16.481006
2019-05-06 01:00:48 avg: 1742 max: 1938 min: 1378 states/s  remain: 136580287 states     3 peers    eta@ 21:46:35.797305
2019-05-06 01:00:53 avg: 1721 max: 1938 min: 1378 states/s  remain: 136575694 states     3 peers    eta@ 22:03:01.154434
2019-05-06 01:00:58 avg: 1682 max: 1938 min: 1378 states/s  remain: 136569043 states     4 peers    eta@ 22:33:15.402442
2019-05-06 01:01:03 avg: 1698 max: 1938 min: 1378 states/s  remain: 136564293 states     3 peers    eta@ 22:20:27.458747

PS. I wrote a tiny python script to overview the process. It's here https://github.com/hayorov/ethereum-sync-mertics

hayorov on 5 May 2019

Would be really useful if this would be trackable somewhere in geth or via some online tool like Etherscan. The latest number of states that I had before syncing completed was 314.8m state entries. I was dumping an output every 15 minutes to the console while syncing (because when you're synced up you cannot read that number anymore):
setInterval(function(){console.log(web3.eth.syncing.knownStates/1000000);}, 15*60*1000)

SCBuergel on 9 May 2019

👍1

322.14m states on may 9th
INFO [05-09|12:10:17.599] UTC - new state entries processed=322150741

~1 day for blockchain. ~5 days for states - 206 GB on disk

andi78 on 9 May 2019

❤2 👍1

Just completed another one, took a little under 48 hours.

Geth 1.8.27
AWS t2.xlarge (4 cores, 16gb)
States 333564669 @ block 7963639

What's interesting is that at some point the states "rate" gets really low:

I suspect this is due to certain parts of the state trie not being available at my peers. Might be that restarting (or some other way of getting new peers) would help in those cases.

FreekPaans on 15 Jun 2019

👍5 ❤1

There are now more than 362,778,222 states.

INFO [07-23|18:30:58.161] Imported new block headers               count=1    elapsed=4.196ms    number=8208527 hash=6891ea…0a52cb age=1m2s
INFO [07-23|18:31:22.870] Imported new state entries               count=484  elapsed=4.748ms    processed=362778222 pending=2760  retry=0   duplicate=1583 unexpected=9221

It would be really nice if there were some way to know how much longer this is going to take.

WyseNynja on 23 Jul 2019

👍2

Hi @WyseNynja , How much time it took you to sync 362,778,222 states?
It's 14th day and I am at 260873292 only :(

ankita-p17 on 24 Jul 2019

Started the node fresh on July 11. The blocks synced fast, but states are going slow. The server is busy doing other things, but this is longer than I expected. It’s still syncing now.

WyseNynja on 24 Jul 2019

Started the node fresh on July 11. The blocks synced fast, but states are going slow. The server is busy doing other things, but this is longer than I expected. It’s still syncing now.

I also started on July 12.But, it's way too slow. Synced 269325649 entries only.
Can you help me with the configuration you are using?

ankita-p17 on 25 Jul 2019

Started the node fresh on July 11. The blocks synced fast, but states are going slow. The server is busy doing other things, but this is longer than I expected. It’s still syncing now.

I also started on July 12.But, it's way too slow. Synced 269325649 entries only.
Can you help me with the configuration you are using?

Sounds like you're on an HDD instead of SSD

FreekPaans on 25 Jul 2019

Started the node fresh on July 11. The blocks synced fast, but states are going slow. The server is busy doing other things, but this is longer than I expected. It’s still syncing now.

I also started on July 12.But, it's way too slow. Synced 269325649 entries only.
Can you help me with the configuration you are using?

Sounds like you're on an HDD instead of SSD

yes. I started with HDD

ankita-p17 on 25 Jul 2019

Started the node fresh on July 11. The blocks synced fast, but states are going slow. The server is busy doing other things, but this is longer than I expected. It’s still syncing now.

I also started on July 12.But, it's way too slow. Synced 269325649 entries only.
Can you help me with the configuration you are using?

Sounds like you're on an HDD instead of SSD

yes. I started with HDD

It can't sync on an HDD - it's too slow

FreekPaans on 25 Jul 2019

For those looking just for the number:
13 Aug 2019: 400.695.865 states

Long way to go.

zet-tech on 13 Aug 2019

👍3 ❤1

I think your sync speed for State entries and blocks has to do with your memory i have a 614000 cache ad im on 30000000 state entries in a few hours and like block 5800000.

moonwolf76 on 15 Sep 2019

and the state entries are fluoing by.

moonwolf76 on 15 Sep 2019

4 days 15 hours for me. Running Ryzen 1700X with SSD and cable modem

24794 pts/2 Sl+ 896:58 build/bin/geth -rpc
[ron@Bertha bin]$ ps -p 24794 -o etime
ELAPSED
4-15:50:16

rstrawsb on 18 Oct 2019

Need one of the eth blockchain services to list state entries along with the blockchain size data etc, welp until then:

2019-11-25: 405,923,438

jmooo on 25 Nov 2019

👍2 ❤1

Just finished fast sync after 5 days 12 hours. This thread here helped me knowing what to expect, so here are my stats.

2019-12-04
Hardware: Raspberry Pi 4 Model B/4GB, SSD attached vis USB3, broadband internet.
Geth version: 1.9.8
Datadir size: 186 GB
Highest block: 9,048,250
Number of imported state entries: 441,013,364

dancrypted on 4 Dec 2019

👍4

12-05-2019 418,072,677 state entries.

randy5555 on 6 Dec 2019

👍2

12-25-2019
Geth version: 1.9.9
Highest block: 9160798
Known states: 475519190

momaee on 25 Dec 2019

👍5

There really should be an API for this.

nyetwurk on 15 Jan 2020

👍1

Jan 17 2020
Known states: 432,391,041
Not sure why that is smaller than the one above.

INFO [01-17|18:39:47.580] Imported new state entries               count=433  elapsed=2.721ms   processed=432391041 pending=0      retry=0    duplicate=71137 unexpected=326170
INFO [01-17|18:39:47.590] Imported new block receipts              count=1    elapsed=7.889ms   number=9300316 hash=7e5fbf…197524 age=24m54s     size=54.71KiB
INFO [01-17|18:39:47.593] Committed new head block                 number=9300316 hash=7e5fbf…197524
INFO [01-17|18:39:47.624] Deallocated fast sync bloom              items=431854683 errorrate=0.001

nyetwurk on 17 Jan 2020

👍2

Jan 17 2019
Known states: 432,391,041
Not sure why that is smaller than the one above.

INFO [01-17|18:39:47.580] Imported new state entries               count=433  elapsed=2.721ms   processed=432391041 pending=0      retry=0    duplicate=71137 unexpected=326170
INFO [01-17|18:39:47.590] Imported new block receipts              count=1    elapsed=7.889ms   number=9300316 hash=7e5fbf…197524 age=24m54s     size=54.71KiB
INFO [01-17|18:39:47.593] Committed new head block                 number=9300316 hash=7e5fbf…197524
INFO [01-17|18:39:47.624] Deallocated fast sync bloom              items=431854683 errorrate=0.001

Nice, updated https://github.com/hayorov/ethereum-sync-metrics

hayorov on 23 Jan 2020

On a system with 4 cores, 16 GB RAM, 320 GB SSD it took around 36 hours to sync in 'fast' syncmode. Once the syncing was done, the system was adjusted to use 2 cores and 10GB RAM and the same disk. This seems to work well.

Sword-Smith on 25 Jan 2020

👍2

Ditto on the ~36 hrs, Known states was 431,372,690 @ block height ~9361043

brockelmore on 27 Jan 2020

👍2

Is it not possible to extract this information from a node that is already synced?

Can this be reopened? If I open a new issue requesting an API to retrieve the current number of states in the trie after the node is done syncing will it be closed summarily?

nyetwurk on 27 Jan 2020

👍2

nyetwurk on 27 Jan 2020

👍2

Still syncing on known states 436141638 and block height 9362396

ankita-p17 on 27 Jan 2020

👍3

15 Feb 2020
446266045

INFO [02-15|22:49:59.641] Imported new state entries               count=484  elapsed=11.987ms  processed=446266045 pending=0      retry=0   duplicate=50314 unexpected=640943

nyetwurk on 16 Feb 2020

👍4 ❤1

17 Mar 2020
466184476

INFO [03-17|02:20:26.688] Imported new state entries               count=343  elapsed=16.813ms  processed=466184476 pending=0      retry=0     duplicate=157290 unexpected=584684

ezoia-com on 16 Mar 2020

👍2

25.03.2020
I am not sure when my blockchain downloaded fully, but i guess that i had to fetch max 475000000 state entries

Glafuski on 25 Mar 2020

INFO [04-11|22:56:13.547] Imported new block receipts              count=1    elapsed=999.7µs    number=9852885 hash=45231e…244fc9 age=20m32s   size=9.53KiB
INFO [04-11|22:56:13.566] Committed new head block                 number=9852885 hash=45231e…244fc9
INFO [04-11|22:56:13.618] Deallocated fast sync bloom              **items=480155136** errorrate=0.001

items=480155136

sirnicolas21 on 11 Apr 2020

My problems with syncing for 2 weeks were relying on HDD and double-NAT. But after adding an SSD read-only cache to my NFS, and after today opening the ports on my outer NAT while still relying on UPnP for my inner router I got it synced.
my final amount of processed state entries was 487040102

INFO [04-14|19:48:25.498] Imported new state entries               count=271  elapsed=2.687ms    processed=487040102 pending=0     retry=0   duplicate=17264 unexpected=61660
INFO [04-14|19:48:25.556] Deallocated fast sync bloom              items=486016461 errorrate=0.001

fruor on 14 Apr 2020

Updated bound:

INFO [05-10|23:51:14.924] Imported new state entries               count=147  elapsed=7.432ms      processed=504700513 pending=0     retry=0   duplicate=1115 unexpected=4208
INFO [05-10|23:51:14.968] Imported new block receipts              count=1    elapsed=37.733ms     number=10040872 hash=27a688…d5f8f4 age=28m12s   size=23.45KiB
INFO [05-10|23:51:14.978] Committed new head block                 number=10040872 hash=27a688…d5f8f4
INFO [05-10|23:51:14.982] Deallocated fast sync bloom              items=504344987 errorrate=0.202
INFO [05-10|23:51:25.223] Imported new chain segment               blocks=3 txs=407 mgas=29.972 elapsed=10.224s      mgasps=2.932 number=10040875 hash=4cb642…c3d687 age=27m45s   dirty=3.67MiB

I.e. 504344987 state entries at 10 May 2020. For fast sync this is about 240GB of data stored on the SSD.

Using a RPi v4, 4GB RAM, externally powered Samsung T5 1TB SSD to sync. Pro tip for Raspberry Pi users: use Ubuntu Server 64bit. Raspbian currently does not natively use 64bit. You can manually "allow" this via some kernel settings but I still ran into out of memory issues. I might have done something wrong there. However, on Ubuntu, I have never had an out of memory error so far since syncing. I have to note that I allocated 8GB swap on the SSD and using --cache=512 as Geth parameter. Also make sure your 30303 port is open, so external nodes can connect (and you can find peers!).

Sadly, I have not written down when I started syncing. However, I know it took less than a week. It is somewhere between 5-6 days I think.

jochem-brouwer on 10 May 2020

👍6 🚀1

11 June 2020
537342779

 INFO [06-11|20:02:41.758] Imported new state entries               count=452  elapsed=6.587ms     processed=537342779 pending=0     retry=0   duplicate=11612 unexpected=28764

dmf7z on 11 Jun 2020

🚀1

Times are in GMT +0. Started about 1 week ago.

Ropsten testnet
INFO [06-19|13:24:06.907] Deallocated fast sync bloom items=245025020 errorrate=0.001
Chaindata Storage - 39.6GB
Ancient Storage - 36.8GB

INFO [06-21|07:57:24.932] Deallocated fast sync bloom items=536929173 errorrate=0.001
Chaindata Storage - 100.2GB
Ancient Storage - 133.2GB

kutysam on 21 Jun 2020

21 June 2020
574408341

INFO [06-21|17:06:28.769] Imported new state entries               count=435  elapsed=3.950ms     processed=574408341 pending=0     retry=0   duplicate=268 unexpected=578

marcelomorgado on 21 Jun 2020

👍4

I think someone has a problem with the state. This won't scale much more.

piluex on 30 Jul 2020

10 Aug 2020
571739565

INFO [08-09|23:26:03.932] Imported new state entries count=56 elapsed=2.469ms processed=571739565 pending=0 retry=0 duplicate=13914 unexpected=49760 INFO [08-09|23:26:03.939] Imported new block receipts count=1 elapsed=6.060ms number=10626424 hash="70eb79…2712a6" age=23m46s size=67.10KiB INFO [08-09|23:26:03.941] Committed new head block number=10626424 hash="70eb79…2712a6"

挺折腾的，在同步期间经常卡在这里：
WARN [08-08|00:09:42.076] Dropping unsynced node during fast sync id=541791c3ce2e53df conn=dyndial addr=198.204.254.10:3030
但是重启后就又可以正常同步了。猜想应该是节点限流，同步一定量的数据就会被节点拒绝。
然后就做了一个shell脚本监测同步日志，当一定时间内的日志增长量过于少时，就怀疑是节点卡住了，脚本会自动kill掉同步程序，再重新执行。
就这样，花了大概2天时间完全同步好了，ssd空间用了249G.

It's kind of annoying, getting stuck here a lot during syncing.
WARN [08-08|00:09:42.076] Dropping unsynced node during fast sync id=541791c3ce2e53df conn=dyndial addr=198.204.254.10:3030
But once you restart it, you can sync again.The conjecture is that the node is limited in current, and a certain amount of data synchronized will be rejected by the node.
Then I made a shell script to monitor the synchronous log. When the log growth is too small within a certain period of time, it is suspected that the node is stuck. The script will automatically kill the synchronous program and execute it again.
In this way, it took about 2 days to fully synchronize, and the SSD space used 249G.

supenghai on 9 Aug 2020

👍2

572083659 August 9th 2020

INFO [08-09|16:37:34.615] Imported new state entries
             count=1606 elapsed=20.127ms    processed=
579048027 pending=0     retry=0   duplicate=0 unexpect
ed=0
INFO [08-09|16:37:34.618] Imported new block receipts
             count=1    elapsed=1.940ms     number=106
26719 hash="444831…2b96ff" age=28m10s   size=87.93KiB
INFO [08-09|16:37:34.621] Committed new head block
             number=10626719 hash="444831…2b96ff"
INFO [08-09|16:37:34.644] Deallocated fast sync bloom
             items=572083659 errorrate=0.001

wellttllew on 10 Aug 2020

👍1

@wellttllew @supenghai
Hey how long did it take for a full sync?
Imported new state entries count=384 elapsed=4.84µs processed=665994734

My state entries has been downloading for like a week.

sssubik on 17 Aug 2020

@wellttllew
I would like to know your server configuration.
My server configuration is: ten-core processors, 32M memory,20M band width.The fast sync took about 48 hours.

supenghai on 17 Aug 2020

When reporting sync times, please include the type of disk as well. That can make a huge difference. HDD < SATA SSD < NVME SSD

WyseNynja on 17 Aug 2020

INFO [08-18|22:42:47.091] Deallocated fast sync bloom items=581136598 errorrate=0.173

Size ~271 GB

jochem-brouwer on 18 Aug 2020

Hey @jochem-brouwer is 581136598 your final state entries?
Because for me it has already been higher than yours and it is still contnuing..
Imported new state entries count=412 elapsed=8.904ms processed=673602785 pending=8320 retry=0 duplicate=0 unexpected=30

sssubik on 19 Aug 2020

I am going on 9 days now for a full "fast"-sync, fast i9 comp, fiber internet with SSD & cache =2048....

steelep on 20 Aug 2020

Is that a SATA SSD? Or an NVME?

On Aug 19, 2020, at 6:42 PM, steelep notifications@github.com wrote:

I am going on 9 days now for a full "fast"-sync, fast i9 comp, fiber internet with SSD & cache =2048....

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

WyseNynja on 20 Aug 2020

SATA, still its a very quick unit.

On Thu, 20 Aug 2020 at 14:34, Bryan Stitt notifications@github.com wrote:

>
>
>

Is that a SATA SSD? Or an NVME?

On Aug 19, 2020, at 6:42 PM, steelep notifications@github.com wrote:

>

I am going on 9 days now for a full "fast"-sync, fast i9 comp, fiber
internet with SSD & cache =2048....

>

—

You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/14647#issuecomment-676861306,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKCS2ZM22O2OLYUPT7YUVTSBSDS5ANCNFSM4DPVTT6A
.

--
Patrick Steele

NZ: 0212384777
HK: 6926 1526

steelep on 20 Aug 2020

@wellttllew
I would like to know your server configuration.
My server configuration is: ten-core processors, 32M memory,20M band width.The fast sync took about 48 hours.

About 3 days , with an aws lightsail $80 plan.
4Cores, 16GB, 320GB SSD.

wellttllew on 23 Aug 2020

Fast sync took about 10 hours on Amazon i3.xlarge instance (4 vCPUs, 30GB RAM, NVMe hard drive).

575809113 state entries as of 2020-08-25.

I used the following flags:
--cache=4096 (essential for speed improvement, but requires more RAM)
--maxpeers=50

The final .ethereum directory size is about 253 GB.
geth version: 1.9.20-unstable-7b5107b7-20200824

sjlnk on 25 Aug 2020

what is the size of the db directory after fully sync ? I haven't done this for a while(didn't have the freezer feature back then). TIA.

garyng2000 on 25 Aug 2020

what is the size of the db directory after fully sync ? I haven't done this for a while(didn't have the freezer feature back then). TIA.

full sizes without anything moved out
mainnet = 402GB
testnet = 96GB

sirnicolas21 on 25 Aug 2020

i have started fast sync and its was took 3 days
on first day every blocks was synced but its syncing states and now its really slow!

here is syncing result:

{
  "currentBlock": 10735343,
  "highestBlock": 10735450,
  "knownStates": 584833661,
  "pulledStates": 584754308,
  "startingBlock": 10734477
}

here is console result:

INFO [08-26|12:13:23.330] Imported new state entries               count=384  elapsed=6.877ms     processed=584758628 pending=77408  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:27.602] Imported new state entries               count=636  elapsed=7.330ms     processed=584759264 pending=77425  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:32.088] Imported new block headers               count=2    elapsed=11.847ms    number=10735455 hash="71d3b7…0e55db"
INFO [08-26|12:13:32.300] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=193.43KiB throttle=339
INFO [08-26|12:13:33.786] Imported new state entries               count=680  elapsed=7.854ms     processed=584759944 pending=77018  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:40.582] Imported new state entries               count=618  elapsed=10.019ms    processed=584760562 pending=76876  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:46.097] Imported new state entries               count=768  elapsed=14.378ms    processed=584761330 pending=76011  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:52.082] Imported new state entries               count=588  elapsed=10.860ms    processed=584761918 pending=75758  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:13:54.640] Imported new state entries               count=384  elapsed=7.061ms     processed=584762302 pending=75829  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:00.586] Imported new state entries               count=620  elapsed=10.992ms    processed=584762922 pending=75610  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:04.091] Imported new state entries               count=384  elapsed=5.460ms     processed=584763306 pending=75509  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:05.281] Imported new block headers               count=1    elapsed=10.060ms    number=10735456 hash="261277…ec1ae9"
INFO [08-26|12:14:05.316] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=195.86KiB throttle=335
INFO [08-26|12:14:09.136] Imported new state entries               count=638  elapsed=6.682ms     processed=584763944 pending=75456  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:09.166] Imported new block headers               count=1    elapsed=11.364ms    number=10735457 hash="6cd8bc…d18c5a"
INFO [08-26|12:14:11.997] Imported new state entries               count=384  elapsed=1.012ms     processed=584764328 pending=76285  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:17.521] Imported new state entries               count=768  elapsed=2.411ms     processed=584765096 pending=77453  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:20.346] Imported new state entries               count=384  elapsed=1.352ms     processed=584765480 pending=78070  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:22.298] Imported new block headers               count=1    elapsed=12.015ms    number=10735458 hash="c50745…4caf1a"
INFO [08-26|12:14:22.325] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=179.93KiB throttle=365
INFO [08-26|12:14:23.382] Imported new state entries               count=384  elapsed=1.053ms     processed=584765864 pending=78578  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:26.121] Imported new state entries               count=384  elapsed=1.387ms     processed=584766248 pending=79245  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:29.057] Imported new state entries               count=395  elapsed=1.074ms     processed=584766643 pending=79953  retry=4   duplicate=1090 unexpected=4261
INFO [08-26|12:14:32.295] Imported new state entries               count=384  elapsed=1.125ms     processed=584767027 pending=80679  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:32.435] Imported new block headers               count=1    elapsed=12.931ms    number=10735459 hash="7d63b2…ff65b6"
INFO [08-26|12:14:33.331] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=187.34KiB throttle=350
INFO [08-26|12:14:35.170] Imported new state entries               count=384  elapsed=1.509ms     processed=584767411 pending=81137  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:37.587] Imported new state entries               count=384  elapsed="939.928µs" processed=584767795 pending=82376  retry=2   duplicate=1090 unexpected=4261
INFO [08-26|12:14:42.914] Imported new state entries               count=768  elapsed=5.270ms     processed=584768563 pending=83624  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:45.925] Imported new state entries               count=384  elapsed=1.518ms     processed=584768947 pending=84116  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:48.339] Imported new state entries               count=384  elapsed=1.183ms     processed=584769331 pending=85239  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:51.306] Imported new state entries               count=433  elapsed=4.434ms     processed=584769764 pending=85273  retry=0   duplicate=1090 unexpected=4261
INFO [08-26|12:14:52.549] Imported new block headers               count=1    elapsed=10.824ms    number=10735460 hash="499459…5ccd22"
INFO [08-26|12:14:52.972] Imported new block headers               count=1    elapsed=10.718ms    number=10735461 hash="c25179…457362"
INFO [08-26|12:14:53.342] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=187.89KiB throttle=349
INFO [08-26|12:14:54.843] Imported new state entries               count=384  elapsed=1.205ms     processed=584770148 pending=85616  retry=0   duplicate=1090 unexpected=4261

anyone know how many states exist now?
and what can i do?

farhadjafari385 on 26 Aug 2020

Hey @sjlnk. Is your sync completed. What is the logs showing now?
I have been in it for weeks and still getting imported state entries

sssubik on 26 Aug 2020

Hey @sjlnk. Is your sync completed. What is the logs showing now?
I have been in it for weeks and still getting imported state entries

Yes, it's completed. I already reported the results above. Please scroll up. ^^

The --cache setting makes a huge difference but is more demanding for the hardware.

sjlnk on 26 Aug 2020

Hey @sjlnk Did you get Imported Chain segment?
I get this for weeks now:

INFO [08-26|17:09:11.398] Imported new block headers               count=3    elapsed=30.970ms   number=10736801 hash=8df84b…d6ab09
INFO [08-26|17:09:15.920] Initializing fast sync bloom             items=11047301 errorrate=0.000 elapsed=13m20.633s
INFO [08-26|17:09:22.638] Imported new state entries               count=719  elapsed=19.118ms   processed=698898959 pending=3397 retry=0  duplicate=0 unexpected=13
INFO [08-26|17:09:23.924] Initializing fast sync bloom             items=11136084 errorrate=0.000 elapsed=13m28.637s
INFO [08-26|17:09:31.931] Initializing fast sync bloom             items=11215396 errorrate=0.000 elapsed=13m36.644s
INFO [08-26|17:09:32.719] Imported new block headers               count=1    elapsed=31.119ms   number=10736802 hash=01f447…129466
INFO [08-26|17:09:38.130] Imported new state entries               count=324  elapsed=14.996ms   processed=698899283 pending=3309 retry=0  duplic

Also Which GETH version are you using?

sssubik on 26 Aug 2020

Hey @sjlnk Did you get Imported Chain segment?
I get this for weeks now:

INFO [08-26|17:09:11.398] Imported new block headers               count=3    elapsed=30.970ms   number=10736801 hash=8df84b…d6ab09
INFO [08-26|17:09:15.920] Initializing fast sync bloom             items=11047301 errorrate=0.000 elapsed=13m20.633s
INFO [08-26|17:09:22.638] Imported new state entries               count=719  elapsed=19.118ms   processed=698898959 pending=3397 retry=0  duplicate=0 unexpected=13
INFO [08-26|17:09:23.924] Initializing fast sync bloom             items=11136084 errorrate=0.000 elapsed=13m28.637s
INFO [08-26|17:09:31.931] Initializing fast sync bloom             items=11215396 errorrate=0.000 elapsed=13m36.644s
INFO [08-26|17:09:32.719] Imported new block headers               count=1    elapsed=31.119ms   number=10736802 hash=01f447…129466
INFO [08-26|17:09:38.130] Imported new state entries               count=324  elapsed=14.996ms   processed=698899283 pending=3309 retry=0  duplic

Also Which GETH version are you using?

Yes I got that message. The version is also reported above.

sjlnk on 26 Aug 2020

Hey @sjlnk. I dont get it why my state entries is so much greater than yours. But still the sync is not completed:

INFO [08-27|09:38:20.250] Initializing fast sync bloom             items=295539384 errorrate=0.000 elapsed=16h22m25.240s
INFO [08-27|09:38:25.986] Imported new block headers               count=2    elapsed=87.677ms      number=10741287 hash="309adc…db751d"
INFO [08-27|09:38:28.254] Initializing fast sync bloom             items=295539813 errorrate=0.000 elapsed=16h22m33.244s
INFO [08-27|09:38:36.256] Initializing fast sync bloom             items=295540299 errorrate=0.000 elapsed=16h22m41.246s
INFO [08-27|09:38:44.262] Initializing fast sync bloom             items=295540709 errorrate=0.000 elapsed=16h22m49.251s
INFO [08-27|09:38:49.951] Imported new state entries               count=305  elapsed=7.255ms       processed=700468113 pending=77486 retry=0   duplicate=0 unexpected=0

I am already at: 700468113

sssubik on 27 Aug 2020

Hey @sjlnk. I dont get it why my state entries is so much greater than yours. But still the sync is not completed:

INFO [08-27|09:38:20.250] Initializing fast sync bloom             items=295539384 errorrate=0.000 elapsed=16h22m25.240s
INFO [08-27|09:38:25.986] Imported new block headers               count=2    elapsed=87.677ms      number=10741287 hash="309adc…db751d"
INFO [08-27|09:38:28.254] Initializing fast sync bloom             items=295539813 errorrate=0.000 elapsed=16h22m33.244s
INFO [08-27|09:38:36.256] Initializing fast sync bloom             items=295540299 errorrate=0.000 elapsed=16h22m41.246s
INFO [08-27|09:38:44.262] Initializing fast sync bloom             items=295540709 errorrate=0.000 elapsed=16h22m49.251s
INFO [08-27|09:38:49.951] Imported new state entries               count=305  elapsed=7.255ms       processed=700468113 pending=77486 retry=0   duplicate=0 unexpected=0

I am already at: 700468113

after 7 day?

u use ssd or nvme?

my result after 5 day:

INFO [08-27|10:48:43.404] Imported new state entries               count=384  elapsed="4.217µs"   processed=592286844 pending=65731  retry=0   duplicate=1090 unexpected=4261
INFO [08-27|10:48:51.810] Imported new block headers               count=2    elapsed=13.184ms    number=10741600 hash="0df0e9…8ddd8a"
INFO [08-27|10:48:52.667] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=201.71KiB throttle=325
INFO [08-27|10:48:52.808] Imported new state entries               count=384  elapsed="5.05µs"    processed=592287228 pending=66124  retry=0   duplicate=1090 unexpected=4261
INFO [08-27|10:49:02.164] Imported new state entries               count=384  elapsed="6.18µs"    processed=592287612 pending=66519  retry=0   duplicate=1090 unexpected=4261

farhadjafari385 on 27 Aug 2020

I'm still at 540M and counting, but I'll tell you when I'm done.

If we take as good the three stats reported in this thread

| DATE | COUNT |
|:--------|:------------|
|14/04/2020| 487.040.102
|10/05/2020| 504.344.987
|25/08/2020| 575.809.113

there's about 500k~600k new state entries every day and so we should be now around ~580M. To be sure you should count the real entries in the levelDB. You can do it with a separated script that count that, but I was already doing it in a separate PR for geth so if you want you can already try it from here.

I suggest always to build from source, but I uploaded also an already compiled version on my Ubuntu-20.04.1 64 bit.

I will send a PR for the master, but please note that if you take from there your DB will be upgraded to the new schema. Nothing to fear of but, just to say it.

This is for example the output for a goerli testnet full synced few minutes ago, where _trie nodes_ are 24272247. You see the line _contract codes_ because I already upgraded my network for tests.

❯ ./build/bin/geth --goerli --verbosity 3 inspect
INFO [08-27|11:48:29.190] Maximum peer count                       ETH=50 LES=0 total=50
INFO [08-27|11:48:29.190] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [08-27|11:48:29.198] Set global gas cap                       cap=25000000
INFO [08-27|11:48:29.198] Allocated cache and file handles         database=/home/developer/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=524288
INFO [08-27|11:48:29.283] Opened ancient database                  database=/home/developer/.ethereum/goerli/geth/chaindata/ancient
INFO [08-27|11:48:29.313] Persisted trie from memory database      nodes=361 size=51.17KiB time=1.37971ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [08-27|11:48:29.316] Loaded most recent local header          number=3296898 hash="886b5b…2b6a4b" td=4811621 age=14m5s
INFO [08-27|11:48:29.316] Loaded most recent local full block      number=3296898 hash="886b5b…2b6a4b" td=4811621 age=14m5s
INFO [08-27|11:48:29.316] Loaded most recent local fast block      number=3296898 hash="886b5b…2b6a4b" td=4811621 age=14m5s
INFO [08-27|11:48:29.316] Loaded last fast-sync pivot marker       number=3292045
INFO [08-27|11:48:37.405] Inspecting database                      count=7148000 elapsed=8.006s
INFO [08-27|11:48:45.406] Inspecting database                      count=14589000 elapsed=16.007s
INFO [08-27|11:48:53.408] Inspecting database                      count=22353000 elapsed=24.009s
+-----------------+--------------------+------------+----------+
|    DATABASE     |      CATEGORY      |    SIZE    |  COUNT   |
+-----------------+--------------------+------------+----------+
| Key-Value store | Headers            | 12.86 MiB  |    20838 |
| Key-Value store | Bodies             | 39.91 MiB  |    21008 |
| Key-Value store | Receipts           | 36.77 MiB  |    21008 |
| Key-Value store | Difficulties       | 1.01 MiB   |    21472 |
| Key-Value store | Block number->hash | 924.96 KiB |    20580 |
| Key-Value store | Block hash->number | 128.95 MiB |  3297807 |
| Key-Value store | Transaction index  | 10.40 MiB  |   303011 |
| Key-Value store | Bloombit index     | 102.89 MiB |  1646592 |
| Key-Value store | Contract codes     | 97.18 MiB  |    17941 |
| Key-Value store | Trie nodes         | 2.37 GiB   | 24272247 |
| Key-Value store | Trie preimages     | 3.51 MiB   |    49418 |
| Key-Value store | Account snapshot   | 0.00 B     |        0 |
| Key-Value store | Storage snapshot   | 0.00 B     |        0 |
| Key-Value store | Clique snapshots   | 79.33 KiB  |       98 |
| Key-Value store | Singleton metadata | 151.00 B   |        5 |
| Ancient store   | Headers            | 1.04 GiB   | NA       |
| Ancient store   | Bodies             | 1.09 GiB   | NA       |
| Ancient store   | Receipts           | 596.63 MiB | NA       |
| Ancient store   | Difficulties       | 31.21 MiB  | NA       |
| Ancient store   | Block number->hash | 118.75 MiB | NA       |
| Light client    | CHT trie nodes     | 0.00 B     |        0 |
| Light client    | Bloom trie nodes   | 0.00 B     |        0 |
+-----------------+--------------------+------------+----------+
|                         TOTAL        |  5.65 GIB  |           
+-----------------+--------------------+------------+----------+
ERROR[08-27|11:48:58.711] Database contains unaccounted data       size=100.74KiB count=3514

Neurone on 27 Aug 2020

Hey @sjlnk. I dont get it why my state entries is so much greater than yours. But still the sync is not completed:

INFO [08-27|09:38:20.250] Initializing fast sync bloom             items=295539384 errorrate=0.000 elapsed=16h22m25.240s
INFO [08-27|09:38:25.986] Imported new block headers               count=2    elapsed=87.677ms      number=10741287 hash="309adc…db751d"
INFO [08-27|09:38:28.254] Initializing fast sync bloom             items=295539813 errorrate=0.000 elapsed=16h22m33.244s
INFO [08-27|09:38:36.256] Initializing fast sync bloom             items=295540299 errorrate=0.000 elapsed=16h22m41.246s
INFO [08-27|09:38:44.262] Initializing fast sync bloom             items=295540709 errorrate=0.000 elapsed=16h22m49.251s
INFO [08-27|09:38:49.951] Imported new state entries               count=305  elapsed=7.255ms       processed=700468113 pending=77486 retry=0   duplicate=0 unexpected=0

I am already at: 700468113

That's strange. I would like to know also why this happens. Perhaps someone more knowledgeable could chime in and enlighten us...

sjlnk on 27 Aug 2020

@sjlnk it's because you've stopped and restarted sync several times. Or so I presume, since it's still Initializing fast sync bloom. Every time you restart, geth goes into a mode where, while importing state, also reads the entire db to create a bloom filter of existing states. Which slows down sync. Also, every time you restart, you then choose a new pivot point to sync to, so more state to download.

@karalabe just merged a great PR that makes fast-sync a lot more stable, and less prone to crashing due to out of memory. I would recommend to update to latest master and either start from scratch or continue (yes, with another restart -- can't be helped).

holiman on 27 Aug 2020

@sjlnk actually, looking at it again, I suspect you have bad read IO. It has only gone through 29M entries for the bloom filter, in 16+h, which is very slow indeed. Might want to check that your disk is good. It's not an HDD, is it? And if it's an SSD, those can get degraded pretty badly with time.

holiman on 27 Aug 2020

I synced a 3 month gap from last sync, took 2.5 weeks to complete. Its an array of 6 10k rpm sas drives in raid 5. Adding 2 more soon. Took about 2 or more tb of write volume. It really is an ssd killer.

cncr04s on 27 Aug 2020

@sjlnk it's because you've stopped and restarted sync several times. Or so I presume, since it's still Initializing fast sync bloom. Every time you restart, geth goes into a mode where, while importing state, also reads the entire db to create a bloom filter of existing states. Which slows down sync. Also, every time you restart, you then choose a new pivot point to sync to, so more state to download.

@karalabe just merged a great PR that makes fast-sync a lot more stable, and less prone to crashing due to out of memory. I would recommend to update to latest master and either start from scratch or continue (yes, with another restart -- can't be helped).

Thanks for clearing it out! Actually my sync completed in 10 hours with a powerful AWS NVMe enabled instance. I just wanted to know why some people have to download more states than others. I never restarted the process so I'm assuming the other people here did and that's why they have so many more states to download.

sjlnk on 27 Aug 2020

Hey @holiman . The logs are from my VPS server. It is an SSD. Does this mean that my server is not powerful enough as it has very less i/o speed. Do I need to upgrade it?

sssubik on 28 Aug 2020

Hey @sjlnk . Could you give the specification of your server? Because I too have a good server.. But it has not completed for like a month. What is your Disk I/O rate specifically..

sssubik on 29 Aug 2020

I'm done with counter at 652299180. Please note that in my case the processed counter at the end was bigger of about ~12M of entries vs the real entries stored in levelDB. I restarted geth during the fast sync a lot of times in these days.

INFO [08-30|10:01:52.403] Imported new state entries               count=1605 elapsed=22.003ms   processed=652296924 pending=3467   retry=0    duplicate=41127 unexpected=121037
INFO [08-30|10:01:53.302] Imported new state entries               count=1157 elapsed=11.998ms   processed=652298081 pending=2059   retry=0    duplicate=41127 unexpected=121037
INFO [08-30|10:01:54.214] Imported new state entries               count=1001 elapsed=14.992ms   processed=652299082 pending=348    retry=0    duplicate=41127 unexpected=121037
INFO [08-30|10:01:54.523] Imported new state entries               count=98   elapsed=2.837ms    processed=652299180 pending=0      retry=0    duplicate=41127 unexpected=121037
INFO [08-30|10:01:54.535] Imported new block receipts              count=1    elapsed=2ms        number=10760885 hash="e5569e…7ead3c" age=29m44s   size=101.58KiB
INFO [08-30|10:01:54.545] Committed new head block                 number=10760885 hash="e5569e…7ead3c"
INFO [08-30|10:01:54.613] Deallocated fast sync bloom              items=639248389 errorrate=0.001
INFO [08-30|10:02:03.025] Imported new chain segment               blocks=12 txs=2533 mgas=149.011 elapsed=8.402s     mgasps=17.735 number=10760897 hash="f8d397…0dafd0" age=26m49s   dirty=18.46MiB
INFO [08-30|10:02:11.409] Imported new chain segment               blocks=14 txs=3369 mgas=174.232 elapsed=8.384s     mgasps=20.780 number=10760911 hash="1adada…67ecc0" age=21m55s   dirty=41.57MiB
INFO [08-30|10:02:19.956] Imported new chain segment               blocks=18 txs=3487 mgas=210.546 elapsed=8.546s     mgasps=24.636 number=10760929 hash="97dcea…9d023b" age=16m20s   dirty=68.14MiB
INFO [08-30|10:02:27.303] New local node record                    seq=39281 id=82090fbe6ea27663 ip=127.0.0.1     udp=30303 tcp=30303
INFO [08-30|10:02:27.389] New local node record                    seq=39282 id=82090fbe6ea27663 ip=93.40.103.165 udp=30303 tcp=30303
INFO [08-30|10:02:28.037] Imported new chain segment               blocks=17 txs=3815 mgas=209.530 elapsed=8.081s     mgasps=25.927 number=10760946 hash="4165bb…e09268" age=11m39s   dirty=94.33MiB
INFO [08-30|10:02:36.621] Imported new chain segment               blocks=12 txs=2487 mgas=136.747 elapsed=8.584s     mgasps=15.930 number=10760958 hash="c59ba3…d33c9e" age=8m32s    dirty=111.89MiB
INFO [08-30|10:02:40.190] New local node record                    seq=39283 id=82090fbe6ea27663 ip=127.0.0.1     udp=30303 tcp=30303
INFO [08-30|10:02:45.795] New local node record                    seq=39284 id=82090fbe6ea27663 ip=93.40.103.165 udp=30303 tcp=30303
INFO [08-30|10:02:45.895] Imported new chain segment               blocks=10 txs=2042 mgas=120.584 elapsed=9.273s     mgasps=13.003 number=10760968 hash="82b2bd…ea0969" age=6m43s    dirty=126.93MiB
INFO [08-30|10:02:46.400] Deep froze chain segment                 blocks=30001 elapsed=18.624s    number=10057008 hash="ffaddb…05d9b1"
INFO [08-30|10:02:54.190] Imported new chain segment               blocks=13    txs=2915 mgas=161.846 elapsed=8.295s     mgasps=19.511 number=10760981 hash="e653ff…70529a" age=3m40s    dirty=146.58MiB
INFO [08-30|10:03:00.113] Imported new chain segment               blocks=7     txs=1296 mgas=87.019  elapsed=5.923s     mgasps=14.691 number=10760988 hash="a655a4…7c6ca5" age=1m31s    dirty=156.69MiB
INFO [08-30|10:03:00.126] Imported new block headers               count=1    elapsed=1m4.052s   number=10760989 hash="31df12…bc9465" age=1m30s
INFO [08-30|10:03:00.139] Imported new block headers               count=1    elapsed=4.998ms    number=10760990 hash="04c374…9a92dc" age=1m27s
INFO [08-30|10:03:00.153] Imported new block headers               count=3    elapsed=6.000ms    number=10760993 hash="c6fffb…265cae"
INFO [08-30|10:03:00.225] Downloader queue stats                   receiptTasks=0    blockTasks=0    itemSize=221.62KiB throttle=296
INFO [08-30|10:03:00.997] Imported new chain segment               blocks=1     txs=235  mgas=12.392  elapsed=763.005ms  mgasps=16.241 number=10760989 hash="31df12…bc9465" age=1m30s    dirty=158.37MiB
INFO [08-30|10:03:01.008] Imported new block headers               count=1    elapsed=572.995ms  number=10760994 hash="e3afed…f9afef"
INFO [08-30|10:03:03.377] Imported new chain segment               blocks=1     txs=200  mgas=12.423  elapsed=2.361s     mgasps=5.261  number=10760990 hash="04c374…9a92dc" age=1m30s    dirty=160.05MiB
INFO [08-30|10:03:04.270] Deep froze chain segment                 blocks=30001 elapsed=17.860s    number=10087009 hash="e2f6c6…fe7f20"
WARN [08-30|10:03:07.574] Fast syncing, discarded propagated block number=10760994 hash="e3afed…f9afef"
INFO [08-30|10:03:08.411] Imported new chain segment               blocks=4     txs=666  mgas=49.755  elapsed=5.023s     mgasps=9.905  number=10760994 hash="e3afed…f9afef" dirty=165.73MiB
INFO [08-30|10:03:08.422] Fast sync complete, auto disabling

Detailed DB stats

> geth --datadir s:\Ethereum inspect
INFO [08-30|12:49:32.043] Maximum peer count                       ETH=50 LES=0 total=50
INFO [08-30|12:49:32.388] Set global gas cap                       cap=25000000
INFO [08-30|12:49:32.393] Allocated cache and file handles         database=s:\Ethereum\geth\chaindata cache=512.00MiB handles=8192
INFO [08-30|12:49:34.040] Opened ancient database                  database=s:\Ethereum\geth\chaindata\ancient
INFO [08-30|12:49:34.074] Disk storage enabled for ethash caches   dir=s:\Ethereum\geth\ethash count=3
INFO [08-30|12:49:34.079] Disk storage enabled for ethash DAGs     dir=C:\Users\giuse\AppData\Local\Ethash count=2
INFO [08-30|12:49:34.090] Loaded most recent local header          number=10761038 hash="95d315…aac7a9" td=17106408363816477706008 age=2h38m10s
INFO [08-30|12:49:34.099] Loaded most recent local full block      number=10760905 hash="a638f7…ecd63c" td=17106043165864029076711 age=3h10m31s
INFO [08-30|12:49:34.106] Loaded most recent local fast block      number=10761038 hash="95d315…aac7a9" td=17106408363816477706008 age=2h38m10s
INFO [08-30|12:49:34.107] Deep froze chain segment                 blocks=18 elapsed=33.995ms number=10670905 hash="8c3832…cb9018"
INFO [08-30|12:49:34.115] Loaded last fast-sync pivot marker       number=10760885
INFO [08-30|12:49:42.130] Inspecting database                      count=6482000 elapsed=8.001s
...
INFO [08-30|13:22:16.661] Counting ancient database receipts       blocknumber=10670906 percentage=100 elapsed=32m42.533s
+-----------------+--------------------+------------+-----------+
|    DATABASE     |      CATEGORY      |    SIZE    |   ITEMS   |
+-----------------+--------------------+------------+-----------+
| Key-Value store | Headers            | 55.00 MiB  |    100554 |
| Key-Value store | Bodies             | 3.64 GiB   |     98505 |
| Key-Value store | Receipts           | 4.20 GiB   |  17751843 |
| Key-Value store | Difficulties       | 6.25 MiB   |    110238 |
| Key-Value store | Block number->hash | 5.16 MiB   |    110222 |
| Key-Value store | Block hash->number | 420.76 MiB |  10761043 |
| Key-Value store | Transaction index  | 27.44 GiB  | 818569194 |
| Key-Value store | Bloombit index     | 1.64 GiB   |   5380096 |
| Key-Value store | Contract codes     | 37.33 MiB  |      6636 |
| Key-Value store | Trie nodes         | 74.51 GiB  | 639023322 |
| Key-Value store | Trie preimages     | 1.07 MiB   |     17051 |
| Key-Value store | Account snapshot   | 0.00 B     |         0 |
| Key-Value store | Storage snapshot   | 0.00 B     |         0 |
| Key-Value store | Clique snapshots   | 0.00 B     |         0 |
| Key-Value store | Singleton metadata | 151.00 B   |         5 |
| Ancient store   | Headers            | 4.51 GiB   |  10670906 |
| Ancient store   | Bodies             | 97.85 GiB  |  10670906 |
| Ancient store   | Receipt lists      | 44.38 GiB  |  10670906 |
| Ancient store   | └ counted receipts | --         | 802441861 |
| Ancient store   | Difficulties       | 166.01 MiB |  10670906 |
| Ancient store   | Block number->hash | 386.71 MiB |  10670906 |
| Light client    | CHT trie nodes     | 0.00 B     |         0 |
| Light client    | Bloom trie nodes   | 0.00 B     |         0 |
+-----------------+--------------------+------------+-----------+
|                         TOTAL        | 259.22 GIB |           |
+-----------------+--------------------+------------+-----------+
ERROR[08-30|13:22:16.783] Database contains unaccounted data       size=121.08KiB count=2632

Neurone on 30 Aug 2020

Hey @Neurone How many days does the full sync take in average? How can I know how close am I? I have been syncing for like a month now..

sssubik on 31 Aug 2020

Hey @sjlnk . Could you give the specification of your server? Because I too have a good server.. But it has not completed for like a month. What is your Disk I/O rate specifically..

I used i3.xlarge on Amazon EC2.

sjlnk on 31 Aug 2020

Finished a fast sync today (block 10770930) with:

processed=608439377

(_was running it on a work machine during day time, so the total time is irrelevant in my case_)

splix on 31 Aug 2020

Hey @Neurone How many days does the full sync take in average? How can I know how close am I? I have been syncing for like a month now..

Not easy to say because I tested many things during the sync process, I restarted a lot of times (40~50 times) and I moved the files around different systems. In any case I searched for all the logs I saved between restarts (not all) and I created the summary below, just to have an idea.

The most of the time is spent for the fast sync of state entries, but block headers are downloaded in few hours in any case. So you should check your stats and calculate your rate to download state entries per hour: if it's high enough, I suggest you to delete the database - not the ancient - and download state entries again from the start.

If your rate is like mine at the end (~8.2M entries per hour) you should be able to reach the current state from scratch in ~3 days.

| SYSTEM / OS|RAM GB|DISK| HOURS ONLINE | RESTARTS | STATE ENTRIES REACHED | STATE ENTRIES PER HOUR | TIME TO INIT FAST SYNC BLOOM WHEN RESTARTED |
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| Rock64 / Armbian 20.02.1 Bionic| 4 | HDD | ~1.800 | ~40 | 469.380.958 | ~100k (decreasing over time, this is the mean value in the last 26 days) | 3h16m29|
| Desktop / Ubuntu 20.04 (WSL2 on Windows 10) | 16 | SSD | 134 | 1 | 578.598.409 | ~815k | 2h38m14|
| Desktop / Windows 10 | 16 | SSD | 9 | 0 | 652.299.180 | ~8.2M | 30m28|

Desktop:
- Processor: Intel i7-3820 3.6Ghz (2012)
- SSD: Crucial CT525MX300SSD1 (2016)

Neurone on 31 Aug 2020

So you should check your stats and calculate your rate to download state entries per hour: if it's high enough, I suggest you to delete the database - not the ancient - and download state entries again from the start.

Hi @Neurone thanks for adding a few bits of your data. I am trying to figure out if I have a high enough rate. Currently looking at about 600k per hour on a 8GB Pi 4, 1 TB Samsung SSD. It has been hard to find info on what might be a high enough rate. I just took some guessed stationary value of 600M and made myself some moving goalposts of low estimated 3000 state changes per block and high estimate 6000 state changes per block. Which would put me at needing 720k-1.44M per hour to just keep up with the chain minimum. What rate do you think is reasonable?

I have been considering deleting the chain and starting again to see if I have just fallen too far behind. Also do you know if the removedb command would keep ancient by specifying the datadir even if they are both in chaindata folder?

Thanks!

Duncan-Brain on 10 Nov 2020

So you should check your stats and calculate your rate to download state entries per hour: if it's high enough, I suggest you to delete the database - not the ancient - and download state entries again from the start.

Hi @Neurone thanks for adding a few bits of your data. I am trying to figure out if I have a high enough rate. Currently looking at about 600k per hour on a 8GB Pi 4, 1 TB Samsung SSD. It has been hard to find info on what might be a high enough rate. I just took some guessed stationary value of 600M and made myself some moving goalposts of low estimated 3000 state changes per block and high estimate 6000 state changes per block. Which would put me at needing 720k-1.44M per hour to just keep up with the chain minimum. What rate do you think is reasonable?

I have been considering deleting the chain and starting again to see if I have just fallen too far behind. Also do you know if the removedb command would keep ancient by specifying the datadir even if they are both in chaindata folder?

Thanks!

`INFO [11-10|15:05:14.458] Imported new state entries count=384 elapsed="11.648µs" processed=501571121 pending=49648 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:05:19.526] Imported new block headers count=1 elapsed=242.947ms number=11229982 hash="aff1eb▒616e67" age=1m18s
INFO [11-10|15:05:22.529] Imported new state entries count=384 elapsed="8.352µs" processed=501571505 pending=50058 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:05:30.183] Imported new state entries count=384 elapsed="180.219µs" processed=501571889 pending=50553 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:05:43.575] Imported new state entries count=384 elapsed="288.309µs" processed=501572273 pending=50935 trieretry=4 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:05:51.439] Imported new state entries count=386 elapsed="190.904µs" processed=501572659 pending=51429 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:05:59.872] Imported new state entries count=384 elapsed="8.074µs" processed=501573043 pending=51883 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:06:11.695] Imported new state entries count=384 elapsed="212.496µs" processed=501573427 pending=52268 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:06:19.745] Imported new block headers count=1 elapsed=568.387ms number=11229983 hash="d10a7b▒22dbf3" age=2m9s
INFO [11-10|15:06:20.106] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=216.91KiB throttle=303
INFO [11-10|15:06:22.905] Imported new state entries count=384 elapsed="17.499µs" processed=501573811 pending=52654 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:06:25.797] Imported new block headers count=1 elapsed=197.033ms number=11229984 hash="624d1f▒04161a" age=1m25s
INFO [11-10|15:06:33.955] Imported new state entries count=384 elapsed="8.407µs" processed=501574195 pending=53041 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:06:38.346] Imported new block headers count=1 elapsed=41.740ms number=11229985 hash="530417▒f68043" age=1m20s
INFO [11-10|15:06:44.704] Imported new state entries count=384 elapsed="8.907µs" processed=501574579 pending=53428 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:06:47.901] Imported new block headers count=1 elapsed=31.135ms number=11229986 hash="11688f▒0590b5"
INFO [11-10|15:06:51.181] Imported new block headers count=1 elapsed=20.278ms number=11229987 hash="2e93db▒b5fa93"
INFO [11-10|15:06:55.595] Imported new state entries count=384 elapsed="163.312µs" processed=501574963 pending=53811 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:07:05.701] Imported new state entries count=384 elapsed="16.314µs" processed=501575347 pending=54197 trieretry=0 coderetry=0 duplicate=15215 unexpected=73697
INFO [11-10|15:07:13.349] Imported new block headers count=1 elapsed=27.639ms number=11229988 hash="9b7a99▒8cd706"

`
@Duncan-Brain
this is a sample of my rasp4 (4gb ram) with samsung T5 ssd
it says its running 4 days but i think i restarted once due to power failure..... so it should be 4-5 days
note that its running a full 64bit Raspbian

sirnicolas21 on 10 Nov 2020

@sirnicolas21 I am sorry I am confused by your response here. What is your rate? It looks to me like you are in the importing state trie phase and are about ~80 blocks behind still? I have been running for 15 days been in the state trie phase for 12 days. I have been upping my --cache allocation to try to increase my rate as described above. I can see you have ~196M less state entries than I so it seems like you may have done something right, or have a ways to go. What geth options did you specify? Thanks!

Duncan-Brain on 10 Nov 2020

@sirnicolas21 I am sorry I am confused by your response here. What is your rate? It looks to me like you are in the importing state trie phase and are about ~80 blocks behind still? I have been running for 15 days been in the state trie phase for 12 days. I have been upping my --cache allocation to try to increase my rate as described above. I can see you have ~196M less state entries than I so it seems like you may have done something right, or have a ways to go. What geth options did you specify? Thanks!

ps -ef | grep geth
pi        1015   757 39 Nov05 pts/1    2-00:59:02 geth --datadir /data/nodes/geth --cache 1024

@Duncan-Brain i only configured the cache on geth nothing else "--cache 1024"

make sure you have your disk formatted on ext4 and not ntfs
i also had to enable swap since 4 gb is not much ram, i doubt you need it but give it a try
make sure you are running raspbian 64 and not 32, default installation is 32 but that will fail badly
make sure that the disk is on the fast usb (the blue ones)

also take note that on the fast sync phase it doesn't matter if i am 80 blocks behind because i must reach the 600mil entries point to fully sync and then use the blockchain, before that ethereum is just resource eating thing and nothing more

right now i am @ 502983948 entries and my previous post 6 hours ago was 501575347 so thats 240.000 entries per hour aprox

after the initial sync phase you download the blocks and parse them one by one and the process is slightly different than what is happening now

and last but not least.... make sure you are not overheating, personally i made a have a small heatsink and fan running on the rpi with auto turn on at around 60c

sirnicolas21 on 10 Nov 2020

Hi @sirnicolas21 . Thanks for the discussion I have all those suggestions implemented (I am using Kauri.io raspberry pi setup guide with a few tweaks ie --cache 2048). As per this discussion thread I am trying to find an upper bound on the number of states (which I understand is a moving target) so that I am confident that after 15 days I can tell myself "I am almost fully synced" or "I need to start over". Over in this thread on Issue #15616 some nodes go up to ~800M some ~650M.

As you are not yet fully synced how do you have confidence that it will fully sync ever?

You can check out some of my pi stats here on a google sheet - Google Sheet

Duncan-Brain on 10 Nov 2020

So you should check your stats and calculate your rate to download state entries per hour: if it's high enough, I suggest you to delete the database - not the ancient - and download state entries again from the start.

Hi @Neurone thanks for adding a few bits of your data. I am trying to figure out if I have a high enough rate. Currently looking at about 600k per hour on a 8GB Pi 4, 1 TB Samsung SSD. It has been hard to find info on what might be a high enough rate. I just took some guessed stationary value of 600M and made myself some moving goalposts of low estimated 3000 state changes per block and high estimate 6000 state changes per block. Which would put me at needing 720k-1.44M per hour to just keep up with the chain minimum. What rate do you think is reasonable?

I have been considering deleting the chain and starting again to see if I have just fallen too far behind. Also do you know if the removedb command would keep ancient by specifying the datadir even if they are both in chaindata folder?

Thanks!

Hi @Duncan-Brain I can't remember my state importing rate but I can share the current stat of my fullly synced node so you can have an idea of numbers you need to reach to get a fully synced node (currently 880M state entries). About removing the db, when you execute removedb command it lets you choose if removing ancient db also: I suggest to keep it because there's no gain in removing it (trie state entries are always stored in leveldb, they never go inside the ancient db)

D:\Program Files\Geth>geth --datadir s:\Ethereum inspect
INFO [11-11|19:19:15.793] Maximum peer count                       ETH=50 LES=0 total=50
INFO [11-11|19:19:16.156] Set global gas cap                       cap=25000000
INFO [11-11|19:19:16.160] Allocated cache and file handles         database=s:\Ethereum\geth\chaindata cache=512.00MiB handles=8192
INFO [11-11|19:19:27.022] Opened ancient database                  database=s:\Ethereum\geth\chaindata\ancient
INFO [11-11|19:19:27.046] Disk storage enabled for ethash caches   dir=s:\Ethereum\geth\ethash count=3
INFO [11-11|19:19:27.051] Disk storage enabled for ethash DAGs     dir=C:\Users\giuse\AppData\Local\Ethash count=2
INFO [11-11|19:19:27.059] Loaded most recent local header          number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.069] Loaded most recent local full block      number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.079] Loaded most recent local fast block      number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.087] Loaded last fast-sync pivot marker       number=10760885
INFO [11-11|19:19:35.100] Inspecting database                      count=4996000 elapsed=8.002s
INFO [11-11|19:19:43.108] Inspecting database                      count=10062000 elapsed=16.010s
INFO [11-11|19:19:51.115] Inspecting database                      count=14730000 elapsed=24.017s
...
INFO [11-11|19:50:53.083] Inspecting database                      count=1821919000 elapsed=31m25.985s
INFO [11-11|19:51:01.090] Inspecting database                      count=1828550000 elapsed=31m33.992s
INFO [11-11|19:51:09.097] Inspecting database                      count=1835247000 elapsed=31m41.999s
+-----------------+--------------------+------------+-----------+
|    DATABASE     |      CATEGORY      |    SIZE    |   ITEMS   |
+-----------------+--------------------+------------+-----------+
| Key-Value store | Headers            | 54.97 MiB  |    100438 |
| Key-Value store | Bodies             | 3.52 GiB   |     98390 |
| Key-Value store | Receipt lists      | 4.48 GiB   |     98390 |
| Key-Value store | Difficulties       | 7.11 MiB   |    113813 |
| Key-Value store | Block number->hash | 6.04 MiB   |    113813 |
| Key-Value store | Block hash->number | 439.41 MiB |  11237928 |
| Key-Value store | Transaction index  | 30.13 GiB  | 898783141 |
| Key-Value store | Bloombit index     | 1.76 GiB   |   5617664 |
| Key-Value store | Contract codes     | 264.01 MiB |     44710 |
| Key-Value store | Trie nodes         | 130.27 GiB | 880287080 |
| Key-Value store | Trie preimages     | 2.98 GiB   |  45170757 |
| Key-Value store | Account snapshot   | 0.00 B     |         0 |
| Key-Value store | Storage snapshot   | 0.00 B     |         0 |
| Key-Value store | Clique snapshots   | 0.00 B     |         0 |
| Key-Value store | Singleton metadata | 151.00 B   |         5 |
| Ancient store   | Headers            | 4.75 GiB   |  11147907 |
| Ancient store   | Bodies             | 108.13 GiB |  11147907 |
| Ancient store   | Receipt lists      | 50.63 GiB  |  11147907 |
| Ancient store   | Difficulties       | 173.74 MiB |  11147907 |
| Ancient store   | Block number->hash | 404.00 MiB |  11147907 |
| Light client    | CHT trie nodes     | 0.00 B     |         0 |
| Light client    | Bloom trie nodes   | 0.00 B     |         0 |
+-----------------+--------------------+------------+-----------+
|                         TOTAL        | 337.97 GIB |           |
+-----------------+--------------------+------------+-----------+
ERROR[11-11|19:51:16.769] Database contains unaccounted data       size=126.40KiB count=2748

Neurone on 12 Nov 2020

SYSTEM / OS RAM GB DISK HOURS ONLINE RESTARTS STATE ENTRIES REACHED STATE ENTRIES PER HOUR TIME TO INIT FAST SYNC BLOOM WHEN RESTARTED
Rock64 / Armbian 20.02.1 Bionic 4 HDD ~1.800 ~40 469.380.958 ~100k (decreasing over time, this is the mean value in the last 26 days) 3h16m29
Desktop / Ubuntu 20.04 (WSL2 on Windows 10) 16 SSD 134 1 578.598.409 ~815k 2h38m14
Desktop / Windows 10 16 SSD 9 0 652.299.180 ~8.2M 30m28

@Neurone Thanks that is helpful I think. Based on the data on my setup I think I am falling behind.

Did the second test setup(Desktop Ubuntu) in your table complete sync? Or did only the last setup(Desktop Windows 10) complete sync?

Duncan-Brain on 12 Nov 2020

Yes, the only node that was able to fully sync it was the desktop one because it was using an SSD (it was Ubuntu on WSL2, now it is Windows because of how bad WSL2 handles intensive disk activities, you can see a x10 difference in performance in the table). My SSD is old and not really fast, but I was able to sync at the end even though with many more state entries then normal (880M vs ~700M if I resync again from scratch). I plan to move all geth data to an external SSD and put it again on my Rock64 (that now it is running only Bitcoin and IPFS ^^) but I think re-syncing again the state is not worth the effort because I will be able to free only about ~25 Gb with a grand total of 338Gb, so I think I'm ok with that for now.

Neurone on 12 Nov 2020

@Duncan-Brain how many it says on processed? mine is now 518m and at this point i can say it started to be really slow

sirnicolas21 on 12 Nov 2020

@Neurone okay thanks, last question from me: What makes you think if you resync again it would be ~700M? And for that matter 25Gb freed up... it seems it could be part of the same calculation. Based on your numbers(~652M August31 880M Today) I would say the average of 3000 states per block seems quite accurate once fully synced. But as you suggest there is some pruning possible there. If I am in fast sync I may yet catch up, if I need to beat 3000 states per block right now then I have been behind for 2 weeks.

@sirnicolas21 I updated my spreadsheet from my previous post so you can see -- currently ~710M

Duncan-Brain on 12 Nov 2020

@Duncan-Brain About ~700M is just a _guess in the middle_ based on stats of other people, stats taken even from this issue: https://github.com/ethereum/go-ethereum/issues/15616. It seems the most fortunate ones - i.e. https://github.com/ethereum/go-ethereum/issues/15616#issuecomment-720311292 - get fully synced with ~620M entries, while less fortunate ones reach ~800M. With my ~880M I think I'm an _extreme_ case because I restarted the process so many times, and I don't want to do it again from scratch, so let's say I'll stop to ~700M if I restart all the state sync process. Because my trie nodes take ~130Gb of disk (you can see it in the inspect stats) it means every 1M states take ~147Mb. So if I save ~180M states (880-700) I'll gain ~26GB of disk.

My disk is a 512GB SSD dedicated to Ethereum, the chain grows relatively slowly and, if I will need more space, I still have those ~150GB of ancient data to move outside the SSD, so for now I prefer to keep things as they are until it will be really needed.

Neurone on 12 Nov 2020

👍1

My disk is a 512GB SSD dedicated to Ethereum, the chain grows relatively slowly and, if I will need more space, I still have those ~150GB of ancient data to move outside the SSD, so for now I prefer to keep things as they are until it will be really needed.

@rjl493456442 is working on pruning, which will be able to remove some of the junk which isn't needed.

holiman on 12 Nov 2020

👍1

Fresh sync stats:

Command:

geth --http

There were 640,610,869 state entries
Full sync took roughly 37 hours
Takes 297G

vogelito on 1 Dec 2020

👍4

How were there only 640,610,869 state entries 5 days ago?
I'm already at 698,492,860 still not done syncing.

AdvancedStyle on 6 Dec 2020

How were there only 640,610,869 state entries 5 days ago?
I'm already at 698,492,860 still not done syncing.

State entries are different for each one. @holiman explained why it is so here: https://github.com/ethereum/go-ethereum/issues/14647#issuecomment-682098181

Here another open thread where you can find check stats and experiences from others: https://github.com/ethereum/go-ethereum/issues/15616

And here data of a eth1 full node that I finished to sync yesterday after 64 hours, processing 650.634.632 state entries, using 282.75 GIB. Full stats below.

Please note that during synchronization the same machine was running a Teku beacon node, so performance could have been better without it. I wanted to test if those two clients can both run smoothly on this mini PC and, spoiler alert, yes they can 😉.

KEY | VALUE
--- | ---
Fast sync complete | 64 hours
Processed state entries | 650.634.632
Storage used | 282.75 GIB
OS | Ubuntu 20.04.1
Client | Geth/v1.9.24-stable-cc05b050/linux-amd64/go1.15.5
Machine | MINISFORUM DeskMini UM300 Mini PC
Processor | AMD Ryzen™ 3 3300U , 4 Cores/4 Threads
Memory | DDR4 8GB Dual channel (min. 1 GB reserved for graphic card, so actual RAM 7GB!)
Storage n.1 | Kingston M.2 2280 256GB SATA SSD _(used by Teku)_
Storage n.2 | Crucial BX500 1TB SATA SSD _(used by Geth)_
Connectivity | Ethernet 1Gb
ISP | Fastweb, Italy

./geth
INFO [12-02|03:57:17.926] Starting Geth on Ethereum mainnet...
INFO [12-02|03:57:17.927] Bumping default cache on mainnet         provided=1024 updated=4096
WARN [12-02|03:57:17.927] Sanitizing cache to Go's GC limits       provided=4096 updated=2323
INFO [12-02|03:57:17.934] Maximum peer count                       ETH=50 LES=0 total=50
INFO [12-02|03:57:17.934] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [12-02|03:57:17.936] Set global gas cap                       cap=25000000
INFO [12-02|03:57:17.936] Allocated trie memory caches             clean=580.00MiB dirty=580.00MiB
...
INFO [12-02|03:57:18.015] Writing default main-net genesis block
INFO [12-02|03:57:18.279] Persisted trie from memory database      nodes=12356 size=1.78MiB time=84.346878ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [12-02|03:57:18.279] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v2: <nil>, Engine: ethash}"
...
INFO [12-02|03:57:18.280] Initialising Ethereum protocol           versions="[65 64 63]" network=1 dbversion=<nil>
WARN [12-02|03:57:18.280] Upgrade blockchain database version      from=<nil> to=8
INFO [12-02|03:57:18.281] Loaded most recent local header          number=0 hash="d4e567…cb8fa3" td=17179869184 age=51y7mo4w
INFO [12-02|03:57:18.281] Loaded most recent local full block      number=0 hash="d4e567…cb8fa3" td=17179869184 age=51y7mo4w
INFO [12-02|03:57:18.281] Loaded most recent local fast block      number=0 hash="d4e567…cb8fa3" td=17179869184 age=51y7mo4w
INFO [12-02|03:57:18.281] Regenerated local transaction journal    transactions=0 accounts=0
INFO [12-02|03:57:18.298] Allocated fast sync bloom                size=1.13GiB
INFO [12-02|03:57:18.304] Starting peer-to-peer node               instance=Geth/v1.9.24-stable-cc05b050/linux-amd64/go1.15.5
...
INFO [12-02|03:57:18.667] Initialized fast sync bloom              items=12356 errorrate=0.000 elapsed=367.872ms
...
INFO [12-02|03:57:28.334] Block synchronisation started
...
INFO [12-02|03:57:35.319] Imported new state entries               count=1152 elapsed=5.229ms     processed=45201 pending=20345 trieretry=0 coderetry=0 duplicate=0 unexpected=0
INFO [12-02|03:57:35.425] Imported new state entries               count=1152 elapsed=4.800ms     processed=46353 pending=20673 trieretry=0 coderetry=0 duplicate=0 unexpected=0
INFO [12-02|03:57:35.751] Imported new state entries               count=1536 elapsed=8.549ms     processed=47889 pending=19409 trieretry=0 coderetry=0 duplicate=0 unexpected=0
...
INFO [12-04|20:52:06.008] Imported new state entries               count=1996 elapsed=62.015ms    processed=650631754 pending=9510   trieretry=0    coderetry=0 duplicate=126032 unexpected=765459
INFO [12-04|20:52:06.239] Imported new state entries               count=2191 elapsed=141.737ms   processed=650633945 pending=2499   trieretry=0    coderetry=0 duplicate=126032 unexpected=765459
INFO [12-04|20:52:10.981] Imported new state entries               count=687  elapsed=45.228ms    processed=650634632 pending=0      trieretry=0    coderetry=0 duplicate=126032 unexpected=765459
INFO [12-04|20:52:10.998] Committed new head block                 number=11388325 hash="4dfe95…2574b4"
INFO [12-04|20:52:11.049] Deallocated fast sync bloom              items=646967270 errorrate=0.006
...
INFO [12-04|20:52:11.049] Deallocated fast sync bloom              items=646967270 errorrate=0.006
...
INFO [12-04|20:58:18.180] Fast sync complete, auto disabling

./geth inspect
INFO [12-04|22:37:47.671] Maximum peer count                       ETH=50 LES=0 total=50
INFO [12-04|22:37:47.671] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [12-04|22:37:47.672] Set global gas cap                       cap=25000000
...
INFO [12-04|22:37:52.507] Loaded most recent local header          number=11388879 hash="0ec041…573550" td=19169271088158317937607 age=3m28s
INFO [12-04|22:37:52.507] Loaded most recent local full block      number=11388879 hash="0ec041…573550" td=19169271088158317937607 age=3m28s
INFO [12-04|22:37:52.507] Loaded most recent local fast block      number=11388879 hash="0ec041…573550" td=19169271088158317937607 age=3m28s
INFO [12-04|22:37:52.510] Loaded last fast-sync pivot marker       number=11388325
INFO [12-04|22:38:00.533] Inspecting database                      count=10557000 elapsed=8.000s
INFO [12-04|22:38:08.533] Inspecting database                      count=21006000 elapsed=16.000s
INFO [12-04|22:38:16.533] Inspecting database                      count=31660000 elapsed=24.000s
...
INFO [12-04|22:52:30.881] Inspecting database                      count=1561224000 elapsed=14m38.348s
INFO [12-04|22:52:38.881] Inspecting database                      count=1573276000 elapsed=14m46.348s
INFO [12-04|22:52:46.881] Inspecting database                      count=1581359000 elapsed=14m54.348s
+-----------------+--------------------+------------+-----------+
|    DATABASE     |      CATEGORY      |    SIZE    |   ITEMS   |
+-----------------+--------------------+------------+-----------+
| Key-Value store | Headers            | 49.31 MiB  |     90033 |
| Key-Value store | Bodies             | 3.28 GiB   |     90033 |
| Key-Value store | Receipt lists      | 4.08 GiB   |     90033 |
| Key-Value store | Difficulties       | 5.66 MiB   |     99853 |
| Key-Value store | Block number->hash | 4.68 MiB   |     99771 |
| Key-Value store | Block hash->number | 445.31 MiB |  11388911 |
| Key-Value store | Transaction index  | 31.01 GiB  | 924916880 |
| Key-Value store | Bloombit index     | 1.80 GiB   |   5693440 |
| Key-Value store | Contract codes     | 1.59 GiB   |    333905 |
| Key-Value store | Trie nodes         | 71.08 GiB  | 647171117 |
| Key-Value store | Trie preimages     | 8.82 MiB   |    132275 |
| Key-Value store | Account snapshot   | 0.00 B     |         0 |
| Key-Value store | Storage snapshot   | 0.00 B     |         0 |
| Key-Value store | Clique snapshots   | 0.00 B     |         0 |
| Key-Value store | Singleton metadata | 151.00 B   |         5 |
| Ancient store   | Headers            | 4.83 GiB   |  11298879 |
| Ancient store   | Bodies             | 111.38 GiB |  11298879 |
| Ancient store   | Receipt lists      | 52.63 GiB  |  11298879 |
| Ancient store   | Difficulties       | 176.19 MiB |  11298879 |
| Ancient store   | Block number->hash | 409.47 MiB |  11298879 |
| Light client    | CHT trie nodes     | 0.00 B     |         0 |
| Light client    | Bloom trie nodes   | 0.00 B     |         0 |
+-----------------+--------------------+------------+-----------+
|                         TOTAL        | 282.75 GIB |           |
+-----------------+--------------------+------------+-----------+
ERROR[12-04|22:52:52.717] Database contains unaccounted data       size=128.10KiB count=2785

Neurone on 6 Dec 2020

My state entries seems to be way longer than what other people are reporting?

Currently (after a week still not synced):

Dec 10 10:14:20 vmi480454.contaboserver.net geth[422]: INFO [12-10|10:14:20.382] Imported new state entries               count=384  elapsed="145.832µs" processed=745635339 pending=115802 trieretry=0    coderetry=0 duplicate=492 unexpected=34366

|    DATABASE     |      CATEGORY      |    SIZE    |   ITEMS   |
+-----------------+--------------------+------------+-----------+
| Key-Value store | Headers            | 215.65 MiB |    394004 |
| Key-Value store | Bodies             | 14.00 GiB  |    393879 |
| Key-Value store | Receipt lists      | 17.90 GiB  |    393879 |
| Key-Value store | Difficulties       | 21.23 MiB  |    404356 |
| Key-Value store | Block number->hash | 17.08 MiB  |    404331 |
| Key-Value store | Block hash->number | 447.19 MiB |  11436819 |
| Key-Value store | Transaction index  | 31.28 GiB  | 933024549 |
| Key-Value store | Bloombit index     | 1.80 GiB   |   5697536 |
| Key-Value store | Contract codes     | 1.59 GiB   |    333673 |
| Key-Value store | Trie nodes         | 83.95 GiB  | 679328402 |
| Key-Value store | Trie preimages     | 547.13 KiB |      8893 |
| Key-Value store | Account snapshot   | 0.00 B     |         0 |
| Key-Value store | Storage snapshot   | 0.00 B     |         0 |
| Key-Value store | Clique snapshots   | 0.00 B     |         0 |
| Key-Value store | Singleton metadata | 151.00 B   |         5 |
| Ancient store   | Headers            | 4.70 GiB   |  11042816 |
| Ancient store   | Bodies             | 105.90 GiB |  11042816 |
| Ancient store   | Receipt lists      | 49.25 GiB  |  11042816 |
| Ancient store   | Difficulties       | 172.04 MiB |  11042816 |
| Ancient store   | Block number->hash | 400.19 MiB |  11042816 |
| Light client    | CHT trie nodes     | 0.00 B     |         0 |
| Light client    | Bloom trie nodes   | 0.00 B     |         0 |
+-----------------+--------------------+------------+-----------+
|                         TOTAL        | 311.62 GIB |           |
+-----------------+--------------------+------------+-----------+
ERROR[12-12|08:25:16.301] Database contains unaccounted data       size=128.19KiB count=2787

AdvancedStyle on 10 Dec 2020

Go-ethereum: What is the upper bound of "imported new state entries"?

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Geth console info

Backtrace

Most helpful comment

All 223 comments

Let's see what happens.

Testet(Ropsten)

Related issues