go-ethereum 🚀 - geth --fast stalls before crossing finish line

I have been having a similar issue recently. Ubuntu 16.04. Stalling on the last ~100-200 blocks. Restarting the geth client has allowed for some of those missing blocks to be processed but it does not keep up with the highest block. The only fluctuation I see in eth.syncing is the number of knownStates and pulledStates.

kevingentile on 20 Aug 2017

I am having the exact same issue as Laughing Cabbage has described, also on Ubuntu 16.04, and also stuck on the last few hundred blocks.
I am running geth1.6.7, at the moment. I have also tried versions 1.7.0, 1.6.6 and 1.6.5, with the same issue. I have tried applying --fast, and have tried without it.
When I restart geth, it usually gets a few more blocks in, and starts "downloading" the chain structure from 0. Downloading is in parenthesis, because when checking the folder into which it should be downloading, the folder access date and time does not change, nor can I find any other folder to which it saves the chain structure to.
Leaving it overnight, will get a chain structure in the millions, but the blocks will still not sync.
Searching the web, I have seen this problem exists for many people, for a very long time, across every platform, and with every version of geth, and no one has come up with any kind of solution. And since I am at best an amateur programmer, I have given up with geth.
I will try parity.io now, hopefully they have allowed people with little and no programming skills to connect to ethereum, and if not, then my solution is to give up on ethereum all together. That will solve the headache this issue is starting to create :-)
I'll check back with geth when in reaches version 2.

Shem-Tov on 21 Aug 2017

If any current devs think they might have a lead as to where a good starting point might be for tracking this issue I'm happy to do some bug hunting, please let me know.

kevingentile on 21 Aug 2017

@Dirksterson @LaughingCabbage I have exactly the same issues for past week and so do many of my colleagues.

After latest advice to run --fast --cache=1024 i now get the following:

WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=655c0278c317a012
WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=f26dce0aea871dc8
WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=0fb49536fda319d3
WARN [08-21|11:48:27] Stalling state sync, dropping peer       peer=ae8de9feee4df4e6
WARN [08-21|11:48:27] Stalling state sync, dropping peer       peer=e7a69c447cb83857
WARN [08-21|11:48:30] Stalling state sync, dropping peer       peer=8e8edc9627fedc6b
WARN [08-21|11:48:32] Stalling state sync, dropping peer       peer=606587b48a16fd10
WARN [08-21|11:48:32] Node data write error                    err="state node 638deb…cf0f09 failed with all peers (4 tries, 4 peers)"

tomtom87 on 21 Aug 2017

👍1

Same here. I am also on v1.6.7.

Current status, after running it for more than a week:

Downloading block 4,179,697 of 4,179,911,
Downloading chain structure 8,242,414 of 8,246,476

csillag on 21 Aug 2017

👍2

Isn't this a duplicate of #14988 and also #14995?

csillag on 21 Aug 2017

The similar issue here. On Aug, 16th I had almost fully synced blockchain, just 10-20 hours behind the current block. I then started geth as:

$ geth --syncmode=fast --cache=$(( 1024 + 512 ))

All the time geth is behind the current block. Currently (Aug, 21st) its state is:

> eth.syncing
{
  currentBlock: 4181084,
  highestBlock: 4182536,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 4179967
}

whereas etherscan.io shows 4185672 as the last block.

There are no errors in geth's output, it is in its normal state of slowly importing new segments and using HDD at speed 5-10 MB/s (both reading and writting). No high CPU usage.

INFO [08-21|14:27:00] Imported new chain segment               blocks=1 txs=60  mgas=6.645  elapsed=26.766s   mgasps=0.248  number=4181082 hash=036737…8ef0ce
INFO [08-21|14:27:16] Imported new chain segment               blocks=1 txs=77  mgas=1.748  elapsed=16.123s   mgasps=0.108  number=4181083 hash=d498b7…8c64a9
INFO [08-21|14:28:44] Imported new chain segment               blocks=1 txs=137 mgas=6.699  elapsed=1m28.060s mgasps=0.076  number=4181084 hash=b8153c…a3bcbf
INFO [08-21|14:30:44] Imported new chain segment               blocks=1 txs=62  mgas=6.691  elapsed=1m59.831s mgasps=0.056  number=4181085 hash=4e7b58…7f71d5

My geth is:

$ geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.6.7-stable/linux-amd64/go1.8
coinbase: <hidden>
at block: 4166508 (Wed, 16 Aug 2017 23:59:48 EEST)
 datadir: <hidden>
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

$ geth version
Geth
Version: 1.6.7-stable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go

darksh1ne on 21 Aug 2017

same issue here, started around the same time. Looks like this is throughout everyone and affecting parity users also now

tomtom87 on 22 Aug 2017

Hello, Ubuntu 16.04 here and same issue: got stuck on the last ~2000 blocks.

gdassori on 28 Aug 2017

If you dont got ssd u aint ever going to get them.

If u got ssd, just constantly restart the docker container and your client and pray. eventually after several days. You must be persistant.. it will crash randomly and then when you reopen it will be syncing.

Really it took me 2 weeks to do this.

On 28 Aug 2017, at 21:11, dax notifications@github.com wrote:

Hello, Ubuntu 16.04 here and same issue: got stuck on the last ~2000 blocks.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 29 Aug 2017

Same problem. Can't sync last ~100 blocks on 1.6.7. Restarting gets close but lots of Stalling state sync, dropping peer messages. SSD and fibre connection.

MrHash on 1 Sep 2017

Try using a ssd drive and docker image. This is working for me, expect atleast 4 hours to sync up. Sundays when volume is low is a good time to try

On 1 Sep 2017, at 20:01, Hasham Ahmad notifications@github.com wrote:

Same problem. Can't sync last ~100 blocks on 1.6.7. Restarting gets close but lots of Stalling state sync, dropping peer messages

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 1 Sep 2017

wtfiwtz on 3 Sep 2017

@wtfiwtz I don't really know enough about the whole process, but I would say yeah probably... for what it's worth...

alfkors on 3 Sep 2017

I was able to get it to successfully sync up, after switching from fast sync to normal sync and giving it a day or two to catch up on the last 35,000 or so blocks - using a 2012-era MacBook Pro with an SSD drive. It was necessary to be on the the latest block to be able to successfully submit a transaction with the Ethereum "Mist" wallet (or you get an error about insufficient gas).

Not sure if the light mode would make any difference, but I think you need to do it will a brand new wallet, not an existing blockchain download.

wtfiwtz on 6 Sep 2017

Ok I had to restart the sync from the beginning and have hit this problem again...

This is what I have found... Blocks are getting discarded from peers because the chain height is incorrectly set to 0.

https://github.com/wtfiwtz/go-ethereum/commit/f34b7758e52e84564693d42ee28b673ff86701f6

INFO [09-23|23:16:30] Loaded most recent local full block      number=0       hash=d4e567…cb8fa3 td=17179869184
INFO [09-23|23:16:30] Loaded most recent local fast block      number=4304570 hash=657bf3…912f25 td=1006522706491316931004

INFO [09-23|23:21:06] Peer discarded announcement              peer=d9c3012a7a0dfb3f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] ** Block number                          num=4304681
INFO [09-23|23:21:06] ** Chain height                          num=0
WARN [09-23|23:21:06] Discarded propagated block, too far away peer=d9c3012a7a0dfb3f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] Peer discarded announcement              peer=a8aafc6f4437be4f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] Peer discarded announcement              peer=ea4587bcfb02c92d number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] ** Block number                          num=4304681
INFO [09-23|23:21:06] ** Chain height                          num=0
WARN [09-23|23:21:06] Discarded propagated block, too far away peer=ea4587bcfb02c92d number=4304681 hash=7e9da0…8db154 distance=4304681

The height is retrieved from a callback function such as this:

    heighter := func() uint64 {
        return blockchain.CurrentBlock().NumberU64()
    }

So this is probably an issue with switching between fast and normal sync modes, where the chain height is assumed to be 0 when it should be equal to the fast chain height on initialization.

Is this an area you are familiar with @karalabe since you did the original fast sync implementation?

wtfiwtz on 23 Sep 2017

If the peer's total diffficulty is much lower, does that mean they are only on the full sync mode and won't work with a fast sync peer?

INFO [09-24|07:31:28] ** Total difficulty                      ours="{neg:false abs:[13700445755005557100 54]}" theirs=17179869184
INFO [09-24|07:31:28] ** fast sync?                            peer=3000f1cf9e63ce38 enabled=1

Pretty much can't find any peers that are not with a significantly lower total difficulty!

This worries me because of the following comment:

// synchronise will select the peer and use it for synchronising. If an empty string is given
// it will use the best peer possible and synchronize if it's TD is higher than our own. If any of the
// checks fail an error will be returned. This method is synchronous
func (d *Downloader) synchronise(id string, hash common.Hash, td *big.Int, mode SyncMode) error {

wtfiwtz on 23 Sep 2017

Ok I left it running this morning, and at some point, it flipped from fast to full mode when it received just 1 more chain segment or block receipts (it hadn't received any for at least 1.5 hours since I last restarted):

INFO [09-24|09:07:27] Peer discarded announcement              peer=b057fec043b525ed number=4305853 hash=37f333…4a74d9 distance=4305853
INFO [09-24|09:07:27] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:27] ** fast sync?                            peer=b057fec043b525ed enabled=1
INFO [09-24|09:07:27] ** Block number                          num=4305854
INFO [09-24|09:07:27] ** Chain height                          num=0
WARN [09-24|09:07:27] Discarded propagated block, too far away peer=b057fec043b525ed number=4305854 hash=2e8a61…ae021f distance=4305854
INFO [09-24|09:07:27] Imported new state entries               count=448  elapsed=1.479ms   processed=2608239 pending=2047  retry=2   duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] Imported new state entries               count=779  elapsed=3.995ms   processed=2609018 pending=2225  retry=22  duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:29] ** fast sync?                            peer=479032d8362da82d enabled=1
INFO [09-24|09:07:31] Imported new state entries               count=1089 elapsed=10.173ms  processed=2610107 pending=1483  retry=1   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries               count=1081 elapsed=14.713ms  processed=2611188 pending=48    retry=0   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries               count=35   elapsed=853.5µs   processed=2611223 pending=0     retry=0   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new block receipts              count=0    elapsed=3.752ms   bytes=0 number=4305451 hash=ac92d6…397f6c ignored=1
INFO [09-24|09:07:35] Committed new head block                 number=4305451 hash=ac92d6…397f6c
INFO [09-24|09:07:35] Imported new chain segment               blocks=1 txs=17 mgas=0.442 elapsed=28.174ms  mgasps=15.701 number=4305452 hash=4a61da…5f72e4
ERROR[09-24|09:07:35]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 9223372036854775807 Engine: ethash}

Number: 4305453
Hash: 0x6c4471bed33ac85f132153650f4f69230e9ef972ff33cba1e79795fb72130c66


Error: unknown ancestor
##############################

WARN [09-24|09:07:35] Synchronisation failed, dropping peer    peer=cb8ebbf8130355a7 err="retrieved hash chain is invalid"
ERROR[09-24|09:07:35] Fast sync complete, auto disabling
INFO [09-24|09:07:35] Removing p2p peer                        id=cb8ebbf8130355a7 conn=inbound duration=1h32m36.442s peers=24 req=false err="useless peer"
INFO [09-24|09:07:36] Ethereum peer connected                  id=8453dbef52518caf conn=dyndial name=Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1
INFO [09-24|09:07:36] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=1009137134152556054860
INFO [09-24|09:07:36] ** fast sync?                            peer=479032d8362da82d enabled=0
WARN [09-24|09:07:36] Ethereum handshake failed                id=8453dbef52518caf conn=dyndial err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:36] Removing p2p peer                        id=8453dbef52518caf conn=dyndial duration=279.836ms    peers=24 req=false err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:37] Peer discarded announcement              peer=ca40c7662d6ac5ed number=4305853 hash=37f333…4a74d9 distance=402
INFO [09-24|09:07:37] Peer discarded announcement              peer=ca40c7662d6ac5ed number=4305854 hash=2e8a61…ae021f distance=403
INFO [09-24|09:07:38] Ethereum peer connected                  id=6949cab8fc6d09bd conn=inbound name=Geth/v1.6.2-unstable-2a41e76b/linux-amd64/go1.8.3

The key log messages here are Committed new head block, Imported new block receipts and Imported new chain segment, which allows the full head blockchain count to update.

So I'm guessing that the network is starved of fast blocks, and they haven't yet reached their intended pivot point... before they flip to full mode.

Also note that you can't force it to use full mode on the command line, it doesn't work.

Is there some way to force this flipping from fast to full mode prematurely? Perhaps if we haven't received a new chain segment for over an hour? Or find a peer that has what we are looking for with a more broader peer search?

wtfiwtz on 24 Sep 2017

I got this peer syncin problem constantly because i was not on usa time server but using asian one.

After changing my ntp time server settings would get 20 peers connecting - previous was 1 to 3. The peers connect but still same errors you show in log.

Currently only a few of our machines wallets will finish sync, latest macs newer then 2015 find it easiest. my 2011 mac is slowest. All have ssd. all are using fibre 100mb connections.

thanks for support

On 24 Sep 2017, at 09:20, Nigel Sheridan-Smith notifications@github.com wrote:

Ok I left it running this morning, and at some point, it flipped from fast to full mode when it received just 1 more chain segment:

INFO [09-24|09:07:27] Peer discarded announcement peer=b057fec043b525ed number=4305853 hash=37f333…4a74d9 distance=4305853
INFO [09-24|09:07:27] * Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:27] * fast sync? peer=b057fec043b525ed enabled=1
INFO [09-24|09:07:27] * Block number num=4305854
INFO [09-24|09:07:27] * Chain height num=0
WARN [09-24|09:07:27] Discarded propagated block, too far away peer=b057fec043b525ed number=4305854 hash=2e8a61…ae021f distance=4305854
INFO [09-24|09:07:27] Imported new state entries count=448 elapsed=1.479ms processed=2608239 pending=2047 retry=2 duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] Imported new state entries count=779 elapsed=3.995ms processed=2609018 pending=2225 retry=22 duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] * Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:29] * fast sync? peer=479032d8362da82d enabled=1
INFO [09-24|09:07:31] Imported new state entries count=1089 elapsed=10.173ms processed=2610107 pending=1483 retry=1 duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries count=1081 elapsed=14.713ms processed=2611188 pending=48 retry=0 duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries count=35 elapsed=853.5µs processed=2611223 pending=0 retry=0 duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new block receipts count=0 elapsed=3.752ms bytes=0 number=4305451 hash=ac92d6…397f6c ignored=1
INFO [09-24|09:07:35] Committed new head block number=4305451 hash=ac92d6…397f6c
INFO [09-24|09:07:35] Imported new chain segment blocks=1 txs=17 mgas=0.442 elapsed=28.174ms mgasps=15.701 number=4305452 hash=4a61da…5f72e4
ERROR[09-24|09:07:35]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 9223372036854775807 Engine: ethash}

Number: 4305453
Hash: 0x6c4471bed33ac85f132153650f4f69230e9ef972ff33cba1e79795fb72130c66

Error: unknown ancestor

#

WARN [09-24|09:07:35] Synchronisation failed, dropping peer peer=cb8ebbf8130355a7 err="retrieved hash chain is invalid"
ERROR[09-24|09:07:35] Fast sync complete, auto disabling
INFO [09-24|09:07:35] Removing p2p peer id=cb8ebbf8130355a7 conn=inbound duration=1h32m36.442s peers=24 req=false err="useless peer"
INFO [09-24|09:07:36] Ethereum peer connected id=8453dbef52518caf conn=dyndial name=Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1
INFO [09-24|09:07:36] * Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=1009137134152556054860
INFO [09-24|09:07:36] * fast sync? peer=479032d8362da82d enabled=0
WARN [09-24|09:07:36] Ethereum handshake failed id=8453dbef52518caf conn=dyndial err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:36] Removing p2p peer id=8453dbef52518caf conn=dyndial duration=279.836ms peers=24 req=false err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:37] Peer discarded announcement peer=ca40c7662d6ac5ed number=4305853 hash=37f333…4a74d9 distance=402
INFO [09-24|09:07:37] Peer discarded announcement peer=ca40c7662d6ac5ed number=4305854 hash=2e8a61…ae021f distance=403
INFO [09-24|09:07:38] Ethereum peer connected id=6949cab8fc6d09bd conn=inbound name=Geth/v1.6.2-unstable-2a41e76b/linux-amd64/go1.8.3
The key log messages here are Committed new head block and Imported new chain segment, which allows the full head blockchain count to update.

So I'm guessing that the network is starved of fast blocks, and they haven't yet reached their intended pivot point... before they flip to full mode.

Also note that you can't force it to use full mode on the command line, it doesn't work.

Is there some way to force this flipping from fast to full mode prematurely? Perhaps if we haven't received a new chain segment for over an hour? Or find a peer that has what we are looking for with a more broader peer search?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 24 Sep 2017

You'll probably find it much easier to be on Parity (https://parity.io) - the wallet can do a light-mode sync in around 20-30 minutes... this is a good short-to-medium term solution. However, on Mac you need to be on OS X Sierra (or use the brew install instead)

I think someone needs to re-architect the fast sync in geth as the client needs to reach out to more diverse peers when it gets "stuck" for long periods of time like this. I have a few ideas, but very limited time, and it really needs to be done (or reviewed) by someone who knows what they are doing :P

wtfiwtz on 25 Sep 2017

Thanks yeh we run parity but it wont sync just the same problem so what we do is run the docker image and everytime it fucks up 'turn it off an on again' Say a little prayer an one out of ten it will work. This is only way we have found to sync to run our business we have one employee her job just come in an run sync every day at 8am before we start then can teamviewer that machine. Ethereum is a labour of love right now.. dunno how that impression affects the new comers out there. Probably aint helping adoption being so un-user friendly.

On 25 Sep 2017, at 07:34, Nigel Sheridan-Smith notifications@github.com wrote:

You'll probably find it much easier to be on Parity (https://parity.io) - the wallet can do a light-mode sync in around 20-30 minutes... this is a good short-to-medium term solution. However, on Mac you need to be on OS X Sierra (or use the brew install instead)

I think someone needs to re-architect the fast sync in geth as the client needs to reach out to more diverse peers when it gets "stuck" for long periods of time like this. I have a few ideas, but very limited time, and it really needs to be done (or reviewed) by someone who knows what they are doing :P

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 25 Sep 2017

Try restarting it a few times... you will re-connect with more diverse peers.

You might also be able to modify the config to increase MaxPeers, though I haven't figured out exactly how yet:

var DefaultConfig = Config{
    DataDir:     DefaultDataDir(),
    HTTPPort:    DefaultHTTPPort,
    HTTPModules: []string{"net", "web3"},
    WSPort:      DefaultWSPort,
    WSModules:   []string{"net", "web3"},
    P2P: p2p.Config{
        ListenAddr:      ":30303",
        DiscoveryV5Addr: ":30304",
        MaxPeers:        25,
        NAT:             nat.Any(),
    },
}

wtfiwtz on 25 Sep 2017

Looks like the configuration file is still coming, maybe later this week: https://github.com/ethereum/go-ethereum/issues/2067

wtfiwtz on 25 Sep 2017

@tomtom87 if you leave it running 24/7 it should keep sync ok... it is only if you disconnect for long periods of time (or the peers you have are not up-to-date either) that it will fall behind. Generally the client will get announcements of the latest blocks but it will automatically discard them if you are more than 32 blocks behind. Again, this is a limitation of the current implementation - it could theoretically cache them in advance of processing.

wtfiwtz on 25 Sep 2017

Oh ok is a little impracticle for all officers to run it every day constantly, as when we go home maybe cleaner tries enter computer or is turn off from sleepmode.

A cache would be great. Well if any debugging or testing needed im here.

thank you

On 25 Sep 2017, at 15:17, Nigel Sheridan-Smith notifications@github.com wrote:

@tomtom87 if you leave it running 24/7 it should keep sync ok... it is only if you disconnect for long periods of time (or the peers you have are not up-to-date either) that it will fall behind. Generally the client will get announcements of the latest blocks but it will automatically discard them if you are more than 32 blocks behind. Again, this is a limitation of the current implementation - it could theoretically cache them in advance of processing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 25 Sep 2017

So, is our Ether lost forever, or is there a way to get it back using another application?

MartinDace on 27 Sep 2017

@MartinDace The state of your ETH accounts are kept on the blockchain and kept current irrespective of which client you use. The only thing that you need is the private key associated with your account. You can view your balance at etherscan.io if you'd like to ease your fears.

schroedingerscode on 27 Sep 2017

But is it possible move ether from there?

MartinDace on 28 Sep 2017

Hi

I am having the same issue with the blockchain syncing . but now I cannot get a eth.syncing but I do get a eth.blockNumber and peerCount reply now -- is this normal and am I synched yet since I do not want to stop Geth now since I have a blockNumber reply for first time so far in this week long process:)
/ $ geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.7.0-stable-6c6c7b2a/linux-amd64/go1.9
coinbase: 0x9ca8c0cf47576208c5b017d9a560ed33d84f2267
at block: 4319007 (Thu, 28 Sep 2017 13:21:59 IST)
datadir: /home/qb78846/.ethereum
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

eth.syncing
false
net.peerCount
9
eth.blockNumber
4319073

[09-28|13:55:43] Imported new chain segment blocks=1 txs=125 mgas=6.531 elapsed=155.054ms mgasps=42.121 number=4319064 hash=d1e5f5…469d33
INFO [09-28|13:56:56] Imported new chain segment blocks=1 txs=124 mgas=6.559 elapsed=225.838ms mgasps=29.042 number=4319065 hash=396dcb…767c1e
INFO [09-28|13:57:37] Imported new chain segment blocks=1 txs=172 mgas=6.683 elapsed=280.647ms mgasps=23.812 number=4319066 hash=036ac3…18debf
INFO [09-28|13:57:44] Imported new chain segment blocks=1 txs=101 mgas=6.616 elapsed=220.905ms mgasps=29.949 number=4319067 hash=ea8c76…884246
INFO [09-28|13:58:01] Imported new chain segment blocks=1 txs=95 mgas=6.687 elapsed=236.463ms mgasps=28.278 number=4319068 hash=5315a5…42028a
INFO [09-28|13:58:02] Imported new chain segment blocks=1 txs=136 mgas=6.698 elapsed=251.738ms mgasps=26.607 number=4319067 hash=e650a5…0e36cd
INFO [09-28|13:58:23] Imported new chain segment blocks=1 txs=204 mgas=6.544 elapsed=323.078ms mgasps=20.256 number=4319069 hash=56271a…cbf0b2
INFO [09-28|13:58:24] Imported new chain segment blocks=1 txs=76 mgas=6.600 elapsed=350.838ms mgasps=18.811 number=4319070 hash=526d33…30fb23
INFO [09-28|13:58:46] Imported new chain segment blocks=1 txs=33 mgas=6.569 elapsed=713.876ms mgasps=9.202 number=4319071 hash=a51bf0…172919
INFO [09-28|13:59:04] Imported new chain segment blocks=1 txs=83 mgas=6.661 elapsed=310.312ms mgasps=21.464 number=4319072 hash=09ff09…30f2bd
INFO [09-28|13:59:16] Imported new chain segment blocks=1 txs=109 mgas=2.409 elapsed=177.983ms mgasps=13.538 number=4319073 hash=6b7e6a…01553f
INFO [09-28|14:00:13] Imported new chain segment blocks=1 txs=104 mgas=6.675 elapsed=180.008ms mgasps=37.082 number=4319074 hash=391698…bbda07
INFO [09-28|14:01:22] Imported new chain segment blocks=1 txs=187 mgas=6.693 elapsed=1.596s mgasps=4.192 number=4319075 hash=3b2aa4…bc3e8b

abl1234 on 28 Sep 2017

I think I am good since I read in another post that the importing chain segment happens approx every 14 seconds and if i check the last block on etherscan.io it is similiar - can anybody confirm this is expceted behaviour ? thnaks in advance

INFO [09-28|14:14:35] Imported new chain segment blocks=1 txs=132 mgas=6.722 elapsed=529.342ms mgasps=12.699 number=4319106 hash=15e59b…55f217
INFO [09-28|14:14:55] Imported new chain segment blocks=1 txs=161 mgas=6.699 elapsed=346.993ms mgasps=19.306 number=4319107 hash=716ce9…9e0a26

abl1234 on 28 Sep 2017

@MartinDace you can add your wallet details to any wallet application you like (by importing it) - as long as you have retained the wallet JSON file and password, or the private key. You can then add new transactions to the Ethereum blockchain from that wallet. The blockchain is a public register of all transactions so if you re-download it (by "syncing") then your wallet will update the the current transaction value that has been agreed by all of the Ethereum peers.

You just need to be patient with geth and Ethereum Mist as the syncing is very slow, even in fast mode. You might be able to try light mode instead.

wtfiwtz on 29 Sep 2017

@abl1234 yes you are close, the chain segments you will need are 4,320,572 at the present moment (I think), so about 1400 to go!

wtfiwtz on 29 Sep 2017

Also https://github.com/ethereum/mist/issues/3097 🎉

wtfiwtz on 29 Sep 2017

I've encountered this problem twice on 1.7:ish.

First occasion, it flipped to from fast to full, but somehow failed to commit the flip, and then kept on fast-synching past pivot point, IIUC.

Second occasion, stuck on importing state entries, no headers/blocks imported any longer. Restarting geth seems to make the block/header import resume.

holiman on 30 Sep 2017

turn it off and on again

Works every time

On 30 Sep 2017, at 15:20, Martin Holst Swende notifications@github.com wrote:

I've encountered this problem twice on 1.7:ish.
First occasion, it flipped to from fast to full, but somehow failed to commit the flip, and then kept on fast-synching past pivot point, IIUC.
Second occasion, stuck on importing state entries, no headers/blocks imported any longer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 30 Sep 2017

@tomtom87 I've tried that a few times without success. Since I bought ether direct from the official ethereum web site and am using the default wallet they offer, does this not imply that the team behind ethereum are not on top of their game? And if so, why should we buy ether at all?

MartinDace on 2 Oct 2017

@MartinDace the issue is the peer-to-peer network is not 100% reliable and that is not the fault of the Ethereum team. If someone has the time and inclination, it would be possible to make the syncing process more efficient by enhancing the geth peer-to-peer client (as described above).

Alternatively, try the geth light mode in 0.7.1 or grab the http://parity.io wallet which is a separate implementation in the language Rust and you might have more luck with that (often syncs within a few hours or less).

wtfiwtz on 2 Oct 2017

🎉1

Hi Nigel
Thanks for the help - I still seem to be still stuck at the importing new chain segments - this copy was taken when the latest block on etherscan.io is the same as the last imported peers

NFO [10-02|11:35:50] Imported new chain segment blocks=6 txs=616 mgas=36.242 elapsed=9.698s mgasps=3.737 number=4328631 hash=56d1b3…86cc70
INFO [10-02|11:35:59] Imported new chain segment blocks=5 txs=512 mgas=29.294 elapsed=8.793s mgasps=3.331 number=4328636 hash=0f1e2b…af67bc
INFO [10-02|11:36:08] Imported new chain segment blocks=6 txs=791 mgas=32.215 elapsed=8.798s mgasps=3.661 number=4328642 hash=33c3e4…006d36
INFO [10-02|11:36:17] Imported new chain segment blocks=5 txs=553 mgas=32.785 elapsed=8.691s mgasps=3.772 number=4328647 hash=abc579…7379fe
INFO [10-02|11:36:25] Imported new chain segment blocks=6 txs=507 mgas=34.194 elapsed=8.807s mgasps=3.882 number=4328653 hash=b1d45a…16485b
INFO [10-02|11:36:35] Imported new chain segment blocks=6 txs=703 mgas=27.595 elapsed=9.226s mgasps=2.991 number=4328659 hash=188adc…88f4bb

Last Block
4330217 (29.91s Avg)
Transactions
63451085
Hash Rate
Network Difficulty
98,899.45 GH/s 2,863.92 TH
14 day Ethereum Transaction History9/189/199/209/219/229/239/249/259/269/279/289/299/3010/1250k300k350k400k

abl1234 on 2 Oct 2017

I passed that number on the imported new chain segment

Last Block
4330217 (29.91s Avg)
Transactions
63451085
Hash Rate
Network Difficulty
98,899.45 GH/s 2,863.92 TH
14 day Ethereum Transaction History9/189/199/209/219/229/239/249/259/269/279/289/299/3010/1250k300k350k400k

abl1234 on 2 Oct 2017

INFO [10-02|11:45:41] Imported new chain segment INFO [10-02|11:45:49] Imported new chain segment INFO [10-02|11:45:58] Imported new chain segment INFO [10-02|11:46:08] Imported new chain segment INFO [10-02|11:46:17] Imported new chain segment INFO [10-02|11:46:27] Imported new chain segment INFO [10-02|11:46:36] Imported new chain segment INFO [10-02|11:46:44] Imported new chain segment INFO [10-02|11:46:53] Imported new chain segment INFO [10-02|11:47:02] Imported new chain segment INFO [10-02|11:47:12] Imported new chain segment INFO [10-02|11:47:24] Imported new chain segment INFO [10-02|11:47:34] Imported new chain segment INFO [10-02|11:47:42] Imported new chain segment INFO [10-02|11:47:51] Imported new chain segment INFO [10-02|11:47:59] Imported new chain segment INFO [10-02|11:48:07] Imported new chain segment INFO [10-02|11:48:15] Imported new chain segment INFO [10-02|11:48:24] Imported new chain segment INFO [10-02|11:48:32] Imported new chain segment INFO [10-02|11:48:41] Imported new chain segment INFO [10-02|11:48:51] Imported new chain segment blocks=6 txs=677 mgas=37.087 elapsed=8.639s mgasps=4.293 number=4329017 hash=b3c0eb…f352f1
blocks=10 txs=703 mgas=44.328 elapsed=8.426s mgasps=5.260 number=4329027 hash=77464a…04834e
blocks=11 txs=904 mgas=64.091 elapsed=8.141s mgasps=7.872 number=4329038 hash=fb3b56…4f4768
blocks=4 txs=444 mgas=22.466 elapsed=10.129s mgasps=2.218 number=4329042 hash=95ed8e…7494ee
blocks=3 txs=323 mgas=14.226 elapsed=9.256s mgasps=1.537 number=4329045 hash=082483…ac5e9a
blocks=4 txs=433 mgas=20.678 elapsed=9.810s mgasps=2.108 number=4329049 hash=ec335e…7db925
blocks=3 txs=158 mgas=13.802 elapsed=9.064s mgasps=1.523 number=4329052 hash=2189dc…c52d5d
blocks=5 txs=486 mgas=33.253 elapsed=8.532s mgasps=3.897 number=4329057 hash=fa1a66…37dc10
blocks=9 txs=613 mgas=40.955 elapsed=8.782s mgasps=4.663 number=4329066 hash=df4112…e2e078
blocks=6 txs=471 mgas=32.958 elapsed=8.375s mgasps=3.935 number=4329072 hash=3b9bb3…2e4d90
blocks=4 txs=281 mgas=21.135 elapsed=10.903s mgasps=1.938 number=4329076 hash=b765a8…d3eae7
blocks=10 txs=836 mgas=45.666 elapsed=11.480s mgasps=3.978 number=4329086 hash=4e4727…ce0d43
blocks=18 txs=1420 mgas=92.512 elapsed=9.831s mgasps=9.410 number=4329104 hash=63cb04…e15978
blocks=5 txs=586 mgas=28.044 elapsed=8.113s mgasps=3.456 number=4329109 hash=5ba422…d2e834
blocks=4 txs=578 mgas=20.966 elapsed=8.636s mgasps=2.428 number=4329113 hash=03f4c8…29fedd
blocks=3 txs=537 mgas=14.794 elapsed=8.123s mgasps=1.821 number=4329116 hash=a41030…1c6dc9
blocks=4 txs=450 mgas=26.650 elapsed=8.236s mgasps=3.236 number=4329120 hash=4aecfb…cf6698
blocks=5 txs=552 mgas=24.531 elapsed=8.535s mgasps=2.874 number=4329125 hash=5c29a2…88954b
blocks=8 txs=588 mgas=18.906 elapsed=8.419s mgasps=2.246 number=4329133 hash=4938a0…1a9f6f
blocks=5 txs=396 mgas=22.023 elapsed=8.492s mgasps=2.593 number=4329138 hash=e38b10…2d1e93
blocks=8 txs=856 mgas=47.102 elapsed=8.771s mgasps=5.370 number=4329146 hash=9a7bef…a8f552
blocks=7 txs=670 mgas=32.824 elapsed=9.705s mgasps=3.382 number=4329153 hash=4c40b0…e92b24

abl1234 on 2 Oct 2017

so I am upto date with the blocks but not getting the sycnhronization completed

abl1234 on 2 Oct 2017

@wtfiwtz Hello Nigel,
I downloaded the wallet at http://parity.io however I've idea how to proceed from there. I have a wallet address for the Mist wallet but I have been sent no private key or list of words so I do not see how I can access the ether than I have purchased. My balance is still registered at zero in the wallet app although I have verified the existence of my balance by using the wallet address at etherscan.io

MartinDace on 2 Oct 2017

In other words it looks to me like ethereum is a scam, since they invite you to buy ether and provide a default wallet from which it is impossible to extract the ether. "...it would be possible to make the syncing process more efficient by..." - If so, why don't they do it? Seems pretty basic to me.

MartinDace on 2 Oct 2017

👍1

Ethereum.org does not "invite you to buy" ether. This ticket is about synching the ethereum blockchain. It's fully possible to 'extract ether' without synching the blockchain, all that is needed is to construct a transaction. That can be done in multiple ways, e.g. via the geth console.

var tx = { to : <receiver> , from:"0x...account...", value: web3.toWei( number_of_ether, "ether")}
var rlpdata = eth.signTransaction(tx)

And the actual transaction can be broadcast using some other node, e.g myetherwallet. There's no need to hijack this particular ticket @MartinDace . There is a lot of information out there if you're willing to dig a bit, otherwise please use the Ethereum gitter rooms to ask for help.

As for the ticket, yes, there's some glitch in switching over from fast-sync to full sync.

holiman on 2 Oct 2017

@holiman It may not be an intentional scam, and perhaps scam is the wrong word, but the web site allows you to download the ethereum wallet and allows you to purchase ether without any warning that what you purchase will never be available to use (as far as I can tell). Given the length of this thread (that I was redirected to from a thread that began with someone else making exactly that complaint) I can hardly be accused of hijacking it. It is fine to explain after the event that I should have done it some other way (incidentally requiring coding skill which I do not have) but no-one so far has provided an answer to the original question, which is, how do we get our ether, once having started this process? If there is a way, please explain.

MartinDace on 2 Oct 2017

👎1 👍1

allows you to purchase ether

What? There is no way of buying 'ether direct from the official ethereum web site ', unless you mean the pre-sale a couple of years ago.

Anyway, the simplest way to create a transaction without any coding skills or command line experience, is probably to use myetherwallet and import the wallet-file there. A note of warning, though: there are phishing-sites which masquerade as myetherwallet, and exposing a wallet to a website is dangerous without proper precautions.

The recommended way is to download myetherwallet and use in an offline computer, create a transaction, take the 'rlp'-data over to an online computer and broadcast the transaction (e.g. using myetherwallet again).

Also : "I have been sent no private key" - and never will you be sent a private key. if you have generated a mist wallet, and sent funds to it, you need to ensure the security of the wallet, which resides in a keystore folder (the location of which is platform-dependant).

Ultimately, remember that _you_ are responsible for your ether - nobody else. All information about interacting with wallet(s) is available in various places - the ethereum wiki, or parity documentation if you're using parity, or our gitter channels.

holiman on 2 Oct 2017

@holiman Thank you for your explanation. I understand the ether is not from the ethereum web site itself. However when buying Bitcoin using (for example) bread all the information is there - the public key, the private key, the rescue words. I have used it without problems including purchasing my so far missing ether. Mist, however, is entirely obscure in its functioning, as well as appearing to be in an endless loop since 11 September. Regrettably my attempt to use myetherwallet stalled at the end and I see no way to use that either.

MartinDace on 2 Oct 2017

Hi

Getting back to the original thread about not syncing - I READ earlier that to get it to sync once needs to RUN Geth FULL sync from the beginning without any stops and it will work - The issue being that from a FULL to FAST sync is causing the problem - I do not want to run this process again since has taken a week to get his far so is there a new version of GETH coming out soon or a patch or should i try a development build 1.71?
Thanks in advance
Using this build
instance: Geth/v1.7.0-stable-6c6c7b2a/linux-amd64/go1.9
coinbase: 0x9ca8c0cf47576208c5b017d9a560ed33d84f2267
at block: 4333102 (Tue, 03 Oct 2017 12:15:38 IST)
datadir: /home/qb78846/.ethereum
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

abl1234 on 3 Oct 2017

???

abl1234 on 4 Oct 2017

Ethereum is a free research software its not finished, its a work in progress. It's evolving and has bugs

I can rescue your ether, log in to myetherwallet with your mist exported json file and it will be there, then send it to exodus wallet and you are good to go.

Mist wallet is only for developers of smart contracts and tokens for retail users try exodus that is equivilent of bread. This needs to be clearly stated to customers really because mist wallet is not for newbs.

If you still got problems with gettin you ETH contact me i will help you recover it.

On 3 Oct 2017, at 04:34, MartinDace notifications@github.com wrote:

@holiman Thank you for your explanation. I understand the ether is not from the ethereum web site itself. However when buying Bitcoin using (for example) bread all the information is there - the public key, the private key, the rescue words. I have used it without problems including purchasing my so far missing ether. Mist, however, is entirely obscure in its functioning, as well as appearing to be in an endless loop since 11 September. Regrettably my attempt to use myetherwallet stalled at the end and I see no way to use that either.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 5 Oct 2017

@tomtom87 thank you. I have found the ethereum wallet files in Libraries - Application support. Where would the json file be?

MartinDace on 5 Oct 2017

General advisory: do _not_ share the wallet file with anyone you don't trust _fully_. The json wallet file contains your encrypted private key. Anyone in control of the private key has full access to all funds on the account.

holiman on 5 Oct 2017

👍5

@MartinDace yes don't send anything like @holiman said. Ok just get the file in the keystore directory and you can use that to unlock on myetherwallet i do it all the time when it wont sync.

tomtom87 on 5 Oct 2017

@tomtom87 thanks for your advice but I don't know where to start. There are three files on my computer with the suffix .json and none of them have the private key. Where is the keystore directory? I cannot find any file or folder with that name. Likewise I don't know how to go about creating a 'Mist exported json file.' I do not see any export facility in the Mist menu. This may all seem obvious to you but I have no idea where to begin. Your further advice would be much appreciated.

MartinDace on 5 Oct 2017

@MartinDace this ticket is not the right forum. Here's some info about where the keystore is located https://ethereum.stackexchange.com/a/109 .

holiman on 5 Oct 2017

Hi mate just go to /Library/Ethereum/keystore

its the file name start UTC does not have json file extension. Open in myetherwallet as json format enter your password to unlock.

If still confused in Mist app just click the backup menu option it will launch finder in the keystore directory. Load the file with filename starting UTC an a bunch of numbers. This is your wallet json. Now in myetherwallet transfer to your exodus and spit on mist never use it again its not production ready.

Peace

On 6 Oct 2017, at 02:21, MartinDace notifications@github.com wrote:

@tomtom87 thanks for your advice but I don't know where to start. There are three files on my computer with the suffix .json and none of them have the private key. Where is the keystore directory? I cannot find any file or folder with that name. Likewise I don't know how to go about creating a 'Mist exported json file.' I do not see any export facility in the Mist menu. This may all seem obvious to you but I have no idea where to begin. Your further advice would be much appreciated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 6 Oct 2017

@tomtom87 Hello Tom, thanks, I finally found it. It's not in Application support/ ethereum where I was looking for it before, but in User/ [name] /Library... as you say. Not easy as the Library was hidden by the Mac OS. I think I have finally rescued my ether, which while all this was going on appears to have increased in value by 50%. There should be a warning on the web site that the wallet is not for regular folk. Thanks also @holiman Martin for a helpful attitude. Over and out!

MartinDace on 10 Oct 2017

I'm thrilled for you

On 10 Oct 2017 8:58 pm, "MartinDace" notifications@github.com wrote:

@tomtom87 https://github.com/tomtom87 Hello Tom, thanks, I finally
found it. It's not in Application support/ ethereum where I was looking for
it before, but in User/ [name] /Library... as you say. Not easy as the
Library was hidden by the Mac OS. I think I have finally rescued my ether,
which while all this was going on appears to have increased in value by
50%. There should be a warning on the web site that the wallet is not for
regular folk. Thanks also @holiman https://github.com/holiman Martin
for a helpful attitude. Over and out!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-335589580,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ae3NWZ7LabS1Mj3DcyK4eU23BBa9MZokks5sq8yAgaJpZM4O7ctW
.

abl1234 on 11 Oct 2017

same problem

ilia2s on 23 Oct 2017

@ilia2s follow the advice above. If you have a mac I can tell you how to find the files.

MartinDace on 30 Oct 2017

thank you, the update helps

ilia2s on 30 Oct 2017

I have the same problem but with the Windows Version.
The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txp
ool:1.0 web3:1.0

eth.syncing
{
currentBlock: 4564714,
highestBlock: 4564788,
knownStates: 1223685,
pulledStates: 1195180,
startingBlock: 4564561
}

can you help me in this case?

MobiD2017 on 16 Nov 2017

It is downloading the state trie. There are over 30M state entries to pull
before it can finalize the last blocks.

On Nov 16, 2017 20:10, "MobiD2017" notifications@github.com wrote:

I have the same problem but with the Windows Version.
The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0
rpc:1.0 txp
ool:1.0 web3:1.0

eth.syncing
{
currentBlock: 4564714,
highestBlock: 4564788,
knownStates: 1223685,
pulledStates: 1195180,
startingBlock: 4564561
}

can you help me in this case?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345009205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6GUHDTzj5HPohPZlmuN3Cn6cV0bHfks5s3HqEgaJpZM4O7ctW
.

karalabe on 16 Nov 2017

😄1

use ssd or it never sync

On 17 Nov 2017, at 01:14, Péter Szilágyi notifications@github.com wrote:

It is downloading the state trie. There are over 30M state entries to pull
before it can finalize the last blocks.

On Nov 16, 2017 20:10, "MobiD2017" notifications@github.com wrote:

I have the same problem but with the Windows Version.
The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0
rpc:1.0 txp
ool:1.0 web3:1.0

eth.syncing
{
currentBlock: 4564714,
highestBlock: 4564788,
knownStates: 1223685,
pulledStates: 1195180,
startingBlock: 4564561
}

can you help me in this case?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345009205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6GUHDTzj5HPohPZlmuN3Cn6cV0bHfks5s3HqEgaJpZM4O7ctW
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 17 Nov 2017

@tomtom87

The system is installed and running on SSD.
This night the System was automatically rebooted.
If I restart the process with "geth –rpc" new chain are imported:
INFO [11-17|06:40:03] Imported new chain segment blocks=1 txs=128 mgas=6.107 elapsed=240.612ms mgasps=25.379 number=4567898 hash=1f699c.19ef7b
INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=27 mgas=0.842 elapsed=54.201ms mgasps=15.539 number=4567899 hash=bceae4.2b190d
INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=86 mgas=6.685 elapsed=157.207ms mgasps=42.524 number=4567899 hash=11a9a7.561e6e
INFO [11-17|06:40:10] Imported new chain segment blocks=1 txs=80 mgas=3.048 elapsed=151.008ms mgasps=20.183 number=4567900 hash=35129d.95609f

but the "geth attach eth.syncing" is showing:

eth.syncing
{
currentBlock: 4567699,
highestBlock: 4567709,
knownStates: 0,
pulledStates: 0,
startingBlock: 4567694
}
eth.syncing
false
eth.syncing
false

MobiD2017 on 17 Nov 2017

Ok, I have had this problem before and solved by using docker image (and
uncountable restarts)

On 17/11/2017 12:47, MobiD2017 wrote:
>

@tomtom87 https://github.com/tomtom87

The system is installed and running on SSD.
This night the System was automatically rebooted.
If I restart the process with "geth –rpc" new chain are imported:
INFO [11-17|06:40:03] Imported new chain segment blocks=1 txs=128
mgas=6.107 elapsed=240.612ms mgasps=25.379 number=4567898
hash=1f699c.19ef7b
INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=27
mgas=0.842 elapsed=54.201ms mgasps=15.539 number=4567899
hash=bceae4.2b190d
INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=86
mgas=6.685 elapsed=157.207ms mgasps=42.524 number=4567899
hash=11a9a7.561e6e
INFO [11-17|06:40:10] Imported new chain segment blocks=1 txs=80
mgas=3.048 elapsed=151.008ms mgasps=20.183 number=4567900
hash=35129d.95609f

but the "geth attach eth.syncing" is showing:
eth.syncing
{
currentBlock: 4567699,
highestBlock: 4567709,
knownStates: 0,
pulledStates: 0,
startingBlock: 4567694
}
eth.syncing
false
eth.syncing
false
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345151503,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB7p9_o_mOyz0ejdD3hz3UZ0s7pRi4qVks5s3R3wgaJpZM4O7ctW.

tomtom87 on 17 Nov 2017

Seems you are indeed in sync. Odd that the eth.syncing endpoint still tells you otherwise. I'll need to take a look if that's indeed the case. Could you double check please?

karalabe on 17 Nov 2017

I have restarted the sync after a removedb, the result is the same.

the data for:
knownStates: 0,
pulledStates: 0,

will not be loaded.

I have restarted the sync process with geth --fast --cache=1024, the result is:
INFO [11-19|21:09:40] Imported new chain segment blocks=4 txs=522 mgas=23.742 elapsed=2.536s mgasps=9.361 number=4584075 hash=8679be.584ea3
INFO [11-19|21:09:40] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=7.000ms mgasps=0.000 number=4584076 hash=03e044.93ba60

eth.syncing shows me:
currentBlock: 4584043,
highestBlock: 4584048,
knownStates: 0,
pulledStates: 0,
startingBlock: 4584043

can you help me in this case?
For me its looks like im sync without the last blocks but it doesn't work.

MobiD2017 on 19 Nov 2017

i synced a fresh node last week it took approx 2 days just under 48 hours. first check you have ssd and can connect to the ethereum network, if its still not working and you are on mac osx try the docker image. Works for me after several soft restarts (not deleting chain data just starting docker again)

On 20 Nov 2017, at 03:13, MobiD2017 notifications@github.com wrote:

I have restarted the sync after a removedb, the result is the same.

the data for:
knownStates: 0,
pulledStates: 0,

will not be loaded.

I have restarted the sync process with geth --fast --cache=1024, the result is:
INFO [11-19|21:09:40] Imported new chain segment blocks=4 txs=522 mgas=23.742 elapsed=2.536s mgasps=9.361 number=4584075 hash=8679be.584ea3
INFO [11-19|21:09:40] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=7.000ms mgasps=0.000 number=4584076 hash=03e044.93ba60

eth.syncing shows me:
currentBlock: 4584043,
highestBlock: 4584048,
knownStates: 0,
pulledStates: 0,
startingBlock: 4584043

can you help me in this case?
For me its looks like im sync without the last blocks but it doesn't work.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 20 Nov 2017

yesterday evening after my posting I have removed the db and start a fresh sync.
All Data will be stored on SSD and I have a windows system.

Started with the following command: geth --fast --cache=2048

eth.syncing show me:

{
currentBlock: 1105238,
highestBlock: 4584613,
knownStates: 1908357,
pulledStates: 1895114,
startingBlock: 0
}

This morning the sync block is higher then 4584613.INFO
[11-20|05:59:05] Imported new chain segment blocks=1 txs=34 mgas=6.660 elapsed=145.607ms mgasps=45.738 number=4586397 hash=2428ac.0ca952
INFO [11-20|05:59:16] Imported new chain segment blocks=1 txs=36 mgas=6.420 elapsed=119.606ms mgasps=53.680 number=4586398 hash=e49bcf.21b785
INFO [11-20|05:59:25] Imported new chain segment blocks=1 txs=43 mgas=1.941 elapsed=80.004ms mgasps=24.263 number=4586399 hash=c4c262.45254b

eth.syncing show me on "false".
If I restart the sync process nothing will change.
Only I get the following Warning message:
WARN [11-20|05:53:40] Blockchain not empty, fast sync disabled

MobiD2017 on 20 Nov 2017

Oh its because you using windows bruh

Just playing, but really is probably somethingto do with it. Try using docker for the node. On linux or mac its take 48 hours currently from fresh install i done two on thursday last week.

Basically its syncing just at a gnats pace.

On 20 Nov 2017, at 12:03, MobiD2017 notifications@github.com wrote:

yesterday evening after my posting I have removed the db and start a fresh sync.
All Data will be stored on SSD and I have a windows system.

Started with the following command: geth --fast --cache=2048

eth.syncing show me:

{
currentBlock: 1105238,
highestBlock: 4584613,
knownStates: 1908357,
pulledStates: 1895114,
startingBlock: 0
}

This morning the sync block is higher then 4584613.INFO
[11-20|05:59:05] Imported new chain segment blocks=1 txs=34 mgas=6.660 elapsed=145.607ms mgasps=45.738 number=4586397 hash=2428ac.0ca952
INFO [11-20|05:59:16] Imported new chain segment blocks=1 txs=36 mgas=6.420 elapsed=119.606ms mgasps=53.680 number=4586398 hash=e49bcf.21b785
INFO [11-20|05:59:25] Imported new chain segment blocks=1 txs=43 mgas=1.941 elapsed=80.004ms mgasps=24.263 number=4586399 hash=c4c262.45254b

eth.syncing show me on "false".
If I restart the sync process nothing will change.
Only I get the following Warning message:
WARN [11-20|05:53:40] Blockchain not empty, fast sync disabled

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 20 Nov 2017

ok, the final point is: this project is sucks, developers are disabled people. don't posting, don't fixing. maybe this project made for you not to mining as long time as it possible. trying to find better solution. thank you for killed my time

TillDev on 29 Nov 2017

👎2 👍2

@MobiD2017 You are (were) in sync at block 4586399. What makes you think there's a problem?

holiman on 29 Nov 2017

Been stuck on "INFO [11-30|20:21:22] Imported new state entries" for a few days already.

This project should be abandoned imho; This is bad for the Ethereum project, since they call it "Official wallet"...

bibz0r on 30 Nov 2017

same problem.. what a waste of time.. Doesnt sync last 150-200 blocks even on an SSD and very fast network.. keeps importing new state entries or dropping peers/stalling.. I Tried restarting, switching between fast and normal mode, and looking for solutions online. Nothing works, this sucks.
also knownStates and pulledStates keep increasing endlessly..

ugotjelly on 6 Dec 2017

So,

Please provide some snippets of logs. It's difficult to say anything based on a report saying just "me to this sucks". It does not help us solve anything. We need to figure out if people are having problems finding peers, or problems downloading data from peers, or if the problem is slow block processing, or if the problem is disk IO. Logs help us do that.

At least one person in this thread was apparently already in sync when reporting the bug.

holiman on 6 Dec 2017

@holiman i restarted it a few times and now its not stalling anymore, now its importing new chain segments and i think its done. Maybe this is how its supposed to go, but It doesn't make sense that I sync'd like 4,000,000 blocks (99% done) in like 2 hours but to sync the last <200 blocks its taking more than 5 hours.. Why is that a thing, i'm pretty sure it would take days or weeks on an normal HDD (because of the slow writing speed, what i was trying to do before) which is absurd.
edit: but now it finally works just gonna be patient i guess. Also i'm almost out of space because the folder is 39 GB, was expecting it to be around 20gb

ugotjelly on 6 Dec 2017

It never finishes on a mechanical hdd and barely on a ssd now

Only gonna get worse as ethereum gets more popular

On 6 Dec 2017, at 15:47, ugotjelly notifications@github.com wrote:

@holiman i restarted it a few times and now its not stalling anymore, now its importing new chain segments and i think almost done. Maybe this is how its supposed to go, but It doesn't make sense that I sync'd like 4,000,000 blocks (99% done) in like 2 hours but to sync the last <200 blocks its taking more than 5 hours.. Why is that a thing, i'm pretty sure it would take days or weeks on an normal HDD (what i was trying to do before) which is absurd

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 6 Dec 2017

There are logs??...

On 06/12/2017 15:15, Martin Holst Swende wrote:
>

So,

Please provide some snippets of logs. It's difficult to say
anything based on a report saying just "me to this sucks". It does
not help us solve anything. We need to figure out if people are
having problems finding peers, or problems downloading data from
peers, or if the problem is slow block processing, or if the
problem is disk IO. Logs help us do that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349566475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB7p99wvSVS3VE0QhGB7W-uwMdXoX37oks5s9k0mgaJpZM4O7ctW.

tomtom87 on 6 Dec 2017

@holiman : After reading this and other threads it seems to be clear there are solid hardware and connectivity requirements to sync the ETH blockchain. E.g. physical hard drives vs. SSD. I gave up on my attempts to try syncing with a new off the shelve MacMini equipped with just a physical drive shortly after starting this thread. It would be great to have a recommendation from the Ethereum team on minimum hardware requirements today plus a recommendation for required hardware for the near and mid term future. Logs of the slower machine all the way up in this thread.

I upgraded to a machine with a SSD and a 3.5Ghz i7 on a 50Mbit DSL line. That worked just fine a few months back overnight. However, last nights sync also didn't complete with a different error message now. "Bad Block". Here is a snippet:

```
WARN [12-06|13:47:57] Synchronisation failed, dropping peer peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:36]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:48:36] Synchronisation failed, dropping peer peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:45]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:48:45] Synchronisation failed, dropping peer peer=1c92bbdd230e8683 err="retrieved hash chain is invalid"
ERROR[12-06|13:50:06]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:50:06] Synchronisation failed, dropping peer peer=6eb61fa67b147e73 err="retrieved hash chain is invalid"

```

Dirksterson on 6 Dec 2017

@Dirksterson the block 4370000 is the fork-block for Byzantium, and the block 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e is indeed the canonical block.

The problem is that you're using some old version of geth, which is not configured for Byzantium: Metropolis: 9223372036854775807

A common source of confusion is when someone installs geth, via e..g apt install geth or go install geth, and then later on builds from source, but keeps using the old binary which resides somewhere in the path.

holiman on 6 Dec 2017

@tomtom87 yes, there are logs -- in case you're starting geth via mist, there are some node.log or something similar which contains info about what's happened.

@ugotjelly it shouldn't go that slow, the first 4M+ blocks are fast-sync to the 'pivot point', after that is reached, every block is processed individually. Still, without logs, it's hard to say if it's due to block-starvation or simply that your machine can't process the blocks quickly enough.

holiman on 6 Dec 2017

I get the dropping peer hash chain is invalid error for months now, you
just have to start over. Really not sure what is causing it. Start the
docker again and try syncing. You do not need to delete chain data.

On 06/12/2017 19:55, Dirksterson wrote:
>

@holiman https://github.com/holiman : After reading this and other
threads it seems to be clear there are solid hardware and connectivity
requirements to sync the ETH blockchain. E.g. physical hard drives vs.
SSD. I gave up on my attempts to try syncing with a new off the shelve
MacMini equipped with just a physical drive shortly after starting
this thread. It would be great to have a recommendation from the
Ethereum team on minimum hardware requirements today plus a
recommendation for required hardware for the near and mid term future.
Logs of the slower machine all the way up in this thread.

I upgraded to a machine with a SSD and a 3.5Ghz i7 on a 50Mbit DSL
line. That worked just fine a few months back overnight. However, last
nights sync also didn't complete with a different error message now.
"Bad Block". Here is a snippet:

WARN [12-06|13:47:57] Synchronisation failed, dropping peer
peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:36]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport:
true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis:
9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:48:36] Synchronisation failed, dropping peer
peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:45]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport:
true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis:
9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:48:45] Synchronisation failed, dropping peer
peer=1c92bbdd230e8683 err="retrieved hash chain is invalid"
ERROR[12-06|13:50:06]

#### BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport:
true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis:
9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050

#

WARN [12-06|13:50:06] Synchronisation failed, dropping peer
peer=6eb61fa67b147e73 err="retrieved hash chain is invalid"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349631420,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB7p97qzzPsWWBkqMJzxls0Z5adwFssYks5s9o7WgaJpZM4O7ctW.

tomtom87 on 6 Dec 2017

@holiman Thank you bunches for pointing that out!!! Greatly appreciated!!! FYI: Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling. I'm trying now with a clean install of Mist 0.9.3, using geth 1.7.2. This should support Byzantium and address the canonical block issue.

Any comment from anyone in the Ethereum team on minimum hardware requirements for all the Ethereum enthusiasts out there that will stumble over this issue? Apparently I wasn't alone, I assume this is a huge trap to fall into for many. Also important for future hardware purchasing considerations.

Will keep this thread posted on progress! Thanks again, Martin!!!

Cheers!
Dirk

Dirksterson on 6 Dec 2017

You're welcome :)

Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling.

@evertonfraga any ideas what would cause that ^ ?

holiman on 6 Dec 2017

Same issue - keeps dropping peers and sync is behing by a 100-200 blocks even though eth.blockNumber seems to think its in sync (etherscan.io confirms its not)

raahil190 on 14 Dec 2017

I got it to work with a hardware upgrade. It syncs today nicely on my MacPro with an I7 3.5Ghz and a SSD via a VDSL connection. To me it seems there are minimum hardware requirements. I have asked several times on a statement from the Ethereum guys. They are usually very helpful (e.g. Byzantium hint on this thread or many, many other threads). However, they seem to be very shy when it comes to a guideline here.

I suggest to everyone with this issue here to upgrade to a SSD hard drive and use a machine that has a processor that is not much older than from a few years ago. And of course some decent internet connection. That will do the trick for most for today. I'm a bit worried how long this will be sufficient.

Dirksterson on 15 Dec 2017

Still no comment on hardware and uplink requirements from Ethereum team. Similar issue on other threads if you google. Oh well....

I keep on updating this thread here with my personal experience, hope this helps you people out there not running into the same traps as I did. Yesterday I tried to sync again with my fast machine that currently syncs easy (see above) but now on a satellite link. Observation: Fail! I measure about 6Mbit/s up- and downstream. But with a 700ms delay (since packets have to travel to space). After successful syncing to block 4723322 (thank you cryptokitties) it stalled. I gave up after 10 hours. Drove an hour today back to civilisation to connect to a 5-7 Mbit DSL line. Progressing sync again. :)

Summary: Satellite link with 6 Mbit connection with 700ms delay NOT sufficient for syncing.

Snippet:

INFO [12-18|22:37:10] Imported new chain segment blocks=5 txs=1027 mgas=35.948 elapsed=3.329s mgasps=10.798 number=4723322 hash=912c52…af50cc
WARN [12-18|22:37:10] Synchronisation failed, dropping peer peer=a979fb575495b8d6 err="retrieved hash chain is invalid"
WARN [12-18|22:37:10] Skipping deep transaction reorg depth=81
INFO [12-18|23:31:59] Regenerated local transaction journal transactions=0 accounts=0
INFO [12-19|00:31:59] Regenerated local transaction journal transactions=0 accounts=0
INFO [12-19|01:31:59] Regenerated local transaction journal transactions=0 accounts=0

Dirksterson on 20 Dec 2017

try to use light client and fast sync... Best use datacenter machine and login remotely via mist command line

On 20 Dec 2017, at 07:46, Dirksterson notifications@github.com wrote:

Still no comment on hardware and uplink requirements from Ethereum team. Similar issue on other threads if you google. Oh well....

I keep on updating this thread here with my personal experience, hope this helps you people out there not running into the same traps as I did. Yesterday I tried to sync again with my fast machine that currently syncs easy (see above) but now on a satellite link. Observation: Fail! I measure about 6Mbit/s up- and downstream. But with a 700ms delay (since packets have to travel to space). After successful syncing to block 4723322 (thank you cryptokitties) it stalled. I gave up after 10 hours. Drove an hour today back to civilisation to connect to a 5-7 Mbit DSL line. Progressing sync again. :)

Summary: Satellite link with 6 Mbit connection with 700ms delay NOT sufficient for syncing.

Snippet:

INFO [12-18|22:37:10] Imported new chain segment blocks=5 txs=1027 mgas=35.948 elapsed=3.329s mgasps=10.798 number=4723322 hash=912c52…af50cc
WARN [12-18|22:37:10] Synchronisation failed, dropping peer peer=a979fb575495b8d6 err="retrieved hash chain is invalid"
WARN [12-18|22:37:10] Skipping deep transaction reorg depth=81
INFO [12-18|23:31:59] Regenerated local transaction journal transactions=0 accounts=0
INFO [12-19|00:31:59] Regenerated local transaction journal transactions=0 accounts=0
INFO [12-19|01:31:59] Regenerated local transaction journal transactions=0 accounts=0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 20 Dec 2017

@holiman missed that notification badly, sorry for the delay.

We're aware of this issue, aiming to fix it for 0.9.4.

Thanks

evertonfraga on 20 Dec 2017

@evertonfraga

What do you mean by 0.9.4 ? This AFAIK is a geth issue and not Ethereum Wallet. I have 3 node running at three different location(countries) and all encountered syncing issues. Can stall for a few hours then start the catch up game which can barely keep up with the delay(as blocks continued to be created when the syncing stalled).
It is kind of like when it functions normally, the node is in sync so can be used for submission of transaction etc. Then it stalled and with luck it can resume again but catch up the blocks and during this period, not usable. So work for a few hours, wait for a few hours and the cycle continue. In a few cases, completely stalled which requires restart.
Need to give parity a try to see if that is better

garyng2000 on 20 Dec 2017

@garyng2000 I was referring to the Ethereum Wallet, in this comment:

Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling.
(https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349664186)

evertonfraga on 20 Dec 2017

ah thanks. there is an embedded version ? Anyway, I myself always started geth in a console then the app which IMO is a better way to isolate the problem. It is usually the backend(geth/parity) that is the problem.

garyng2000 on 20 Dec 2017

I had the same issue with Ubuntu 16.04 LTS and Geth 1.7.3, but could resolve it. Maybe it will help beginners too. In short: be patient and expect a lot of data to be downloaded.

1) Download "https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.7.3-4bb3c89d.tar.gz" binary for Linux (or 32 bit version). Unpack, open a terminal and "cd" into the folder that contains the "geth" binary. Then run:
./geth --fast --cache=1048 console

2) Open another terminal.
cd .ethereum
watch -n 2 du -h .

3) Now wait while "geth" is running and watch the "chaindata" folder grow. I own a new fast laptop with 8 Cores and NVMe. After two or three hours it nearly reached 100 percent, but not completely. I let it run overnight, next morning it was showing latest block (you will find latest block on "etherscan.io").
In the meantime "chaindata" folder was grown to 109 Gigabyte, so make sure you have enough place on your disc. Today it will be even bigger, with older computers and slow internet connection it might take several days.

4) Download https://github.com/ethereum/mist/releases/download/v0.9.3/Ethereum-Wallet-linux64-0-9-3.zip from Github, unzip ZIP archive, change into "linux-unpacked" folder and do ./ethereumwallet. It will connect with the running instance of Geth and the interface will pop up. If you already have an existing wallet, put it in .ethereum/keystore.

surferboy250 on 8 Jan 2018

Still having problems - always stalls within 200 blocks of latest block. After that, it just downloads state entries forever

suspended on 9 Jan 2018

downloading states is normal, just need to wait. I think now is at least 60M. just wait, we have recently do it twice and at the end of the day, it did finished.

cecilia-to on 9 Jan 2018

I had problems earlier with a node loosing sync. Restarted and in 3 hours it had managed to catch up. SSD is struggling now really bad ... message to the bridge we just canny hold her captain we dont have the powaaahh

On 10 Jan 2018, at 00:06, ceciliato310 notifications@github.com wrote:

downloading states is normal, just need to wait. I think now is at least 60M. just wait, we have recently do it twice and at the end of the day, it did finished.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

tomtom87 on 9 Jan 2018

I. Must. Have. Warp. Sync, Scotty...

MrHash on 9 Jan 2018

👍2 👎1

Any resolution? I can provide my setup for analysis. Can't solve it for a week.

legkodymov on 10 Jan 2018

Scrap Geth and use Parity

suspended on 13 Jan 2018

I was able to successfully sync on a Macbook Pro, Sierra, 16mb ram. Steps I took:
1- free up machine from all other noncritical memory-using tasks
2- delete chaindata folder
3- fast sync using geth
4- do not open Mist browser until process is completed.

I synced before 24 h had passed.

ainsleys on 13 Jan 2018

There are 50+ million state entries to download. Downloading blocks and receipts finishes a lot faster, but it's not done until all state entries are downloaded. This takes time.

holiman on 13 Jan 2018

Yes, it's a good point. I think there are two main points of confusion for people:
1- not understanding the normal behavior of fast sync (to switch to normal sync at a certain point, thus appearing to suddenly and drastically slow down just before completion)
2- a strange UI characteristic on the Mist load screen, where the size of "downloading chain structure" appears to outpace the amount downloaded.
Anyway, happy syncing :).

ainsleys on 13 Jan 2018

👍1

is this problem going to be addressed?

tuxx42 on 13 Jan 2018

👍2

this is the nature of P2P. it depend on what peer you find and your peer depends on other and they come and go. there is an option to specifically add certain peers which I believe will eventually be the case that there will be a pool of relatively stable peers(infura?). The program though knows how to 'pick up', just need to wait and if you see it really stuck, stop and restart and see if it can find better peers. why I don't recommend to let Mist control geth, run it seperately

cecilia-to on 13 Jan 2018

👍1

I've never had a problem syncing a geth 1.7.3 on a Debian node, but I ran into the "I just can't get the last 100 blocks in sync" issue on a Ubuntu 16.04 node I was spinning up.

I had been using the PPA version of geth, and I switched to the direct-download release 1.7.3 version of geth and this cleared right up on the same box.

I assume that something in the compilation of the two releases is at fault, but I will note that I moved from running geth from /usr/bin to /usr/local/bin. This machine is otherwise a stock Ubuntu 16.04 server installation.

mjdillon on 17 Jan 2018

👍2

I'm having the same problem using geth on windows, gets stuck on the last hundred blocks

alvaradojl on 18 Jan 2018

👍1

I'm on ubuntu and I have the same error. Is the ubuntu version deprecated ?

winteraz on 18 Jan 2018

For some reason I got a 2x increase in mgasps when I swapped the
pre-compiled binary in for the PPA version on ubuntu. I got another 3x
increase in mgasps on an identical system switching to a stock debian 9
server installation, all of the above untuned.

I haven't had time to look into CLANG vs GCC compiling from source, and
probably won't. It's interesting people on windows have this problem as
well, I was suspecting it had something to do with what versions of
libraries are on Ubuntu vs. Debian.

On Thu, Jan 18, 2018 at 11:33 AM, winteraz notifications@github.com wrote:

I'm on ubuntu and I have the same error. Is the ubuntu version deprecated ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-358702563,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEe1ZrJf9kl7m3ePL_g1e-t_DjNBSKhks5tL3JpgaJpZM4O7ctW
.

mjdillon on 18 Jan 2018

👍1

Same issue running last stable docker (1.7.3)

allquantor on 22 Jan 2018

Same issue running 1.7.3 on Ubuntu 16, 4G Mem, 100G Harddrive

I have restarted quite a few times and every time it gets stuck at a different place, this is really making me loose faith in Ethereum

matYang on 25 Jan 2018

Same here.
Turning it off and on solves it. BUT IF I CAN SOLVE IT BY TURNING IT OFF AND ON WHY CANT THE PROGRAM DETECT NOTHING IS BEING DONE AND TURN ITS SYNC OFF AND ON AGAIN???

ZxMYS on 31 Jan 2018

Can not sync to mainnet

I have tried both --fast and full sync on Mac 10.10 and Ubuntu 16.04. Always gets to within 100-200 blocks of current block height (within about 4-6 hours of start of sync), but never finishes, even days later. Restarting geth helps kick off the program again to move ahead a few blocks, but eventually it just starts spitting out Imported new state entries and stops progressing.

> web3.eth.syncing { currentBlock: 5008772, highestBlock: 5008970, knownStates: 1413559, pulledStates: 1399554, startingBlock: 5008691 }

Syncs fine on rinkeby testnet

I'm happy to help diagnose the issue or provide info, but at this point I don't know where to look. Is this expected?

mattotodd on 1 Feb 2018

Same issue
1.7.3-stable-4bb3c89d
geth --cache=2048

aleksey-makarov on 4 Feb 2018

That's not an issue. That is how it works. The state trie currently is 70+M entries. You need to wait it out until all of those are downloaded.

karalabe on 4 Feb 2018

👎10

How many entries are for today? More than 70M ?
Is there any place where anybody can check it?

michaelx2018 on 6 Feb 2018

@karalabe , how can I wait it out if import of every chain segment takes between 10s and 2min? Blocks are created faster than geth is syncing.

juliuscrn on 17 Feb 2018

👍3

Stop saying "this is how it works", please read the data carefully. We are clearly stating that sync speed is slower than block count increase. We will NEVER catch up. You cannot say "this is how it works" in this case, for obvious reasons...

AlisterAmo on 26 Feb 2018

😄6 ❤1

Yea, it's broken. Is anyone able to sync to current block height from scratch?

austorms on 6 Mar 2018

Syncing Ethereum is a pain point for many people, so I'll try to detail what's happening behind the scenes so there might be a bit less confusion.

The current default mode of sync for Geth is called fast sync. Instead of starting from the genesis block and reprocessing all the transactions that ever occurred (which could take weeks), fast sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain.

Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case, since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross checked with the latest blocks. This phase is called the state trie download and it actually runs concurrently with the block downloads; alas it take a lot longer nowadays than downloading the blocks.

So, what's the state trie? In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the account's are not tampered with. This cryptographic linking is done by creating a tree data structure above the accounts, each level aggregating the layer below it into an ever smaller layer, until you reach the single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie.

Ok, so why does this pose a problem? This trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that noone in the network is trying to cheat you. This itself is already a crazy number of data items. The part where it gets even messier is that this data is constantly morphing: at every block (15s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing 200 times per second. The worst part is that while you are synchronizing, the network is moving forward, and state that you begun to download might disappear while you're downloading, so your node needs to constantly follow the network while trying to gather all the recent data. But until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts.

If you see that you are 64 blocks behind mainnet, you aren't yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You'll need to wait that out too before your node comes truly online.

Q: The node just hangs on importing state enties?!

A: The node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing.

The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we'll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn't know about until then. This is why you might be tempted to think it's stuck on the same numbers. It is not, rather it's discovering and downloading the trie as it goes along.

Q: I'm stuck at 64 blocks behind mainnet?!

A: As explained above, you are not stuck, just finished with the block download phase, waiting for the state download phase to complete too. This latter phase nowadays take a lot longer than just getting the blocks.

Q: Why does downloading the state take so long, I have good bandwidth?

A: State sync is mostly limited by disk IO, not bandwidth.

The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way.

Not only is storing the data very suboptimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even a fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive.

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.

karalabe on 6 Mar 2018

👍29 ❤26 👎1

@karalabe I thought, I would fully understand the fast sync mode, but now I am a little bit confused. I was thinking that fast sync will pick highest node known - 64 and start fetching the state for this block only, while in parallel downloading all the headers/receipts until this block. As you explained, header/receipt download will finish significantly faster than importing the states, so at the end of the fast sync part of the process, we'll see a ton of "importing state entries" messages in the log, before the node switches to the full sync.

Now, what I do not understand: why is data morphing along the way? Is he trying to reconstruct the trie always based on the stateHashRoot known from the most recent header downloaded?

Is the algorithm like:

Downloaded 2000 headers
Get the state root of the most recent header (no. 1999)
get state for children hashes, get state for children's children hashes, etc...
Meanwhile next 2000 headers are downloaded
restart the state import with the new stateHashRoot of the most recent header (3999)
...
?

I thought it would be more straight-forward (without morphing state-trie), like:

get Block from max known height - 64 -> Block P
in PARALLEL: Start downloading the state trie for the block P only (the target trie stays constant, since we're not changing the block)
in PARALLEL: download all headers + receipts until P
when both finished, continue with full sync from P on. Here you'll have to make a full sync of some 6000-20.000 blocks, but that actually works well. The most critical part is the fragile and long-running initial state download part.

EDIT: Moreover, I am able to fast sync the BC on a machine with HDD. Tried it yesterday with geth 1.7.2. The only problem that I had are some deadlocks in acquiring a semaphore. I had to restart some 10-20 times as some people described above. It took in sum ~1day to finish the fast sync part and switich to full sync. From there on I had slow "import new chain segment", maybe ~2 block/15s in average. But still possible to catch up.

EDIT2: ok, I just noticed that my mental model doesn't make sense, since you can not know if P is ok if you don't have all the block headers before it. So geth is doing state download based on the stateroot hash of the most recent header it has verified so far, right? And beacause it's like that, it has to cope with the morphing trie. In this case, wouldn't if be better to serialize the algorithm in order to reduce it's complexity. If you first focus to download the headers as fast as possible and than start to fetch the states for only one single block. You will not have to reorganize the trie all the time. It could be faster than what we have now?

It would be:

calculate the baseline block number @ BC Height - 64 -> Block P
Download all headers + receipts until P (no state downloading, trie inserts -> faster header/receipt downloads?)
Then, download the state trie of P (and only P -> no morphing data -> simpler algorithm -> better robustness, less IO).
Continue with full sync from there on and catch up with the remaining 6000-20.000 blocks

ivica7 on 6 Mar 2018

@karalabe Thank you so much for the detailed explanation!

alexo382 on 7 Mar 2018

👍2

@karalabe Appreciate your time, and thank you for the detailed answer. Everytime geth crashes the states reset to zero. Will they continue download from the point they stopped or they are forgotten and they start from the begining again? How do I prevent geth process from being killed. I run Ubuntu 16.04 / 3GB RAM/ SSD and I configured geth as following in the supervisor configuration:
command=/usr/bin/geth --syncmode "fast" --cache=2024 --rpc --rpcaddr 127.0.0.1 rpcport=8545 --rpcapi "web3,eth,debug" --maxpeers "128" autostart=true autorestart=true

EDIT: Now I know the state tries will be forgotton after geth restarts, so I lowered the cache to 768 to avoid being killed by the system, and it seems to work (Synced ~18M in 12 hours, guess will take about 3 days to finish)

sleimana on 10 Mar 2018

@ivica7 I think currently it IS downloading the state only at latest block according to https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/downloader.go#L1366 (and you'll see the latest param passed in processFastSyncContent is the latest block known when sync begins). It did cancel the state downloading process along the way when it finds the state root to download is getting too much behind the new latest.

xiaoyao1991 on 7 Apr 2018

If you download the full blockchain onto 8 HDDs with a Simple (Striped) Partition (Storage Spaces) it is fast enough to keep up with the speed demand.

stricq on 9 Apr 2018

Honestly, this is not a disk i/o issues. Too many users complaining of this issue are using SSD's. I am running into this as well with an NVME SSD with over 200GB's free on the drive. Disc usage tops out at 7% with Geth running.

I have restarted many times, cleared the blockchain data and started fresh, have 175mbps down on my internet. So the issue doesn't appear to be on my end.

However, i constantly see things like this related to peers:
WARN [04-14|18:05:35] Node data write error err="state node e2a085…302335 failed with all peers (3 tries, 3 peers)"
WARN [04-14|18:05:35] Synchronisation failed, retrying err="state node e2a085…302335 failed with all peers (3 tries, 3 peers)"
WARN [04-14|18:05:43] Synchronisation failed, dropping peer peer=74750dbb2b166634 err="action from bad peer ignored"
INFO [04-14|18:07:18] Imported new block headers count=1 elapsed=4.009ms number=5442163 hash=ec78a0…4c2272 ignored=0
INFO [04-14|18:07:48] Imported new block headers count=1 elapsed=3.999ms number=5442164 hash=cd8840…75a192 ignored=0
WARN [04-14|18:07:51] Node data write error err="state node be2a05…8dfaa0 failed with all peers (2 tries, 2 peers)"
WARN [04-14|18:07:51] Rolled back headers count=60 header=5442164->5442104 fast=5442096->5442096 block=0->0
WARN [04-14|18:07:51] Synchronisation failed, retrying err="state node be2a05…8dfaa0 failed with all peers (2 tries, 2 peers)"
INFO [04-14|18:07:54] Imported new block headers count=0 elapsed=910.8µs number=5442164 hash=cd8840…75a192 ignored=68
INFO [04-14|18:08:04] Imported new block headers count=1 elapsed=3.927ms number=5442165 hash=1c0405…2540b9 ignored=0

Etherscan shows Last Block = 5442168 (14.3s)

When I check:
eth.syncing
{
currentBlock: 5442096,
highestBlock: 5442167,
knownStates: 9130491,
pulledStates: 9125287,
startingBlock: 5442004

My highest block is only one behind and I am less than 100 blocks behind.

It looks like a peering issue and them failing on a regular basis.

Mergathal on 15 Apr 2018

it is a peering issue, based on your data, still a long way to go. the # of state is in the range of at least 50M.

garyng2000 on 15 Apr 2018

@Mergathal have a look at this blog write-up:
http://www.freekpaans.nl/2018/04/anatomy-of-a-geth-full-sync/
http://www.freekpaans.nl/2018/04/anatomy-geth-fast-sync/

It is not entirely due to disk I/O, for sure... there are inherent delays in syncing, due to how many peers, and the information you are waiting on from those peers.

If you are waiting on a peer to respond, you might need to reach a timeout before continuing with parts of the block sync.

Also, if the most recent data is not being shared quickly between all peers in the network, then parts of the network are starved of the latest blocks, and there will be a lag in the syncing process.

This info comes from here: https://github.com/ethereum/go-ethereum/issues/14647#issuecomment-378141722

wtfiwtz on 15 Apr 2018

Also @Mergathal, don't restart geth as it will need to restart the validation of the chain structures from the beginning (... i'm on 1.7.2 on my MacBook Pro, but probably still an issue with the latest v1.8.3).

wtfiwtz on 15 Apr 2018

This sections right here strikes me as a big part of this issue.

"What also surprises me is that all the Ethereum data is already larger than the entire Bitcoin data directory (about 200GB), while Bitcoin is almost 3 times older than Ethereum. Clearly, Ethereum grows much faster than Bitcoin. I guess that it’ll become even harder to do full syncs in the future, and that will probably mean the number of full nodes will decrease. That can’t be good."

My thought is that we are starting to connect to a lot more peers that are starting to struggle and fail to sync fully and this is causing us to stay in a perpetual state of out of sync. If these peers never fully sync themselves, we will never catch up either unless we are syncing with only good peers who are fully synced.

I am running the latest 1.8.3 and yes, it is still an issue. I have left mine alone for days and it still is no closer to syncing fully.

Mergathal on 15 Apr 2018

Actually the latest 1.8.2 and above should save its current sync state: https://github.com/ethereum/go-ethereum/pull/16224

wtfiwtz on 15 Apr 2018

@Mergathal that is the nature of a blockchain-based approach. Since BitCoin only has blocks targeting every 10 minutes, the throughput is lower and the number of blocks is lower.

Ethereum generates a new block every 30-60 seconds, allowing more transactions and faster response times. There will naturally be more data generated due to this approach. The data would need to be pruned somehow to keep it at a reasonable level.

Interestingly, in http://www.freekpaans.nl/2018/04/anatomy-geth-fast-sync/, it only took 77Gb of data in the blockchain stored locally for a completed fast sync. I've routinely destroyed fast syncs with much more data than that (... I have limited space on MacBook Pro). It seems to me that the longer that you are pulling down the state tries, the more data that is stored locally. It may also depend on how long you are "full syncing" for as well, once the fast sync is complete. I'm yet to fully understand why but it's an interesting observation.

wtfiwtz on 15 Apr 2018

we constantly 'refresh' by fast sync from scratch to keep the size in check. An initial fast sync is only around 60G(as of may be a month ago) then the size grow. after one month we are seeing 140G. Not sure if it is because older state needs to be pulled in or what. Does anyone with 'true' full sync knows the current disk size ?

garyng2000 on 15 Apr 2018

@garyng2000 a full sync took 220Gb according to the articles linked above. So it would be approximately 80Gb a month as a "fast sync" switches to a "full sync".

wtfiwtz on 15 Apr 2018

🎉1

@wtfiwtz
that is something puzzle me, if it is 80GB a month we are talking TB data soon but how come a 'true' full sync is only 220G ? If that is the case, may be I should do a true full sync(from scratch) that can take a bit of time but the disk growth rate would be slower ? strange.

garyng2000 on 15 Apr 2018

👍1

@garyng2000 it could be because the accumulated state is bigger as you participate in the immediate verification of the transactions, where as post-verification is not as much information to download from peers. However, you would need someone more knowledgeable about Ethereum's inner workings to confirm or deny that.

wtfiwtz on 15 Apr 2018

👍1

I'm on geth v1.8.4 and Ubuntu 16.04. Not only is geth stopping before final sync, but it completely stalls around 30-60 minutes after starting a sync. The CPU usage drops to ~3% of capacity and stays there.

screen shot 2018-04-19 at 6 10 37 pm

I see continuous error messages for connecting to nodes, and the state and blocks completely stop updating. I have to restart geth (I use systemd restart). This is very concerning because I don't want my node to stall in the middle of serving our dapp.

GeeeCoin on 20 Apr 2018

@GeeeCoin you might want to try v1.8.3 - have a simular issue to yours when I moved from .3 to .4

suspended on 20 Apr 2018

😕1

@suspended v1.8.6 has the same unresolved issue. **downgrading to geth v1.8.3 worked for about 3 weeks, but now facing the same issues

GeeeCoin on 21 Apr 2018

🎉1

I am also having the same sync problems... dropping peers etc. I am almost synced (about 50-100 blocks behind if I let it run). If I restart geth it catches up until peers start to drop again.

Using Ubuntu 16.04. I have tried different versions of Geth down to 1.8.2. Built the dev version too with no change.

I have lots of experience running a node having done it since the start... but I did re-download the block chain a month or 2 ago.

I use a SATA 500GB SSD but it is encrypted on the drive level and the home directory which is where the blockchain is stored. The encryption means that the read/write abilities are slower and using a disk monitor it shows a high level of activity constantly while geth is running.

I understand storing/using the blockchain on encrypted drive is probably not the best setup (for speed and amount of read writes/life of SSD) so I'm guessing the next thing I should try is a new separate un-encrypted SSD to store the chain... but I have not got round to doing so yet (having another SSD purely for eth blockchain is fairly expensive option). Currently my chaindata folder is 358.8GB

Looks like Ubuntu 16.04 is a consistent part of this thread/problem?

mtj151 on 13 May 2018

@mtj151 good observation. I'm not ruling out any factors at this point. Is anyone using AWS by any chance?

GeeeCoin on 18 May 2018

I have also noticed that I am unable to send transactions while I am getting the "Synchronisation failed, retrying err="block download cancelled (requested)"" warnings.

I sent one transaction fine but then the warnings come up and it wouldn't let me send another transaction (even after the messages stopped and syncing started again). I had to completely restart geth to be able to send the transaction.

mtj151 on 21 May 2018

@GeeeCoin I was unable to get a Geth node to stay up to date with chaintip on AWS in any meaningful time without using Provisioned IOPS SSDs on EBS-optimized instances or the i3 storage-optimized instances with 8GB RAM or greater. Even then, I had to write a watchdog to kick geth over every now and then for when it would drop all its peers or lag too much behind the chaintip. Now I just have dedicated boxes for geth nodes running NVMe SSD in the datacenter, and a NUC for LAN dev which has a 1 TB SATA SSD, 8GB RAM and a quad-core processor.

ghost on 25 May 2018

@10a7 appreciate the data point. If NUC is outperforming a quad core with 8GB in AWS, that's a problem. Amazon may have network latency that hasn't been optimized with the t. class. The i3 looks like an option. We're taking a look at Quarian; thanks for building that out!

GeeeCoin on 25 May 2018

Sounds like 10a7 had the same problem with lagging behind the chain tip... good description of the problem. Did NVMe SSD fix the problem?? I'm looking at getting one in the coming weeks to run geth.

mtj151 on 7 Jun 2018

@mtj151 NVMe SSD doesn't seem to matter. I have no trouble keeping SATA SSDs and bcache-fronted magnetic arrays intact and synced I/O wise.

If you are synced and "importing new chain segment", it seems to mostly be network issues that cause my nodes to fall behind. Restarting geth often helps to get different peers. Geth sync-after-fast-pivot is also much more reliable for me if I am not behind a NAT, and can forward/open 30303/tcp.

ghost on 8 Jun 2018

FWIW I was able to get geth to fully sync by waiting until eth.blockNumber is near the numbers in eth.syncing and then restarting geth. I was able to do this at ~160m states. After restarting geth, it took about 20 min to catch up to the blockchain and now eth.syncing is false and the only output now is 'imported new chain segment' every time a new block is found.

jdowning100 on 17 Jun 2018

@
Syncing Ethereum is a pain point for many people, so I'll try to detail what's happening behind the scenes so there might be a bit less confusion.

The current default mode of sync for Geth is called fast sync. Instead of starting from the genesis block and reprocessing all the transactions that ever occurred (which could take weeks), fast sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain.

Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case, since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross checked with the latest blocks. This phase is called the state trie download and it actually runs concurrently with the block downloads; alas it take a lot longer nowadays than downloading the blocks.

So, what's the state trie? In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the account's are not tampered with. This cryptographic linking is done by creating a tree data structure above the accounts, each level aggregating the layer below it into an ever smaller layer, until you reach the single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie.

Ok, so why does this pose a problem? This trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that noone in the network is trying to cheat you. This itself is already a crazy number of data items. The part where it gets even messier is that this data is constantly morphing: at every block (15s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing 200 times per second. The worst part is that while you are synchronizing, the network is moving forward, and state that you begun to download might disappear while you're downloading, so your node needs to constantly follow the network while trying to gather all the recent data. But until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts.

If you see that you are 64 blocks behind mainnet, you aren't yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You'll need to wait that out too before your node comes truly online.

Q: The node just hangs on importing state enties?!

A: The node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing.

The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we'll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn't know about until then. This is why you might be tempted to think it's stuck on the same numbers. It is not, rather it's discovering and downloading the trie as it goes along.

Q: I'm stuck at 64 blocks behind mainnet?!

A: As explained above, you are not stuck, just finished with the block download phase, waiting for the state download phase to complete too. This latter phase nowadays take a lot longer than just getting the blocks.

Q: Why does downloading the state take so long, I have good bandwidth?

A: State sync is mostly limited by disk IO, not bandwidth.

The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way.

Not only is storing the data very suboptimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even a fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive.

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.

karalabe on 4 Oct 2018

👍9

@karalabe Thanks for breaking this down again. We knew most of this about Geth/Eth already, but I'm really surprised as to how suboptimal the state trie system is at being stored to disk; I thought the whole point of building ethereum this way (with modified patricia trees etc.) was to minimize footprint/disk mods, but looks like innovation in storage structures is still needed.

GeeeCoin on 4 Oct 2018

@karalabe . Nice introduction. Understanding fast sync internal better.

hustnn on 11 Oct 2018

@karalabe So is there any way of knowing how close you are to being finished syncing? None of the metrics from the eth_syncing call seems to carry meaningful information about this.

quietnan on 22 Oct 2018

👍2

@karalabe So is there any way of knowing how close you are to being finished syncing? None of the metrics from the eth_syncing call seems to carry meaningful information about this.

https://github.com/ethereum/go-ethereum/issues/16558
https://eips.ethereum.org/EIPS/eip-2029

If those are actually implemented, you'll at least be able to scrape the number of states from an external reference.

nyetwurk on 27 Jan 2020

Go-ethereum: geth --fast stalls before crossing finish line

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Backtrace

Most helpful comment

All 152 comments

#### BAD BLOCK

#

#### BAD BLOCK

#

#### BAD BLOCK

#

#### BAD BLOCK

#

#### BAD BLOCK

#

#### BAD BLOCK

#

#### BAD BLOCK

#

Related issues