Nano-node: Bootstrapping blocks slowly crawl to a halt
Summary
When bootstrapping blocks, slowly the number of blocks fetched/downloaded per second goes down.
As an example, when I first started fetching blocks from other peers, I downloaded 2,000 blocks/second.
After hitting 21.3 million blocks (60 million for someone else), it slowed down to 20-30 blocks/second.
Local device resources seem to be fine
Node version
V22.0
Build details
Docker container, nothing out of the ordinary.
OS and version
Linux 20.04
Steps to reproduce the behavior
- Follow installation instruction in the docs
- Start bootstrapping
- Monitor block/sec progress
Expected behavior
The software continues to sync blocks at a similar/ascending rate, not a descending rate
Actual behavior
The software syncs blocks at a descending rate
Possible solution
Honestly, I don't have any idea
Supporting files
_No response_
DarrenAlex
All 17 comments
Same issue for me with multiple nodes/bootstrapping attempts (Centos 8). The slowdown happens around ~59-60M blocks for me though. In the last 4 hours, I've only gained ~100 counted blocks.
$ cat startTime.txt
Universal time: Thu 2021-05-13 20:49:04 UTC
$ curl -d '{"action": "block_count"}' 127.0.0.1:7076
{
"count": "60680998",
"unchecked": "60623590",
"cemented": "37743403"
}
$ timedatectl
Universal time: Mon 2021-05-17 16:15:28 UTC
Not sure if related, but I see a bunch of these messages in the logs:
[2021-May-17 16:12:27.089344]: Error initiating bootstrap connection to [::ffff:194.195.243.169]:57930: No route to host
qwahzi
on 17 May 2021
I also receive those messages in the log file. I've uploaded the log files (only lines which contain the word "bootstrap") to https://node.erawan.me/log.txt
The only difference is that the counted and cemented blocks are stuck at 1 for me.
If you sum them all up, it's close to the network's total blocks. Is the cemented block number going up?
DarrenAlex
on 17 May 2021
The same problem!
atomlab
on 20 May 2021
+1
smithy9901
on 20 May 2021
My CentOS 8 node (RocksDB) suddenly caught up on count (from 70M to 117M) overnight. On V21.3 with LMDB, the same node was stuck at <60M count for over a month
curl -d '{"action": "block_count"}' 127.0.0.1:7076
{
"count": "117825631",
"unchecked": "972886",
"cemented": "52043280"
}
My Windows 10 RocksDB + pruning node is still stuck:
{
"count": "51852881",
"unchecked": "77711715",
"cemented": "39891722",
"full": "13068695",
"pruned": "38784186"
}
Running into the same issue.
{
"count": "50864256",
"unchecked": "65442030",
"cemented": "33401959"
}
Gets to this point and now it's slowed to a crawl that will never catch up.
I did catch this in the logs but I'm not sure if it matters.
[2021-May-26 04:10:40.444040]: Starting legacy bootstrap attempt with ID auto_bootstrap_0
[2021-May-26 04:10:40.444231]: Bootstrap attempt stopped because there are no peers
[2021-May-26 04:10:40.444312]: Exiting legacy bootstrap attempt with ID auto_bootstrap_0
[2021-May-26 04:10:40.452590]: Beginning pending block search
[2021-May-26 04:10:40.452708]: IPC: server started
In addition I'm only at 43% synced shown by the nano node stats monitor and have database size of 100G. Is nano really already at over 200G for it's blockchain size?
My node has had consistent memory, cpu, and disk usage throughout the entire sync however I'm not really sure what it's doing with all this data it's writing. I did start from scratch with a v22 node so it shouldn't have to be doing any conversion between node database versions.
michaeljs1990
on 26 May 2021
I had the same message in my logs when my Windows 10 node disconnected:
2021-May-22 10:31:30.217942]: UPnP local address: 192.168.86.25, discovery: 0, IGD search: 1
[2021-May-22 10:32:34.046196]: Total recently pruned block count: 0
[2021-May-22 10:33:02.575300]: Beginning pending block search
[2021-May-22 10:37:54.012282]: Total recently pruned block count: 0
[2021-May-22 10:38:02.575701]: Beginning pending block search
[2021-May-22 10:41:25.958997]: Starting legacy bootstrap attempt with ID auto_bootstrap_85
[2021-May-22 10:41:25.958997]: Bootstrap attempt stopped because there are no peers
[2021-May-22 10:41:25.958997]: Exiting legacy bootstrap attempt with ID auto_bootstrap_85
[2021-May-22 10:43:02.576639]: Beginning pending block search
[2021-May-22 10:43:13.742616]: Total recently pruned block count: 0
[2021-May-22 10:45:16.239424]: UPnP local address: 192.168.86.25, discovery: 0, IGD search: 1
[2021-May-22 10:48:02.577391]: Beginning pending block search
[2021-May-22 10:48:33.333862]: Total recently pruned block count: 0
[2021-May-22 10:53:02.577717]: Beginning pending block search
[2021-May-22 10:53:54.688580]: Total recently pruned block count: 0
[2021-May-22 10:56:25.959882]: Starting legacy bootstrap attempt with ID auto_bootstrap_86
[2021-May-22 10:56:25.959882]: Bootstrap attempt stopped because there are no peers
[2021-May-22 10:56:25.959882]: Exiting legacy bootstrap attempt with ID auto_bootstrap_86
[2021-May-22 10:58:02.578748]: Beginning pending block search
[2021-May-22 10:59:02.242813]: UPnP local address: 192.168.86.25, discovery: 0, IGD search: 1
[2021-May-22 10:59:19.093811]: Total recently pruned block count: 0
[2021-May-22 11:03:02.579804]: Beginning pending block search
[2021-May-22 11:04:39.309938]: Total recently pruned block count: 0
[2021-May-22 11:08:02.581172]: Beginning pending block search
After restarting the node again and leaving it for another week, it seems to be making some progress:
{
"count": "61293347",
"unchecked": "67421914",
"cemented": "51092686",
"full": "12016287",
"pruned": "49277060"
}
Centos 8 node full sync status update:
{
"count": "118174879",
"unchecked": "902500",
"cemented": "74119823",
"full": "50116606",
"pruned": "68058273"
}
You can download the latest bootstrap database file from here and then start up the node. Took me about an hour between download and extract of the 7z file and then I was at 100% a few minutes after starting. Doesn't fix the underlying issue but if you are trying to get a node up and running it's the fastest way right now.
michaeljs1990
on 27 May 2021
downloading the latest bootstrap database file from here
The whole point of bootstrapping is to not use the ledger download.
DarrenAlex
on 28 May 2021
My CentOS 8 node (RocksDB) suddenly caught up on count (from 70M to 117M) overnight. On V21.3 with LMDB, the same node was stuck at <60M count for over a month
```
Same jump here, I've been on 72M/70M count/cemented at 7:13 today and now at 119M/77M count/cemented at 14:04. Days before it was crawling slowly with only hundreds of count increase in hours.
(RocksDB)
regfaker
on 28 May 2021
downloading the latest bootstrap database file from here
The whole point of bootstrapping is to not use the ledger download.
I agree but if your goal is to get a node running that looks to be the only way to get one up and running before the network started moving a bit faster. In the end if the bootstrap file was bad your blocks would be rejected by the network anyway.
michaeljs1990
on 28 May 2021
Centos 8 node status update, after 17 days:
$ cat startTime2.txt
Universal time: Thu 2021-05-13 20:49:04 UTC
$ curl -d '{"action": "block_count"}' 127.0.0.1:7076
{
"count": "118491518",
"unchecked": "956586",
"cemented": "98261377",
"full": "33622456",
"pruned": "84869062"
}
$ timedatectl
Universal time: Sun 2021-05-30 14:11:20 UTC
Windows 10 node status update (still pretty stuck):
{
"count": "63138031",
"unchecked": "65450762",
"cemented": "57874107",
"full": "7837324",
"pruned": "55300707"
}
Same here, windows. 14 days. Confirming at 30 cps.
Joohansson
on 2 Jun 2021
Both of my nodes are now unstuck and getting pretty close:
Centos 8:
{
"count": "119027602",
"unchecked": "791562",
"cemented": "108980055",
"full": "28467965",
"pruned": "90559637"
}
Windows 10:
{
"count": "118949710",
"unchecked": "1785684",
"cemented": "86111699",
"full": "39533449",
"pruned": "79416261"
}
Mine started to move now as well, 5000 bps. Block count 92M
Edit day 16: Still 31M unconfirmed
Joohansson
on 3 Jun 2021
My Centos 8 node officially finished syncing from scratch last night (sometime in the last 12 hours). For some reason it kept restarting itself roughly once a day over the last week, but there were no details in the logs
$ cat startTime.txt
Universal time: Thu 2021-05-13 20:49:04 UTC
$ curl -d '{"action": "block_count"}' 127.0.0.1:7076
{
"count": "120612378",
"unchecked": "28987",
"cemented": "120612370",
"full": "25587436",
"pruned": "95024942"
}
$ timedatectl
Universal time: Sun 2021-06-06 13:10:32 UTC
$ du -xhs /nano
22G /nano
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
XXXX nanocurrency/nano:V22.0 "/usr/bin/entry.sh n…" 2 weeks ago Up 16 hours 127.0.0.1:7076->7076/tcp, 0.0.0.0:7075->7075/tcp, 0.0.0.0:7075->7075/udp, :::7075->7075/tcp, :::7075->7075/udp, 127.0.0.1:7078->7078/tcp nano-container
My Windows 10 node is still going, but making progress:
{
"count": "118992721",
"unchecked": "2149929",
"cemented": "95323574",
"full": "34485057",
"pruned": "84507664"
}
I still have 21M to confirm and mostly doing 25-30 cps so I give up. It was just an experiment but simply taking too long for me.
Joohansson
on 8 Jun 2021
Related issues
Marc477
·
16Comments
triwebb1
·
21Comments
rkeene
·
71Comments
cryptocode
·
13Comments
pedfx
·
15Comments
Most helpful comment
You can download the latest bootstrap database file from here and then start up the node. Took me about an hour between download and extract of the 7z file and then I was at 100% a few minutes after starting. Doesn't fix the underlying issue but if you are trying to get a node up and running it's the fastest way right now.