Nano-node: Database migration of V22.0 doesn't use full performance and takes more than 24 hours

[thisdudeisvegan]

Summary

Hello,
I talked with several users about this issue now and the only problem they had, was, that the database migration took 4+ hours (8 hours in the case of one person). However, I did the update to my node, which is running with 10 CPU cores and 48GB RAM. The server is a dedicated server with an SSD.
The process of the update is working as it should, but the database migration process doesn't use the resources it could use. In my case this means, that the update is now coming to an end after more than 24 hours.

Checking I/O showed me, that the node is only using about 2-60MB (in peaks) read and write speed of the SSD. The SSD can achieve about 2.6GB per second under load.
I wasn't able to figure out why the update process isn't using the resources it could use, but I wanted to open an issue here. Maybe someone had the same or a similar issue and is able to figure out the reason for this.

My log from the update process:
Starting up Nano node... Upgrade in progress...

The ledger size is currently at 148GB (so nearly 150GB), which means the update should be finished soon. For reference it's currently 9.30pm server time. (21:30:00)

Node version

V22.0

Build details

The node is running with the "latest" docker image. (the latest tag got removed in V22.0, but it's 22.0)

OS and version

Ubuntu 18.04.5 LTS x86_64

Steps to reproduce the behavior

(Can't really tell the reproduction, because this issue came to existence while updating the node the normal and documented way.)

Expected behavior

Update process shouldn't take longer than 4 hours.

Actual behavior

Update process is taking 24+ hours until now and isn't completely finished yet.

Possible solution

_No response_

Supporting files

_No response_

[manhlicious]

Do you have a lot of free space on the SSD? Since the SSD performance will reduce if there are little free space left

[thisdudeisvegan]

Do you have a lot of free space on the SSD? Since the SSD performance will reduce if there are little free space left

Hello, the SSD has about 800GB free space left, so this shouldn't be the issue I guess.

[thisdudeisvegan]

A short follow up:
The update is now finished after about 2 days. The node is now back up and running again with full power and is also voting. Somehow I had to bridge the nano node monitor and the node, so that the nano node monitor was working again.

However, I'm still left with the question if the extremely long update process was a weird bug of the current version, that only happens with a small chance, or if it was any kind of issue on our side. Maybe someone has an idea, but I don't have high hopes that the bug can be reproduced as the bug occurred during the regular update process. (Which in itself is very simple.)

[BatuAksoy]

Same situation on our node too. Process is stuck at Write state blocks to new format for more than 2 days.

CPU and memory usage is stable. Disk space is enough, Disk read is around 30M/s and Disk write occasionally raises 150+ M/s and drops back.

[thisdudeisvegan]

Same situation on our node too. Process is stuck at Write state blocks to new format for more than 2 days.

CPU and memory usage is stable. Disk space is enough, Disk read is around 30M/s and Disk write occasionally raises 150+ M/s and drops back.

Sad to hear, but at least good to know, that it then seems to be a bug of the current version that only appears sometimes instead of being an issue at my end. Hope this thread will help the devs to find and maybe fix the issue in some way.

[Falsen]

This happend on our node too. I created a new droplet on DO for the upgrade (16 GB RAM, 320 GB SSD), it ran for 3 hours before I stopped and am now retrying on a new node. (I thought that something had gone wrong). The node never used more than 2% CPU, 20% RAM, and the disk usage slowly increases up from around 20%. Disk write is around 15 MB/s, should be able to do much more. Will see how it goes on second try!