Raspiblitz: Blockchain sync keeps failing when transferring files through WinSCP

[Z-and]

I've tried three times to sync the blockchain data from my windows computer to the Pi through WinSCP. As far as I can tell, everything is setup properly (v.0.17.1 with txindex=1 in the config file). I transferred all the necessary folders (blocks, chainstate, and indexes) into /mnt/hdd/bitcoin/, the whole process takes around 4-5 hours. After the transfer completes and I try hitting enter to continue with the setup but I get the following message.

[openoms]

Just to confirm: did you restart bitcoind on your Windows computer after setting txindex=1? (it needs to reindex)

To rule out a power supply issue, please copy-paste and run this commend in your RaspiBlitz terminal:
wget https://raw.githubusercontent.com/openoms/raspiblitz/powertest/home.admin/XXpowertest.sh && sudo chmod +x XXpowertest.sh && ./XXpowertest.sh
It runs for a minute, just copy the output here afterwards.

[CommanderPoe]

Did you stop the bitcoin client before copying it?

[Z-and]

Just to confirm: did you restart bitcoind on your Windows computer after setting txindex=1? (it needs to reindex)

To rule out a power supply issue, please copy-paste and run this commend in your RaspiBlitz terminal:
wget https://raw.githubusercontent.com/openoms/raspiblitz/powertest/home.admin/XXpowertest.sh && sudo chmod +x XXpowertest.sh && ./XXpowertest.sh
It runs for a minute, just copy the output here afterwards.

I did in fact restart bitcoind on the Windows computer after setting txindex=1, however it only took a minute or so for Bitcoin Core to load up. Should reindexing normally take longer? I will double check the power supply when I get home (currently at work) but I don't think that's the issue. I have a second power supply in case it is though.

Did you stop the bitcoin client before copying it?

Yes

[openoms]

I did in fact restart bitcoind on the Windows computer after setting txindex=1, however it only took a minute or so for Bitcoin Core to load up. Should reindexing normally take longer? I will double check the power supply when I get home (currently at work) but I don't think that's the issue. I have a second power supply in case it is though.

Check if Bitcoin Core on your windows is fully synced with txindex=1 still active.
It helps us if your run the powertest, the more environments are tested, the better.

[Z-and]

I did in fact restart bitcoind on the Windows computer after setting txindex=1, however it only took a minute or so for Bitcoin Core to load up. Should reindexing normally take longer? I will double check the power supply when I get home (currently at work) but I don't think that's the issue. I have a second power supply in case it is though.

Check if Bitcoin Core on your windows is fully synced with txindex=1 still active.
It helps us if your run the powertest, the more environments are tested, the better.

Bitcoin Core is fully synced with txindex=1 active in the config file. Also, I've attached an image of the CPU test

[openoms]

thanks fro running the test. It shows that your Pi is running quite hot and throttles the CPU significantly.
This should not cause a problem only impacts the performance. You can try removing part of you case to allow more airflow, apply a heatsink, use a fan, but not essential. Your power supply is probably OK.

Regarding your disk the next step to check the filesystem of you HDD:
unmount:
sudo umount /dev/sda1
scan and repair:
sudo fsck /dev/sda1

Please copy the output here. You can try to copy the blockchain again once finished.

[fluidvoice]

thanks fro running the test. It shows that your Pi is running quite hot and throttles the CPU significantly.
This should not cause a problem only impacts the performance. You can try removing part of you case to allow more airflow, apply a heatsink, use a fan, but not essential. Your power supply is probably OK.

Regarding your disk the next step to check the filesystem of you HDD:
unmount:
sudo umount /dev/sda1
scan and repair:
sudo fsck /dev/sda1

Please copy the output here. You can try to copy the blockchain again once finished.

While that is quite hot. I would have thought the throttle would be more than just 1200Mhz.
I really think everyone running Raspiblitz should be using a decent heatsink with a fan.
It's a small extra cost to extend the life of the Pi board. Myself included, need to get a fan.

[CommanderPoe]

I really think everyone running Raspiblitz should be using a decent heatsink with a fan.

Gezz... My pi came with a heatsink and i didnt even use it... i hav always wondered what is that lil piece of metal for! facepalm!

[openoms]

Gezz... My pi came with a heatsink and i didnt even use it... i hav always wondered what is that lil piece of metal for! facepalm!

Hope you still have it! If you device to stick it on could you run our little test before and after? Would be interesting to see the difference in the results:
wget https://raw.githubusercontent.com/openoms/raspiblitz/powertest/home.admin/XXpowertest.sh && sudo chmod +x XXpowertest.sh && ./XXpowertest.sh

[Z-and]

thanks fro running the test. It shows that your Pi is running quite hot and throttles the CPU significantly.
This should not cause a problem only impacts the performance. You can try removing part of you case to allow more airflow, apply a heatsink, use a fan, but not essential. Your power supply is probably OK.

Regarding your disk the next step to check the filesystem of you HDD:
unmount:
sudo umount /dev/sda1
scan and repair:
sudo fsck /dev/sda1

Please copy the output here. You can try to copy the blockchain again once finished.

No problem, I already do have heatsinks installed onto the Pi. I'm guessing it was running hot because I tried reindexing the blockchain overnight before running those tests.

When I try unmounting I get the following error message:

umount: /mnt/hdd: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

[openoms]

Couple more methods here to get that HDD unmount: https://github.com/rootzoll/raspiblitz/issues/442#issuecomment-478698639
Restart with sudo reboot , press CTRL+C after login and try in order.
When managed to unmount run the sudo fsck /dev/sda1 to check you HDD.

sudo umount /dev/sda1
fuser -cuk /mnt/hdd
sudo umount -l /dev/sda1
sudo umount -f /dev/sda1

sudo fsck /dev/sda1

[fluidvoice]

Gezz... My pi came with a heatsink and i didnt even use it... i hav always wondered what is that lil piece of metal for! facepalm!

Hope you still have it! If you device to stick it on could you run our little test before and after? Would be interesting to see the difference in the results:
wget https://raw.githubusercontent.com/openoms/raspiblitz/powertest/home.admin/XXpowertest.sh && sudo chmod +x XXpowertest.sh && ./XXpowertest.sh

I've found these small heat sinks that have no fan do almost nothing to help heat for long-duration loads like in our case when Raspiblitz is syncing the blockchain DB. I've been using one of these and my temps are still quite high. In fact some of these heatsinks make things worse because the double sided tape they use to attach the heatsink is a poor heat conductor, and actually acts as an insulator! Optimally, heatsinks should be using thermal paste, not one of these stickers, though some of them work reasonably OK.
see this dual-fan heatsink test: https://www.youtube.com/watch?v=j_QrsNwVwNA

[Z-and]

Couple more methods here to get that HDD unmount: #442 (comment)
Restart with sudo reboot , press CTRL+C after login and try in order.
When managed to unmount run the sudo fsck /dev/sda1 to check you HDD.

sudo umount /dev/sda1
fuser -cuk /mnt/hdd
sudo umount -l /dev/sda1
sudo umount -f /dev/sda1

sudo fsck /dev/sda1

I tried what you suggested but ended up with an error message. Please see screenshot below (keep in mind the partition with the blockchain on my HDD is sda2, not sda1)

[openoms]

@Z-and sda2 is strange. What is on sda1? I would suggest to format your HDD on a different computer and reattach. Run ./30initHDD or just start over again with a fresh SDcard.

[Z-and]

@Z-and sda2 is strange. What is on sda1? I would suggest to format your HDD on a different computer and reattach. Run ./30initHDD or just start over again with a fresh SDcard.

Ok I'll give that a try, but keep in mind I've reformatted the HDD once before and it didn't seem to change anything. I'm currently redownloading a copy of the blockchain (the original copy from my PC) to see if that fixes anything.

[openoms]

Having your HDD as sda2 suggests that there are 2 partitions on it. You really want it to only have one. Wipe it clean ;)

[openoms]

Try a partition manager on Windows to format your HDD. I have used the Minitool Partition Wizard or have a look at this tweet: https://twitter.com/TheShitcoiner/status/1113585224942870528?s=19

[Z-and]

Try a partition manager on Windows to format your HDD. I have used the Minitool Partition Wizard or have a look at this tweet: https://twitter.com/TheShitcoiner/status/1113585224942870528?s=19

I downloaded Minitool Partition Wizard (despite the amount of spamware it forces you to download) and removed the 16MB partition, not sure why it was there. I also redownloaded a fresh copy of the blockchain and now just doing another sync to the Pi HDD. I'll report back once finished.

[Z-and]

Couple more methods here to get that HDD unmount: #442 (comment)
Restart with sudo reboot , press CTRL+C after login and try in order.
When managed to unmount run the sudo fsck /dev/sda1 to check you HDD.

sudo umount /dev/sda1
fuser -cuk /mnt/hdd
sudo umount -l /dev/sda1
sudo umount -f /dev/sda1

sudo fsck /dev/sda1

The blockchain sync failed for a 4th time. This is after removing the partition and downloading a new version of the blockchain on my main PC. I'm about ready to give up and go back to running the Stadicus version, so frustrating...

However, on a brighter note, I was able to successfully unmount and check the HDD. I got the following output

[openoms]

The blockchain sync failed for a 4th time. This is after removing the partition and downloading a new version of the blockchain on my main PC. I'm about ready to give up and go back to running the Stadicus version, so frustrating...
However, on a brighter note, I was able to successfully unmount and check the HDD. I got the following output

Sorry to read that. Your fsck result looks OK.
What errors the blockchain sync failed with?
It can be also a network problem with the connection going down while transferring.

The Stadicus guide (on which the RaspiBlitz is based) has the exact same method for transferring the blockchain, just more manual.

If there is a problem with the scp transfer failing -because of an unstable network connection- you could still try downloading by the torrent method. Why not to give it a try?

[Z-and]

The blockchain sync failed for a 4th time. This is after removing the partition and downloading a new version of the blockchain on my main PC. I'm about ready to give up and go back to running the Stadicus version, so frustrating...
However, on a brighter note, I was able to successfully unmount and check the HDD. I got the following output

Sorry to read that. Your fsck result looks OK.
What errors the blockchain sync failed with?
It can be also a network problem with the connection going down while transferring.

The Stadicus guide (on which the RaspiBlitz is based) has the exact same method for transferring the blockchain, just more manual.

If there is a problem with the scp transfer failing -because of an unstable network connection- you could still try downloading by the torrent method. Why not to give it a try?

I don't remember what the error was for the blockchain sync, but I went ahead and downloaded the torrent which worked without any issues. I'm still puzzled by why the "copy" with WinSCP didn't work even after reformatting the HD and downloading a new copy of the blockchain. Oh well, at least I'm up and running now. Appreciate the help.

[fluidvoice]

The blockchain sync failed for a 4th time. This is after removing the partition and downloading a new version of the blockchain on my main PC. I'm about ready to give up and go back to running the Stadicus version, so frustrating...
However, on a brighter note, I was able to successfully unmount and check the HDD. I got the following output

Sorry to read that. Your fsck result looks OK.
What errors the blockchain sync failed with?
It can be also a network problem with the connection going down while transferring.
The Stadicus guide (on which the RaspiBlitz is based) has the exact same method for transferring the blockchain, just more manual.
If there is a problem with the scp transfer failing -because of an unstable network connection- you could still try downloading by the torrent method. Why not to give it a try?

I don't remember what the error was for the blockchain sync, but I went ahead and downloaded the torrent which worked without any issues. I'm still puzzled by why the "copy" with WinSCP didn't work even after reformatting the HD and downloading a new copy of the blockchain. Oh well, at least I'm up and running now. Appreciate the help.

SCP scp does not guarantee file integrity since it does not use file checksums like rsync does. So though unlikely it's possible you had some files corrupted when copying.

[fluidvoice]

why the "copy" with WinSCP didn't work

@Z-and If you can, please try testing out other Windows methods so maybe we can improve things for other Windows users: https://github.com/rootzoll/raspiblitz/issues/502#issue-431079789
I know it may take a long time, but maybe you can do it in the background (if you have the disk space)?