Raspiblitz: "The bitcoind service is not running." happened 3 times in last week. I did fresh reinstall every time.

[fifthedition]

I got this issue 3 times since raspiblitz 1.4 was released. I do a fresh install of 1.4 rapsiblitz with formated hdd. As soon as I start to install services or mobile wallet support, I run to this message after reboot:

               Blockchain Info                                                
│  The bitcoind service is not running.   │                                               
│  Login for more details:                │                                               
│  Use CTRL+c to EXIT to Terminal         │

So far I received the above message after I installed RTL, Block Explorer and ZAP mobile wallet client. It seemed that if dont enable anything the raspiblitz will run perfectly fine.

here is a whole log file obtained with this code
sudo cat /mnt/hdd/bitcoin/debug.log | nc termbin.com 9999
https://termbin.com/fkxx

This full log is after fresh installation and an attempt to enable ZAP for iOS in the menu. After reboot I got the above message (The bitcoind service is not running.)

Here is also XXdebugLogs.sh

***************************************************************
* RASPIBLITZ DEBUG LOGS 
***************************************************************
blitzversion: 1.4
chainnetwork: bitcoin / main
 23:03:01 up 23:56,  2 users,  load average: 1.17, 1.15, 0.97

*** BLOCKCHAIN SYSTEMD STATUS ***
● bitcoind.service - Bitcoin daemon
   Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-02-26 23:02:44 GMT; 17s ago
  Process: 2831 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED (code=exited, status=0/SUCCESS)
  Process: 2833 ExecStart=/usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin/bitcoind.pid (code=exited, status=0/SUCCESS)
 Main PID: 2881 (bitcoind)
    Tasks: 13 (limit: 4915)
   Memory: 169.6M
   CGroup: /system.slice/bitcoind.service
           └─2881 /usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin/bitcoind.pid

Feb 26 23:02:43 raspberrypi systemd[1]: Starting Bitcoin daemon...
Feb 26 23:02:44 raspberrypi systemd[1]: Started Bitcoin daemon.

*** LAST BLOCKCHAIN ERROR LOGS ***
sudo journalctl -u bitcoind -b --no-pager -n8
-- Logs begin at Tue 2020-02-25 23:05:52 GMT, end at Wed 2020-02-26 23:03:01 GMT. --
Feb 26 23:01:52 raspberrypi systemd[1]: Started Bitcoin daemon.
Feb 26 23:02:13 raspberrypi systemd[1]: bitcoind.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 23:02:13 raspberrypi systemd[1]: bitcoind.service: Failed with result 'exit-code'.
Feb 26 23:02:43 raspberrypi systemd[1]: bitcoind.service: Service RestartSec=30s expired, scheduling restart.
Feb 26 23:02:43 raspberrypi systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 1684.
Feb 26 23:02:43 raspberrypi systemd[1]: Stopped Bitcoin daemon.
Feb 26 23:02:43 raspberrypi systemd[1]: Starting Bitcoin daemon...
Feb 26 23:02:44 raspberrypi systemd[1]: Started Bitcoin daemon.
1582758162 ERROR: Error found in Logs
1582758163 ERROR: Error found in Logs

*** LAST BLOCKCHAIN 20 INFO LOGS ***
sudo tail -n 20 /mnt/hdd/bitcoin/debug.log
2020-02-26T23:02:44Z Cache configuration:
2020-02-26T23:02:44Z * Using 2.0 MiB for block index database
2020-02-26T23:02:44Z * Using 63.8 MiB for transaction index database
2020-02-26T23:02:44Z * Using 8.0 MiB for chain state database
2020-02-26T23:02:44Z * Using 438.2 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2020-02-26T23:02:44Z init message: Loading block index...
2020-02-26T23:02:44Z Opening LevelDB in /mnt/hdd/bitcoin/blocks/index
2020-02-26T23:02:44Z Opened LevelDB successfully
2020-02-26T23:02:44Z Using obfuscation key for /mnt/hdd/bitcoin/blocks/index: 0000000000000000
2020-02-26T23:02:56Z LoadBlockIndexDB: last block file = 1978
2020-02-26T23:02:56Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=103, size=120842534, heights=618810...618993, time=2020-02-24...2020-02-25)
2020-02-26T23:02:56Z Checking all blk files are present...
2020-02-26T23:02:57Z Opening LevelDB in /mnt/hdd/bitcoin/chainstate
2020-02-26T23:02:57Z Opened LevelDB successfully
2020-02-26T23:02:57Z Using obfuscation key for /mnt/hdd/bitcoin/chainstate: eadc35e843c5834c
2020-02-26T23:02:57Z Loaded best chain: hashBestChain=000000000000000000083b08e2a4be8f59caad8f6bfbde20b68a2dea3ec04ca4 height=618993 date=2020-02-25T22:19:02Z progress=0.999337
2020-02-26T23:02:57Z init message: Rewinding blocks...
2020-02-26T23:02:58Z init message: Verifying blocks...
2020-02-26T23:02:58Z Verifying last 6 blocks at level 3
2020-02-26T23:02:58Z [0%]...[16%]...[33%]...[50%]...[66%]...
*** LND SYSTEMD STATUS ***
● lnd.service - LND Lightning Daemon
   Loaded: loaded (/etc/systemd/system/lnd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-02-26 23:02:57 GMT; 4s ago
  Process: 3474 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log lightning STARTED (code=exited, status=0/SUCCESS)
 Main PID: 3476 (lnd)
    Tasks: 12 (limit: 4915)
   Memory: 279.9M
   CGroup: /system.slice/lnd.service
           └─3476 /usr/local/bin/lnd --tor.active --tor.streamisolation --tor.v3 --listen=127.0.0.1:9735

Feb 26 23:02:57 raspberrypi systemd[1]: Starting LND Lightning Daemon...
Feb 26 23:02:57 raspberrypi systemd[1]: Started LND Lightning Daemon.

*** LAST LND ERROR LOGS ***
sudo journalctl -u lnd -b --no-pager -n12
-- Logs begin at Tue 2020-02-25 23:05:52 GMT, end at Wed 2020-02-26 23:03:02 GMT. --
Feb 26 23:01:30 raspberrypi systemd[1]: lnd.service: Scheduled restart job, restart counter is at 1104.
Feb 26 23:01:30 raspberrypi systemd[1]: Stopped LND Lightning Daemon.
Feb 26 23:01:52 raspberrypi systemd[1]: Starting LND Lightning Daemon...
Feb 26 23:01:52 raspberrypi systemd[1]: Started LND Lightning Daemon.
Feb 26 23:01:57 raspberrypi lnd[31973]: Unable to create chain control: unable to connect to bitcoind: -28: Loading block index...
Feb 26 23:01:57 raspberrypi systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 23:01:57 raspberrypi systemd[1]: lnd.service: Failed with result 'exit-code'.
Feb 26 23:02:57 raspberrypi systemd[1]: lnd.service: Service RestartSec=1min expired, scheduling restart.
Feb 26 23:02:57 raspberrypi systemd[1]: lnd.service: Scheduled restart job, restart counter is at 1105.
Feb 26 23:02:57 raspberrypi systemd[1]: Stopped LND Lightning Daemon.
Feb 26 23:02:57 raspberrypi systemd[1]: Starting LND Lightning Daemon...
Feb 26 23:02:57 raspberrypi systemd[1]: Started LND Lightning Daemon.

*** LAST 30 LND INFO LOGS ***
sudo tail -n 30 /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log
2020-02-26 22:59:18.823 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2020-02-26 22:59:24.333 [INF] LNWL: Opened wallet
2020-02-26 22:59:24.545 [INF] LTND: Primary chain is set to: bitcoin
2020-02-26 22:59:24.572 [ERR] LTND: Unable to create chain control: unable to connect to bitcoind: -28: Loading block index...
2020-02-26 22:59:24.577 [INF] LTND: Shutdown complete2020-02-26 23:00:24.829 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=production, logging=default
2020-02-26 23:00:24.829 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-02-26 23:00:25.250 [INF] CHDB: Checking for schema update: latest_version=12, db_version=12
2020-02-26 23:00:25.253 [INF] RPCS: password RPC server listening on 0.0.0.0:10009
2020-02-26 23:00:25.253 [INF] RPCS: password gRPC proxy started at 0.0.0.0:8080
2020-02-26 23:00:25.253 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2020-02-26 23:00:30.452 [INF] LNWL: Opened wallet
2020-02-26 23:00:30.741 [INF] LTND: Primary chain is set to: bitcoin
2020-02-26 23:00:30.765 [ERR] LTND: Unable to create chain control: unable to subscribe for zmq block events: dial tcp :28332: connect: connection refused
2020-02-26 23:00:30.770 [INF] LTND: Shutdown complete
2020-02-26 23:01:52.580 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=production, logging=default
2020-02-26 23:01:52.581 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-02-26 23:01:52.991 [INF] CHDB: Checking for schema update: latest_version=12, db_version=12
2020-02-26 23:01:52.994 [INF] RPCS: password RPC server listening on 0.0.0.0:10009
2020-02-26 23:01:52.994 [INF] RPCS: password gRPC proxy started at 0.0.0.0:8080
2020-02-26 23:01:52.995 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2020-02-26 23:01:57.125 [INF] LNWL: Opened wallet
2020-02-26 23:01:57.370 [INF] LTND: Primary chain is set to: bitcoin
2020-02-26 23:01:57.398 [ERR] LTND: Unable to create chain control: unable to connect to bitcoind: -28: Loading block index...
2020-02-26 23:01:57.402 [INF] LTND: Shutdown complete
2020-02-26 23:02:57.580 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=production, logging=default
2020-02-26 23:02:57.580 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-02-26 23:02:58.004 [INF] CHDB: Checking for schema update: latest_version=12, db_version=12
2020-02-26 23:02:58.009 [INF] RPCS: password RPC server listening on 0.0.0.0:10009
2020-02-26 23:02:58.009 [INF] RPCS: password gRPC proxy started at 0.0.0.0:8080
2020-02-26 23:02:58.009 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.

- TOUCHSCREEN is OFF by config

- Loop is OFF by config

*** LAST 20 RTL LOGS ***
sudo journalctl -u RTL -b --no-pager -n20
-- Logs begin at Tue 2020-02-25 23:05:52 GMT, end at Wed 2020-02-26 23:03:02 GMT. --
Feb 25 23:09:12 raspberrypi systemd[1]: Started RTL daemon.

- Electrum Rust Server is OFF by config

- BTCPayServer is OFF by config

- LNBits is OFF by config

*** HARDWARE TEST RESULTS ***
UndervoltageReports in Logs: 0
raspberryPi=4
powerFAIL=0
powerWARN=0
powerMIN='862500 microVolt'
tempFAIL=0
tempWARN=8
tempMAX='7200 centiGrad'
IMPORTANT: There are some hardware issues with your setup.
'Run Hardwaretest' in main menu or: sudo /home/admin/05hardwareTest.sh

*** SYSTEM STATUS (can take some seconds to gather) ***
localIP='192.168.1.105'
tempCelsius='60.3'
uptime=86199
upsStatus='OFF'
startcountBlockchain=1685
bitcoinActive=1
bitcoinErrorShort='Verifying blocks'
bitcoinErrorFull='error code: -28 error message: Verifying blocks...'
startcountLightning=1106
lndActive=1
lndRPCReady=0
walletOpened=0
walletReady=0
walletLocked=1
lndRPCReady=0
blitzTUIActive=0
blitzTUIRestarts=0
scriptRuntime=1


*** OPTION: SHARE THIS DEBUG OUTPUT ***
An easy way to share this debug output on GitHub or on a support chat
use the following command and share the resulting link:
/home/admin/XXdebugLogs.sh | nc termbin.com 9999

Any help on this would be appreciated.

PS: I am not having FanShim plugged in as that one is causing a lot of problems as it seems

[openoms]

The blockchain is corrupted on the disk:

*** ThreadSync: Failed to read block 000000008ec9525cb2092c6bd7d1c2580e595bcfa54bac51cf6f0278b242f25d from disk

Type:
repair
and Reset the Blockchain
Sync or copy it over again.

To avoid issues in the future use tested parts and do not pull the power while writing the disk.

[fifthedition]

Thank you for the lightning fast reply (pun intended).
I will try that.
I am using 7-8 year's old HDD maybe it started to fail.

[fifthedition]

So, I upgraded my old HDD to new Samsung 860 QVO 1TB SSD. I downloaded new copy of the blockchain via torrent trough raspiblitz and I got the same error.

I suspect that there is something wrong with block 73 in the blockchain downloaded via torrent.

2020-03-01T00:46:26Z ERROR: ReadBlockFromDisk(CBlock&, CBlockIndex*): GetHash() doesn't match index for CBlockIndex(pprev=0x567e230, nHeight=73, merkle=41b48c64cba68c21e0b7b37f589408823f112bb7cbccef4aece29df25347ffb4, hashBlock=000000008ec9525cb2092c6bd7d1c2580e595bcfa54bac51cf6f0278b242f25d) at FlatFilePos(nFile=0, nPos=16371)
2020-03-01T00:46:26Z *** ThreadSync: Failed to read block 000000008ec9525cb2092c6bd7d1c2580e595bcfa54bac51cf6f0278b242f25d from disk
2020-03-01T00:46:26Z Error: Error: A fatal internal error occurred, see debug.log for details

here is the full log:
https://termbin.com/qr3b

I also tried clean instal of raspiblitz and torrent blockchain download but once I enable blockchain indexing I run into the same error and block 73 is causing problems.

Anybody got the same issue?

[openoms]

Interesting, curious if anybody experiences the same.

With the SSD:

• make sure you use a tested USB adapter (for example the one form the shopping list or others listed in the issues)
• the RPi4 can sync the blockchain from the bitcoin network on 2-4 days so would recommend not using the Torrent option.

[rootzoll]

You have 8 temp warnings on your RaspiBlitz running at 72 Grad. This may impact the stability of your RaspiBlitz. Do you have a fan for cooling like adviced in the shopping list?

[rootzoll]

Just saw that you removed the shim fan. Which country are you living in ... I would recommend checking your amazon if you can order one of those new LCD-Hats with a fan on the back. They seem to work OK ... you could help testing them and let us know your local shopping link.

[nielsoo]

Hi All

I have the same issue, with a freshly bought (official) raspiblitz.

the INTERESTING part is that the error I have happens to be on the EXACT same block

2020-06-23T10:55:57Z ERROR: ReadBlockFromDisk(CBlock&, CBlockIndex): GetHash() doesn't match index for CBlockIndex(pprev=0x5a6def8, nHeight=73, merkle=41b48c64cba68c21e0b7b37f589408823f112bb7cbccef4aece29df25347ffb4, hashBlock=000000008ec9525cb2092c6bd7d1c2580e595bcfa54bac51cf6f0278b242f25d) at FlatFilePos(nFile=0, nPos=16371)
2020-06-23T10:55:57Z ** ThreadSync: Failed to read block 000000008ec9525cb2092c6bd7d1c2580e595bcfa54bac51cf6f0278b242f25d from disk
2020-06-23T10:55:57Z Error: Error: A fatal internal error occurred, see debug.log for details
2020-06-23T10:55:57Z txindex thread exit

so this most likely isn't any "hardware" issue like some people suggest here :-)

[rootzoll]

@nielsoo did you used the prepared blockchain that came with the RaspiBlitz?

[nielsoo]

Yes I did, but I have deleted it now and am now syncing from scratch (I am syncing from bitcoin network) I will let you know if the issue returns or not.

Op di 23 jun. 2020 17:27 schreef Christian Rotzoll <[email protected]

:

@nielsoo https://github.com/nielsoo did you used the prepared
blockchain that came with the RaspiBlitz?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rootzoll/raspiblitz/issues/1054#issuecomment-648237491,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABM3L7BJGSJ5MRSC2VZUUVDRYDCV5ANCNFSM4K4PWWDA
.

[Animalmother9000]

Hi Nielsoo,

when did you bought your RaspiBlitz?

[nielsoo]

ordered may 21th arrived june 4th

[nielsoo]

Blockchain synced and up and running for a few days now without issues.. so it just seemed to have been a bad block in the shipped presynced blockchain :-)

[notverygoodatcoding]

Snap. Same error, same block (73) after torrenting. Just lettting you know.

[rootzoll]

@notverygoodatcoding using the torrent is not recommended anymore (removed in latest version). If you haev RP4 then let sync by itself of you have an old RP3 then sync the Blockchain with a Bitcoin Core on your laptop and then use COPY option over LAN.

[notverygoodatcoding]

Thanks for the response, and great project by the way. I've had loads of fun messing around with my node for the past three months and feel like I've learnt a lot. Currently syncing the chain whilst growing my facial hair.......Should be synced and fully bearded any day now......