Raspiblitz: RESEARCH: Booting from HDD/SSD drive

[fluidvoice]

experimenting with booting an (unmodified) RPi 3B+ from USB with no SD-card present.
First attempt did not work with Etcher burned Raspiblitz image and also using dd
Then I tried the full Raspbian Desktop image burned with Etcher and it worked.
So certainly it's possible to boot from USB. I'm gonna have to look into more about how @rootzoll is building the Raspiblitz image (build_sdcard.sh). Anyone can help, know why this is?

[openoms]

'To enable the USB boot bit, the Raspberry Pi 3 needs to be booted from an SD card with a config option to enable USB boot mode.

Once this bit has been set, the SD card is no longer required. Note that any change you make to the OTP is permanent and cannot be undone.'

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md

[fluidvoice]

'To enable the USB boot bit, the Raspberry Pi 3 needs to be booted from an SD card with a config option to enable USB boot mode.

Once this bit has been set, the SD card is no longer required. Note that any change you make to the OTP is permanent and cannot be undone.'

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md

@openoms You don't have to do this for the Pi 3B+ and later Pi's the bit is set when it's made (factory set). That's why, as I said in my post, that my unmodified Pi boots the stock Raspbian from USB OK.

from that link... "The Raspberry Pi 3+ is able to boot from USB without any changes, but the Raspberry Pi 3 requires the USB boot bit to be set in the OTP (one-time programmmble) memory."

[openoms]

ok, that is right. Most of the issues seems to be sorted on the 3+. Still it is possible that some USB drives don't work as expected as it is described in this blog post linked from the one above: https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/

[fluidvoice]

ok, that is right. Most of the issues seems to be sorted on the 3+. Still it is possible that some USB drives don't work as expected as it is described in this blog post linked from the one above: https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/

Thanks for the link. Might refer to it if I have problems. But the article is from 2016 so it prob is of little help since Raspbian and potentially many USB drives may have changed quite a bit since then.

[fluidvoice]

Things learned so far:

1. Quality of USB-flash drives just as important as SD cards. One attempt at running the build_sdcard.sh with a (32GB Kingston DT50 $6) showed extremely slow write speed that took forever, the drive ran extremely hot, and then near the end it failed. I think slow write speed is the sign of a low quality device. This drive has advertised ~100MBps read but only 15MBps write speed, and when it gets hot it falls even farther to ~5MBps. Look for a drive with good write speed and a reputation for endurance. Second attempt using 32GB Sandisk Ultra Dual, which is also not really a high quality drive but has much better write speed succeeded.

1. When build_sdcard.sh succeeded then after reboot the system hung and would not complete bootup. So debugging needed to figure out why it's not working. Something the script is changing/updating is breaking it.

2. The bootup stops until you connect KB/mouse then progresses further. There is a fix for this, will add later.

3. then the bootup progresses further until it detects the drive, in this case sda then stops/freezes. Perhaps it needs some fstab sorta config. Need to look into it.

[fluidvoice]

Some Bootup info: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow.md

It looks most likely this is what I'm missing. Will have to try this:

To get the root OS running off the SSD on any model Pi you only need to edit 2 files. You need to edit /boot/cmdline.txt on the SD card and replace the PARTUUID of the SD card with that of the SSD (2nd partition), and you need to edit the /etc/fstab file on the SSD drive and replace the root mount of the 2nd SD card partition (/dev/mmcblk0p2) with the same PARUUID you put into cmdline.txt.

To get your PARTUUID, enter the following command into a terminal:
ls -l /dev/disk/by-partuuid/

/boot/cmdline.txt changed (root=/dev/mmcblk0p2 to root=PARTUUID=7d832c35-02 ) so it looks like this...

dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 root=PARTUUID=7d832c35-02 rootfstype=ext4 elevator=deadline rootwait fbcon=map:10 fbcon=font:ProFont6x11 logo.nologo

/etc/fstab required no changes. Looks like this...

proc /proc proc defaults 0 0
PARTUUID=7d832c35-01 /boot vfat defaults 0 2
PARTUUID=7d832c35-02 / ext4 defaults,noatime 0 1

This also needs testing:

I tried pure USB boot and found it problematic. Power-on boot worked fine, but restart and shutdown were unreliable (worked sometimes and failed others).

And see if there is any workaround for this:

SD boot is faster than pure USB boot due to the delay of checking for an SD card before booting from USB.

[fluidvoice]

That worked. It booted up and LCD now says

HDD needs SetUp
and
Login to your RaspiBlitz with:
SSH [email protected]
Use password: raspiblitz

after doing the above, the config looks like this:
admin@RaspiBlitz:~ $ ls -l /dev/disk/by-partuuid
total 0
lrwxrwxrwx 1 root root 10 Mar 19 01:19 7d832c35-01 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 19 01:19 7d832c35-02 -> ../../sda2
admin@RaspiBlitz:~ $ sudo grep PARTUUID= /etc/fstab
PARTUUID=7d832c35-01 /boot vfat defaults 0 2
PARTUUID=7d832c35-02 / ext4 defaults,noatime 0 1
admin@RaspiBlitz:~ $ grep "root=" /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 root=PARTUUID=7d832c35-02 rootfstype=ext4 elevator=deadline rootwait fbcon=map:10 fbcon=font:ProFont6x11 logo.nologo

(for Pi boards older than the 3B+) run:
admin@RaspiBlitz:~ $ sudo vcgencmd otp_dump | grep 17:
17:3020000a

If you get a response of 17:3020000a then your USB boot bit has already been set. If you have a 1 where the 3 is, then you'll need to program that bit. To do that add program_usb_boot_mode=1 to the end of your config.txt file (in the boot partition) then reboot your Pi and check the bit again.

Partitions with HDD not connected yet...
admin@RaspiBlitz:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 28.7G 0 disk
├─sda1 8:1 1 43.9M 0 part /boot
└─sda2 8:2 1 28.6G 0 part /

[fluidvoice]

FYI Just ran @openoms DietPi build script on a USB flash drive and it boots with the same minor change to the /boot/cmdline.txt file: https://github.com/rootzoll/raspiblitz/issues/244#issuecomment-477415392

[fluidvoice]

Booted Raspiblitz from USB via DietPi build script (https://github.com/rootzoll/raspiblitz/issues/244#issuecomment-477580329) it seems perhaps it did find the blockchain DB on /dev/sda1 (HDD) and booted from /dev/sdb1 ...

admin@DietPi:/home/bitcoin/.bitcoin $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 341.2G 0 part /mnt/hdd
├─sda2 8:2 0 1000M 0 part
├─sda3 8:3 0 588.4G 0 part
└─sda4 8:4 0 999M 0 part
sdb 8:16 1 28.7G 0 disk
├─sdb1 8:17 1 43.9M 0 part /boot
└─sdb2 8:18 1 28.6G 0 part /
admin@DietPi:/home/bitcoin/.bitcoin $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.3G 24G 13% /
devtmpfs 460M 0 460M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 12M 452M 3% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
tmpfs 1023M 0 1023M 0% /tmp
tmpfs 10M 1.4M 8.7M 14% /DietPi
tmpfs 50M 20K 50M 1% /var/log
/dev/sdb1 44M 24M 20M 54% /boot
/dev/sda1 335G 273G 46G 86% /mnt/hdd

[openoms]

Booted Raspiblitz from USB via DietPi build script ( #244 (comment) ) it seems perhaps it did find the blockchain DB on /dev/sda1 (HDD) and booted from /dev/sdb1

I wonder what governs which device will become sda or sdb? The SD at least is reliably distinct.

It supposed to depend on the physical address of the device so I presume if you swap the usb ports the HDD and USBdrive is plugged in, they can change places being sda or sdb? Need to test this.

More on this:
We could try differentiating the disks by label instead of device name (sdx) . Like the HDD partition with the blockchain has the BLOCKCHAIN label fixed already.
https://debian-administration.org/article/522/Mounting_file-systems_by_label_rather_than_device_name

[fluidvoice]

Booted Raspiblitz from USB via DietPi build script ( #244 (comment) ) it seems perhaps it did find the blockchain DB on /dev/sda1 (HDD) and booted from /dev/sdb1

I wonder what governs which device will become sda or sdb? The SD at least is reliably distinct.

Good question.
I might try it. But I think it might be timing related. Will google it n see if there is any info on it.

More on this:
We could try differentiating the disks by label instead of device name (sdx) . Like the HDD partition with the blockchain has the BLOCKCHAIN label fixed already.

Yes, the script already looks for this label and is how it determines it. I remember reading it is stated that the drive must be formatted Ext4 and labelled "BLOCKCHAIN".
https://github.com/rootzoll/raspiblitz/blob/ff3d36b92c719809426c159cc1e8ee5c4838f31d/home.admin/40addHDD.sh#L23

https://debian-administration.org/article/522/Mounting_file-systems_by_label_rather_than_device_name

[fluidvoice]

I wonder what governs which device will become sda or sdb? The SD at least is reliably distinct.

@openoms maybe the sda/sdb device assignment has to do with entries in /etc/fstab ; perhaps it's OS dependant and assigns sda/sdb via PARTUUID's before UUID's? anyway, I think it doesn't matter given the scripts are looking for the BLOCKCHAIN label.

my fstab:

UUID=8ece4d87-2aef-4566-b108-4cc517b89d1a /mnt/hdd ext4 noexec,defaults 0 2
PARTUUID=b8ef7a27-02 / auto defaults,noatime,rw 0 1
PARTUUID=b8ef7a27-01 /boot auto defaults,noatime,rw 0 1

[adevoss]

To boot from USB I had to edit _bootstrap.sh. The HDD search is hardcoded /dev/sda.
I made an edit hddDeviceName="sdb1" and
comment out the resize rootfs part in 90finishSetup.sh

Can boot from USB be a cause of #1070 ?

[adevoss]

When updating to v1.5 I got it working with USBdrive, no SDcard.
If it's helpful I can post the steps I took.

[adevoss]

maybe the sda/sdb device assignment has to do with entries in /etc/fstab ; perhaps it's OS dependant and assigns sda/sdb

afaik this is done by the linux udev system.
One of my steps to use an USBdrive is linking /dev/sdb to HDD serial number in a custom udev rule.
That way USB is always /dev/sda and HDD always /dev/sdb.

[rootzoll]

@Arno-GitHub yes a small in detail description what is needed to do that would be interesting :)

[adevoss]

The script /home/admin/config.scripts/blitz.datadrive.sh makes the Pi unbootable when /etc/fstab is changed. The root partition entry is deleted.
This can be fixed by restoring the original /etc/fstab. No need to write image again.

This is what I did to use USB drive instead of SD Card:

Summary:
Edit /boot/config.txt as mentioned before.
Use the correct /etc/fstab to (not) mount HDD.

Full description:

Hardware:
Set OTP bit (Please search internet for howto)

USB drive:
Write image to USB drive and resize root partition to use drive at full capacity. (Please search internet for howto)

Raspbian:
/boot/cmdline.txt:
Change root=/dev/sda2

(only if HDD is reconized as /dev/sda)
Edit: This step is not to much use because for it to take effect the OS has to be started and so the HDD already is not the boot device.
/etc/udev/rules.d/60-datadisk.rules:
create udev rule for datadisk to be /dev/sdb not /dev/sda (boot device)
KERNEL=="sd*[0-9]", ENV{ID_SERIAL_SHORT}=="[S/N]", SYMLINK+="sdb"
Replace [S/N] with serial number of HDD.

Raspiblitz:

create /etc/fstab.unmounted with USB as boot/root partition

proc            /proc           proc    defaults          0       0
/dev/sda1  /boot           vfat    defaults          0       2
/dev/sda2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

create /etc/fstab.mounted with HDD entry

proc            /proc           proc    defaults          0       0
/dev/sda1  /boot           vfat    defaults          0       2
/dev/sda2  /               ext4    defaults,noatime  0       1
/dev/sdb1  /mnt/hdd        ext4    defaults,noexec  0       2
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

cp -av /etc/fstab /etc/fstab.bak
cp -av /etc/fstab.unmounted /etc/fstab

(tested only for ext4)

/home/admin/config.scripts/blitz.datadrive.sh

Line 583: cp -av /etc/fstab.mounted /etc/fstab
(Just before the line: "... fstab updated for EXT4 layout ... "

Edit: Formatting

[adevoss]

Replacing my Pi3B with 4B.
With the steps I described above I couldn't get the Pi to boot.
The Pi doesn't boot with USB drive as /dev/sda AND the SSD connected.
After much trial and error my new approach is:

Put image file on USB drive (2020-05-27-raspios-buster-lite-armhf.img)
Resize root partition
Activate ssh
Optional: Static IP address

/boot/cmdline.txt:
Change root=/dev/sdb2

sudo ./build_sdcard.sh (modified version)

The script runs without errors.
Next step is to reboot and see if Raspiblitz is being installed with data on the SSD.

[adevoss]

/home/admin/config.scripts/blitz.datadrive.sh

Line 583: cp -av /etc/fstab.mounted /etc/fstab
(Just before the line: "... fstab updated for EXT4 layout ... "

This is still needed for v1.6
Edit: The line above should have been: This is an issue when testing with v1.6

[rootzoll]

@adevoss what is needed for v1.6? .. to have booting from SSD will most prob come with a future version, but we will not be able to put it into v1.6

[adevoss]

@rootzoll I didn't mean to say this has to be fixed in v1.6.
Also this is not about booting from SSD but from a USB stick. USB stick as boot device and SSD drive as datadrive.

What I'm trying to say is when testing with v1.6 and blitz.datadrive.sh makes the mount permanent in /etc/fstab it results in a /etc/fstab file that makes the Pi unbootable.

I didn't save /etc/fstab but I know that the root partition entry was missing and there was a line with only a disk id.
So I created /etc/fstab.mounted. In this file I put back the root partition and the SSD datadisk is mounted at /mnt/hdd.
By adding the extra line in blitz.datadrive.sh fstab is replaced and the Pi boots after the next reboot.

It's a quick fix I know. When my node is running again I will see if I can write a more permanent solution to fix this issue.

[adevoss]

The script runs without errors.
Next step is to reboot and see if Raspiblitz is being installed with data on the SSD.

After fixing /etc/fstab the setup process finished with a few errors:

• TimeOuts when installing BTCPay server (can't find this in logs)

• npm WARN tarball tarball data for [email protected] (sha1-mnHEh0chjrylHlGmbaaCA4zct78=) seems to be corrupted. Trying one more time.

• errors like this:
  /home/btcrpcexplorer/.cache/node-gyp/12.16.3/include/node/node.h:642:3: note: in expansion of macro ‘NODE_MODULE_X’
  NODE_MODULE_X(modname, regfunc, NULL, 0) // NOLINT (readability/null_usage)
  ^~~~~

• https://localip:3001
  502 Bad Gateway

These errors are not related to USB booting so I think USB booting is possible with a few extra steps.

Hardware: Pi 4B 2GB, no LCD, wifi diabled and USB and SSD no SDcard
Base image: PiOS Buster Lite
Raspiblitz: v1.6.0rc1
Disk space used: 4,7 GB

Installed:
rtl
lightning loop
btc explorer
btcpay server

[adevoss]

Reading https://github.com/rootzoll/raspiblitz/issues/1285#issuecomment-651017746 (dying USB thumb drive) the logs can be held in RAM at the own risk of the user. Logs are gone after reboot.

The first reason for me to use an USB thumb drive are the wear of the SD card. This maybe is not solved by the USB thumb drive.
The second reason for me is a practical one: The physical size of the SD card. Easy to loose and difficult to access with some cases. At least the case I'm using for another Pi which I use as a desktop.

Any thoughts on using a thumb drive to boot by anyone?

[rootzoll]

@adevoss I dont think a thumb drive for booting or running the OS is an option. Flashing a sd card is a standard practice when operating a RaspberryPi. Most cases give easy access to the sd card.

For the future still its the idea to just run the OS the first time (or on update) from sd card and then use an extra partition of the HDD/SSD to run the OS ... but more research and tests needed targeting in the area of a 1.7 or 1.8 release later this year.

[adevoss]

@rootzoll Correct. At this moment another USB thumb drive went read only. I stop using a thumb drive for the OS.

[openoms]

At this moment another USB thumb drive went read only. I stop using a thumb drive for the OS.

re reliability of thumbdrives please see this issue: https://github.com/rootzoll/raspiblitz/issues/924

[rootzoll]

I renamed the research issue. Because I think the goal to aim for should be:

• on setup boot from sd card (that may have an additional image file stored)
• the HDD/SSD will be setup with a boot partition and the copy the OS over to that HDD/SSD partition (from image file)
• now the RaspiBlitz will boot and run OS from HDD/SSD
• if the user inserts a sd card with a fresh image it should refresh the HDD/SSD partition and go into recover/update

I am still not sure if this is possible ... @adevoss do you see any problems in this setup?

Open Questions:

• What to do with SSD/HDDs on update that are already setup .. can we create a boot partition with no problem on a HDD/SSD that alread has data on it?
• How does the system detect a fresh SD card entered for for update & recovery? Will the HDD/SSD boot detect a new card and will switch back to sd card boot? Can we set a "boot priority like sd card first over HDD/SSD" and after setup we make the boot partition on the ols sd card unvalid?

[adevoss]

I am still not sure if this is possible ...

I think it is. Didn't try to boot from SSD but why should it be different than from an thumbdrive. I don't have a spare SSD (only one for my live node)
Put the OS on the first partition and the ext4 data on the second partition. No experience with BTRFS.
(Edit: should of course be boot/root/data partition)

What to do with SSD/HDDs on update that are already setup

Just overwrite the OS partition with a new one. That is the same what is advised for an update now, isn't it?

How does the system detect a fresh SD card

The bootloader has a boot_order. (https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md see BOOT_ORDER)
Default on a Pi 4B is MSD first then SD card. MSD stands for Mass Storage Device.
On a Pi 4B you have to disconnect the SSD before boot or change the boot order (possible but not easy imo)
On a PI 3B the boot order is reversed. The result is the same. If SD card is present at boot it boots from there.

So booting from SD card is an update. Booting without SD card, no update.

[rootzoll]

Sounds good :)

So booting from SD card is an update. Booting without SD card, no update.

That can be a fallback .. but I would even like if people just keep the sd card in the RaspiBlitz. That would just be confinient (you dont have to search for the sd card, etc) and it can be used as a medium for critical backup data (storing the SCB copy for example).

[rootzoll]

I am moving this issue to the v1.7 milestone as a research topic. Because focus for the v1.7 is a refactor of the setup process - so it makes sense to also take a look at HDD/SSD booting in that regard.

[adevoss]

That can be a fallback .. but I would even like if people just keep the sd card in the RaspiBlitz.

Can be done with Pi 4B. The boot order is SSD then SD card.
With Pi 3B the update can be on the SD card. Only when booting it has to be removed.

How does the system detect a fresh SD card entered for for update & recovery?

Can be done with cron job.
When file x exists on SD card then an update has to be done. And test that every x minutes.