DietPi-Drive_Manager | Rewrite/improve

Created on 19 Jun 2018 · 36Comments · Source: MichaIng/DietPi

Required Information:

DietPi version | 6.9
Distro version | 9.4
Kernel version | 4.14.32+
SBC device | Odroid HC2
Power supply used | 12V 2A
SDcard used | SanDisk ultra 16gb

Additional Information:

Nextcloud, Ampache, Webmin,NoIP,failfoban,certbot
- Fresh Install
- dietpi-bugreport ID = 979b351b-9ad4-440e-8621-6ae5ed95ffe8

Steps to reproduce:

Fresh install
drivemanager-> no hard drive visible
manuell formating + mount -> still not visible

Expected behaviour:

Option to use the harddrive as useredata
option to mount harddrive

Actual behaviour:

no option/way to use the hardrive

Extra details:

Unfortunately, I can't use the hard disk on my odroid hc2 (wdRed 2TB). I did a fresh install and i want to use the harddrive(2tb) for my userdata (nextcloud, ampache...)

During isntall i cant choose the drive for my useredata and i cant mount it with the drive'_manager.
After the first install i tried to format the drive and mount it manually.

Unfortunatelly i still cant use the drive for my userdata. The Strange thing is, that the drive_manager says that the user data is on sda1 and on mmcblk1p1.

which irritates me too is that sda1 via drive_manager only is 15Gb big? lsbkid shows me the 1.8tb drive as sda1??

drive_manager shows this:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
└─sda1 8:1 0 1.8T 0 part /datastore
mmcblk1 179:0 0 14.9G 0 disk
├─mmcblk1p1 179:1 0 71M 0 part /boot
└─mmcblk1p2 179:2 0 14.8G 0 part /
CODE: SELECT ALL

nano /etc/fstab

Samba Client------------------------------------------------------

/mnt/samba . Please use dietpi-config and the Networking Options: NAS menu to setup this mount

FTP Client Mount--------------------------------------------------

/mnt/ftp_client . Please use dietpi-config and the Networking Options: NAS menu to setup this mount

NFS Client Mount--------------------------------------------------

/mnt/nfs_client . Please use dietpi-config and the Networking Options: NAS menu to setup this mount

TMPFS / MISC ------------------------------------------------------

proc /proc proc defaults 0 0
tmpfs /tmp tmpfs defaults,size=1023M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs defaults,size=20m,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /DietPi tmpfs defaults,size=10m,noatime,nodev,nosuid,mode=1777 0 0

Internal Drives---------------------------------------------------

UUID=af314fbe-6db6-4970-84b4-19fd22bd6bf7 / auto defaults,noatime 0 0
UUID=7451-4AA3 /boot auto defaults,noatime 0 0
UUID=cce05d0d-6515-497e-a473-b5660d25ab86 / auto defaults,noatime 0 0
/var/swap none swap sw 0 0
UUID="cce05d0d-6515-497e-a473-b5660d25ab86" /datastore ext4 defaults 0 0

Unfortunately I am not a Linux expert and would be very happy about simply commands

I really hope you can help me. I love dietpi!!!

Bug Priority High

Source

butterbrot21

Most helpful comment

Working on a rewrite, should be done in a few days. Now support network mounts aswell.

Fourdee on 21 Jun 2018

👍2 ❤1

All 36 comments

I can replicate this but I can manually mount a drive.

LexiconCode on 19 Jun 2018

Sounds good can you help me with wth this?
After manually mount, can you see the drive in the drive_manager?
If so is it possible to use it as userdata?

this is what i did:
mkdir /datastore
mount /dev/sda1 /datastore
blkid /dev/sda1
->UUID="cce05d0d-6515-497e-a473-b5660d25ab86"
nano /etc/fstab
-->UUID=cce05d0d-6515-497e-a473-b5660d25ab86 /datastore ext4 defaults 0 0

now i see the drive but i cant use it for my userdata:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
└─sda1 8:1 0 1.8T 0 part /datastore
mmcblk1 179:0 0 14.9G 0 disk
├─mmcblk1p1 179:1 0 71M 0 part /boot
└─mmcblk1p2 179:2 0 14.8G 0 part /
CODE: SELECT ALL

drive_manager shows this:
─(/dev/sda)────────────────────────────────────────
│ / : /dev/sda1 | ext4 | Size: 15G | Used: 1.9G (14%)
│ ─(/dev/mmcblk1)────────────────────────────────────────
│ /boot : /dev/mmcblk1p1 | vfat | Size: 71M | Used: 32M (45%)
│ / : /dev/mmcblk1p2 | ext4 | Size: 15G | Used: 1.9G (14%)
│ ───────────────────────────────────────────────────────
--> here sda1 is only 15G same as mmcblk1p2
-both mmcblk1p2 and sda1 shows me the x for userdata, is this normal?

Is there a other way to mount the drive an use it for userdata?
Thanks for the response and help!

butterbrot21 on 19 Jun 2018

@butterbrot21

Thanks for the report 👍

Appears /dev/sda1 is being incorrectly identified as rootfs

│ / : /dev/sda1 | ext4 | Size: 15G | Used: 1.9G (14%)
│ / : /dev/mmcblk1p2 | ext4 | Size: 15G | Used: 1.9G (14%)

#ref
UUID=cce05d0d-6515-497e-a473-b5660d25ab86 / auto defaults,noatime 0 0
/var/swap none swap sw 0 0
UUID="cce05d0d-6515-497e-a473-b5660d25ab86" /datastore ext4 defaults 0 0

I'll run some local tests, see if we can replicate.

@butterbrot21 @LexiconCode
Was this on a fresh DietPi image install, or after running the dietpi-update ?

Fourdee on 20 Jun 2018

Hey,

Yes it was on a fresh install.

It looked as if it had automatically run an update after first login via putty. But i dont run dietpi-update.
After install i did the normal setup with selected software.

One time i tied to format the disk and mount it manually before the first setup/inmstaller.
Unfortunately without success.

i hope i can help I'm really looking forward to use dietpi on my odroid hc2 :)

Keep up the good work 👍

butterbrot21 on 20 Jun 2018

🈯️ https://github.com/Fourdee/DietPi/commit/8f053fdb26c00c34ddc988a649cdcaef409dad85 >> Another bug with incorrect cat listing:

Fourdee on 20 Jun 2018

Notes:

Need to improve detection of currently mounted drives, regardless of fstab entries
We cant use mount for current mount checks, as a manually added mount on same drive shows up 2 results:

/dev/sda1 /mnt/99dcbd13-9812-4949-87da-7420f2fd1489 ext4 rw,noatime,data=ordered 0 0
/dev/sda1 /datastore ext4 rw,relatime,data=ordered 0 0

Yet, df result shows the "latest" one:

/dev/sda1         14712712 40988  14655340       1% /datastore

df faster than lsblk

root@DietPi:~# time lsblk /dev/sda1 | awk '{print $NF}'
MOUNTPOINT
/datastore

real    0m0.024s
user    0m0.008s
sys     0m0.015s
root@DietPi:~# time df -P /dev/sda1 | awk '{print $NF}'
on
/datastore

real    0m0.012s
user    0m0.003s
sys     0m0.013s

Fourdee on 20 Jun 2018

in another try/reinstal, I tried to format sda after the first setup/software-install and mount it afterwords, becaouse it looked like that the userdata was on sda and on sd:

Unfortunately without success. After a reboot I could not access via putty anymore^^
I think the error message was " ssh [email protected] .... declined"
Maybee this info can help a little bit.
I'm not an expert with linux and programming, but I try to learn quickly :)

I am very happy and excited that you look at the issue for the hc2 👍

butterbrot21 on 20 Jun 2018

Notes to self, detection rewrite:

Networked drives add/remove?
RootFS transfer needs redoing

Fourdee on 20 Jun 2018

Working on a rewrite, should be done in a few days. Now support network mounts aswell.

Fourdee on 21 Jun 2018

👍2 ❤1

Sounds great! Thanks for the fast response and fix!
Keep up the good work 👍

butterbrot21 on 21 Jun 2018

ToDo:

RootFS transfer
Implement FTP/NFS + Custom mount location options (+samba)

Fourdee on 22 Jun 2018

Fourdee on 23 Jun 2018

@Fourdee
As you already touch this. I have an issue with drive manager currently not recognizing my two drives, that are attached via HDD docking station:

fdisk -l

Disk /dev/mmcblk0: 59.8 GiB, 64223182848 bytes, 125435904 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa277a2ba

Device         Boot Start       End   Sectors  Size Id Type
/dev/mmcblk0p1       8192     96453     88262 43.1M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      98304 125435903 125337600 59.8G 83 Linux

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

mount

/dev/sda on /mnt/sda type ext4 (rw,noatime,data=ordered,x-systemd.automount)

df

/dev/sda 2884154032 226415000 2511209320 9% /mnt/sda

lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  2.7T  0 disk /mnt/sda
sdb           8:16   0  1.8T  0 disk
mmcblk0     179:0    0 59.8G  0 disk
├─mmcblk0p1 179:1    0 43.1M  0 part /boot
└─mmcblk0p2 179:2    0 59.8G  0 part /

/dev/sda is my data drive, /dev/sdb my backup drive, thus only mounted once at night for backup.
Don't ask my why their dev name does not contain any partition number. Of course they just have one partition, ones formated via Raspbian.
Docking station: Inateck FD2003B with two docks

The two docks already needed some Linux/Debian feature to be recognized correctly, that came first with Jessie, if I remember right. Before that it was not able to connect two separate drives via single USB port or something like that. Can't remember any more exactly, but I guess there might be a connection with the strange dev naming.

That time I wrote down the following steps to create the HDD file systems, thus there was no partition number since the first time I attached them:

    fdisk -l => sda
    mkfs.ext4 /dev/sda
    mkdir /mnt/sda
    mount /dev/sda /mnt/sda

The problem I guess is, how the check if the /dev/sda is already the partition/fs, or if it's just the raw device, where partitions are needed, especially if it's not mounted (otherwise mount would work well as check). fdisk -l does not have any drive or partition info about them at all.

I remember some DietPi versions ago, they WERE recognized! I know this because I always needed to adjust the mount points from /dev/UUID to /mnt/sda|sdb again, as these locations are used within some scripts, Nextcloud etc 😆.

MichaIng on 23 Jun 2018

👍1

@MichaIng

no partition number detection appears to be working for me with latest testing version:

Fourdee on 23 Jun 2018

👍1

@Fourdee
Ah great, nice rework btw., drive_manager detection speed is impressively improved!

Do you know about the condition for no partition number? When I use USB stick on RPi or VM, even with just 1 partition, the drive to mount always has a partition number 1 and the /dev/sdX without partition number exist besides, but itself is not mountable.

MichaIng on 24 Jun 2018

Note to self:
We can use to detect unformatted connected drives:
Check block devices against blkid

root@DietPi:~# ls -lha /sys/block/
loop0/   loop4/   mmcblk0/ ram11/   ram15/   ram5/    ram9/
loop1/   loop5/   ram0/    ram12/   ram2/    ram6/    sda/
loop2/   loop6/   ram1/    ram13/   ram3/    ram7/
loop3/   loop7/   ram10/   ram14/   ram4/    ram8/

root@DietPi:~# blkid
/dev/mmcblk0p1: LABEL="boot" UUID="5DB0-971B" TYPE="vfat" PARTUUID="27504eef-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="060b57a8-62bd-4d48-a471-0d28466d1fbb" TYPE="ext4" PARTUUID="27504eef-02"
/dev/mmcblk0: PTUUID="27504eef" PTTYPE="dos"

Do you know about the condition for no partition number? When I use USB stick on RPi or VM, even with just 1 partition, the drive to mount always has a partition number 1 and the /dev/sdX without partition number exist besides, but itself is not mountable.

Unsure, I believe it depends if a partition table is created or not.

Fourdee on 24 Jun 2018

Hmmm:

root@DietPi:~# blkid
/dev/mmcblk0p1: LABEL="boot" UUID="5DB0-971B" TYPE="vfat" PARTUUID="27504eef-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="060b57a8-62bd-4d48-a471-0d28466d1fbb" TYPE="ext4" PARTUUID="27504eef-02"
/dev/mmcblk0: PTUUID="27504eef" PTTYPE="dos"
root@DietPi:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    1 14.9G  0 disk
mmcblk0     179:0    0 14.9G  0 disk
├─mmcblk0p1 179:1    0 41.8M  0 part /boot
└─mmcblk0p2 179:2    0 14.8G  0 part /
root@DietPi:~# lsblk^C
root@DietPi:~# parted /dev/sda
Error: Could not stat device /dev/sda - No such file or directory.
Retry/Cancel? c
root@DietPi:~# parted /dev/sda1
Error: Could not stat device /dev/sda1 - No such file or directory.
Retry/Cancel?

dd if=/dev/zero of=/dev/sda bs=4K count=1000
parted
mklabel msdos
#MUST REBOOT HERE system not updating partitions....... even partprobe does nothing...
mkpart primary ext4 0% 100%

Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? i
(parted) p
Model:  (file)
Disk /dev/sda: 4096kB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start  End     Size    Type     File system  Flags
 1      512B   4096kB  4095kB  primary  ext4         lba


---
root@DietPi:~# mkfs.ext4 -F -m 0 /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
The file /dev/sda1 does not exist and no size was specified.
root@DietPi:~# mkfs.ext4 -F -m 0 /dev/sda
mke2fs 1.43.4 (31-Jan-2017)
Found a dos partition table in /dev/sda
Discarding device blocks: done
Creating filesystem with 4000 1k blocks and 1000 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

root@DietPi:~# resize2fs /dev/sda
resize2fs 1.43.4 (31-Jan-2017)
The filesystem is already 4000 (1k) blocks long.  Nothing to do!

root@DietPi:~# blkid
/dev/mmcblk0p1: LABEL="boot" UUID="5DB0-971B" TYPE="vfat" PARTUUID="27504eef-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="060b57a8-62bd-4d48-a471-0d28466d1fbb" TYPE="ext4" PARTUUID="27504eef-02"
/dev/mmcblk0: PTUUID="27504eef" PTTYPE="dos"
root@DietPi:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    1 14.9G  0 disk
mmcblk0     179:0    0 14.9G  0 disk
├─mmcblk0p1 179:1    0 41.8M  0 part /boot
└─mmcblk0p2 179:2    0 14.8G  0 part /
root@DietPi:~# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model:  (file)
Disk /dev/sda: 4096kB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  4096kB  4096kB  ext4

Fourdee on 24 Jun 2018

@Fourdee

About missing fs table: As I just used mkfs command that time, where drive_manager uses parted to create fs table first (?). Might this be the reason? I am wondering if then fs table is not needed or can be skipped if just one partition will be in place? Or is there any risk without or benefit with fs table first?
As you are anyway working on drive_manager, I just want to remind our opened tasks for this:
- Allow fs check on reboot for root and boot partition. However, we did not yet patch the needed fstab flag: https://github.com/Fourdee/DietPi/commit/0eb0db0209985ff91fcd425bbd4ca4a4da6626d8
- Expand FS manually: https://github.com/Fourdee/DietPi/issues/1821
- But you can also leave this to me, just wanted to collect here.

MichaIng on 25 Jun 2018

@MichaIng

Allow fs check on reboot for root and boot partition. However, we did not yet patch the needed fstab flag: 0eb0db0

Let me send commits 1st, then its open for implementation.

Expand FS manually: #1821

Done, just doing last testing.

Hmm..

root@DietPi:~# dd if=/dev/zero of=/dev/sda bs=4K
dd: error writing '/dev/sda': No space left on device
184628+0 records in
184627+0 records out
756232192 bytes (756 MB, 721 MiB) copied, 1.56076 s, 485 MB/s
root@DietPi:~# partprobe /dev/sda
root@DietPi:~# dd if=/dev/zero of=/dev/sda bs=4K
dd: error writing '/dev/sda': No space left on device
184628+0 records in
184627+0 records out
756232192 bytes (756 MB, 721 MiB) copied, 1.55418 s, 487 MB/s
root@DietPi:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    1  14.3G  0 disk
mmcblk0     179:0    0 116.5G  0 disk
├─mmcblk0p1 179:1    0   128M  0 part /boot
└─mmcblk0p2 179:2    0 116.4G  0 part /

Fourdee on 26 Jun 2018

🈯️ Handled in script with: https://github.com/Fourdee/DietPi/commit/c06fe531818d5ebb6b2c2f63af45dea9d3641229

How to test no partition table formatting:

PREP: Clear any tables, then reboot:

dd if=/dev/zero of=/dev/sda bs=4K count=1337
reboot

Run dietpi-drive_manager and format drive
When the error occurs /dev/sdX does not exist, reboot system
Run dietpi-drive_manager again and mount drive

Fourdee on 26 Jun 2018

Ok, outstanding:

Allow fs check on reboot for root and boot partition. However, we did not yet patch the needed fstab flag: 0eb0db0
test RootFS transfer (again)

@MichaIng
If you want a crack at either of the top, go for it.

Fourdee on 26 Jun 2018

Note to self:
🈯️ Reserved blocks : needs to be below mount/unmount options in whip list.

Fourdee on 26 Jun 2018

RPi RootFS transfer:

Hangs on boot, waiting for UUID device. UUID's match.

Fourdee on 27 Jun 2018

@Fourdee

mkfs.ext4 /dev/sda indeed creates file system without partition table. I am wondering what the benefit of a partition table is in this case, as with one partition it works just well? But those drives are now well detected an handled by dietpi-drive_manager 👍.
I found, even that I usually disable RAMlog, /var/log is mounted to RAM again, overmounting existing logs, as dietpi-ramlog.service is not active. Reason is that currently every drive_manager start resets /etc/fstab, re-adding block devices well, but resets all network drives, ram drives and swap file. We need to check for installed/configured network drives, dietpi-ramlog and swapfile and of course manually configured RAM drives. Actually the scraping and re-adding of all existing /etc/fstab entries is missing.
x-systemd.automount is removed, if not supported by kernel, but it is never added currently. I just checked, what the benefits of systemd mount management are and if I understand it right, it just allows in combination with noauto to not mount the drive at boot, but on first access. Quite an idea for external drives actually, but of course not for root+boot fs: http://man7.org/linux/man-pages/man5/systemd.mount.5.html
On my VM drive_manager identifies /dev/sr0, which is the empty CD/DVD drive. Of course for native PCs and VMs it would be nice, if those were auto mounted, if disk is inserted, but format and all those things of course do not work here. On older kernels, if the removable drive is IDE (instead of SCSI), it could be also /dev/hdc: https://superuser.com/questions/272657/whats-the-difference-between-dev-hdc-dev-sr0-dev-cdrom#272682
Currently drive_manager calls df, blkid and lsblk to identify all drives and information. Actually all this info can scraped from a single lsblk call e.g.: lsblk -npao KNAME,TYPE,RO,SIZE,UUID,PARTUUID,FSTYPE,MOUNTPOINT
- This would allow to skip all this "already found" checks etc.
- KNAME=/dev/path, TYPE let's us identify/exclude ROM drives, loop drives and disks vs. partitions
- RO=readonly (0 or 1)
- The others are self explaining and should be empty from the back if device is unmounted>not formated>don't know in which case UUIDs are not present, but e.g. this is the case for /dev/sda, if /dev/sda1 is the partition.
- Not sure, if we need PARTUUID, actually it seems UUID is always available on at least every formatted drive, whereas PARTUUID is sometimes missing:

2018-06-28 16:49:52 root@micha:/var/log# lsblk -ao NAME,TYPE,UUID,PARTUUID
NAME        TYPE UUID                                 PARTUUID
loop0       loop
loop1       loop
loop2       loop
loop3       loop
loop4       loop
loop5       loop
loop6       loop
loop7       loop
sda         disk 192978f5-3451-45ce-b00b-e5e96738ddeb
sdb         disk 90b1e520-1301-459c-ac28-d98d43e59a39
mmcblk0     disk
├─mmcblk0p1 part A365-6756                            a277a2ba-01
└─mmcblk0p2 part 90a83158-560d-48ee-9de9-40c51d93c287 a277a2ba-02
2018-06-28 16:56:15 root@micha:/var/log# lsblk -npao KNAME,TYPE,RO,SIZE,UUID,PARTUUID,FSTYPE,MOUNTPOINT
/dev/loop0     loop  0
/dev/loop1     loop  0
/dev/loop2     loop  0
/dev/loop3     loop  0
/dev/loop4     loop  0
/dev/loop5     loop  0
/dev/loop6     loop  0
/dev/loop7     loop  0
/dev/sda       disk  0  2.7T 192978f5-3451-45ce-b00b-e5e96738ddeb                                      ext4   /mnt/sda
/dev/sdb       disk  0  1.8T 90b1e520-1301-459c-ac28-d98d43e59a39                                      ext4
/dev/mmcblk0   disk  0 59.8G
/dev/mmcblk0p1 part  0 43.1M A365-6756                            a277a2ba-01                          vfat   /boot
/dev/mmcblk0p2 part  0 59.8G 90a83158-560d-48ee-9de9-40c51d93c287 a277a2ba-02                          ext4   /

What the hell are those 7 loop drives??

root@VM-Stretch:~# lsblk -ao NAME,TYPE,UUID,PARTUUID
NAME   TYPE UUID                                 PARTUUID
sda    disk
└─sda1 part 4ca3f13e-cf61-4f47-a250-d616ac55f9e4 4004d63c-01
sdb    disk 0e1fef85-e95c-4777-83ea-4339d1d043e5
sr0    rom
root@VM-Stretch:~# lsblk -npao KNAME,TYPE,RO,SIZE,UUID,PARTUUID,FSTYPE,MOUNTPOINT
/dev/sda  disk  0    8G
/dev/sda1 part  0    8G 4ca3f13e-cf61-4f47-a250-d616ac55f9e4 4004d63c-01                          ext4   /
/dev/sdb  disk  0  982M 0e1fef85-e95c-4777-83ea-4339d1d043e5                                      ext4   /mnt/0e1fef85-e95c-4777-83ea-4339d1d043e5
/dev/sr0  rom   0 1024M

Another feature wish I have, is allowing to set a manual mount target. I just don't like those long UUIDs, even that I understand the benefit of allowing no chance that another drive is accidentally over-mounting an used mount target.
Also I think it would be safe to just dietpi-services stop when doing some actual critical task like moving userdata or rootfs. As even unmounting those drives is blocked, I think just we could really reduce services stop to those two actions + doing fs check on userdata and swapfile partition (as those are unmounted temporarily).
About system check on reboot:
- bootfs actually can be checked live, as it is just needed on startup (and for dietpi-ramdisk).
- touch /forcefsck does not work at all on x86_64 (on RPi it still it, although it is told to be obsolete), but fsck.mode=force does work (as well on RPi). Also at least on x86_64 VM the fsck flag in /etc/fstab is not needed for this. Maybe needs re-check again on RPi with current APT firmware package. Only thing then is that we need to remove the fsck.mode=force again after reboot. Maybe add some postboot.d script that removes the entry and itself again?
- An alternative is to use tune2fs -c 1 to make let the system check this drive on every reboot and change this back to initial value again (which is then needed to be safed). ~~But this definitely needs the fstab flag~~. €: Interesting, it does not on VM!

MichaIng on 28 Jun 2018

@MichaIng

x-systemd.automount is removed, if not supported by kernel, but it is never added currently. I just checked, what the benefits of systemd mount management are and if I understand it right, it just allows in combination with noauto to not mount the drive at boot, but on first access. l

Remove the USB drive, reboot, plug in the USB drive, automatically mounts the drive when plugged in. Or at least it should 😃

Yes your right, appears to disable for all devices. I'll take a look.

EDIT: CONFIG_AUTOFS4_FS=y, built into kernel, module will fail?
EDIT2: As far as i'am aware Rock64 was the only SBC with this issue: https://github.com/Fourdee/DietPi/issues/1607#issuecomment-372030565

I found, even that I usually disable RAMlog, /var/log is mounted to RAM again, overmounting existing logs, as dietpi-ramlog.service is not active. Reason is that currently every drive_manager start resets /etc/fstab, re-adding block devices well, but resets all network drives, ram drives and swap file. We need to check for installed/configured network drives, dietpi-ramlog and swapfile and of course manually configured RAM drives. Actually the scraping and re-adding of all existing /etc/fstab entries is missing.

Yep, this needs fixing.

We no longer scrape fstab, instead we generate a fresh /etc/fstab each time via detected devices.
The reason for this, basically, scraping fstab is messy and unpredictable. With the system scrapes, we have a valid, standardized entry we can work with.

What the hell are those 7 loop drives??

Inactive default loop devices available to system, i believe. same as ls -lha /dev/tty*

Another feature wish I have, is allowing to set a manual mount target. I just don't like those long UUIDs, even that I understand the benefit of allowing no chance that another drive is accidentally over-mounting an used mount target.

Yep, agree nice idea 👍

Fourdee on 28 Jun 2018

@MichaIng

An alternative is to use tune2fs -c 1 to make let the system check this drive on every reboot and change this back to initial value again (which is then needed to be safed). But this definitely needs the fstab flag. €: Interesting, it does not on VM!

Tried this on NanoPC T4, doesn't seem to have any effect (include fstab 1 change), no info of repair in dmesg or logs.

Hmm, had this idea of copying /sbin/e2fsck to a ramdisk, we can use during boot before rootfs mount, then repair via service, but, not had a chance to try it yet.

Fourdee on 28 Jun 2018

@Fourdee
Hmm that's sad. Boot cmd line is a bid messy, since it is found very differently on the SBCs:

x86_64: /etc/default/grub
RPi: /DietPi/config.txt
Odroid: /boot/boot.ini
Sparky: /boot/uEnv.txt
NanoPi: ?
tune2fs -C 2 -c 1 /dev/sda1 doesn't work either?
tune2fs -l /dev/sda1 | grep 'Last checked' shows old date as well?
I just tested on VM and also can't find any hint about the check in the logs, but I could see it on terminal during boot and the 'Last checked' time proves it as well.

I seems that indeed all systems handle fsck on reboot quite different and methods that work on the one, wont on the other: https://superuser.com/questions/401217/how-to-check-root-partition-with-fsck

There is this:

2018-06-29 10:56:34 root@micha:/var/log# cat /lib/systemd/system/systemd-fsck-root.service
[Unit]
Description=File System Check on Root Device
Documentation=man:systemd-fsck-root.service(8)
DefaultDependencies=no
Wants=systemd-fsckd.socket
Before=local-fs.target shutdown.target
After=systemd-fsckd.socket
ConditionPathIsReadWrite=!/
ConditionPathExists=!/run/initramfs/fsck-root

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-fsck
TimeoutSec=0

https://manpages.debian.org/experimental/systemd/systemd-fsck-root.service.8.en.html

This is a static unit, thus should run automatically, if conditions are met.
It depends on fstab flag and either fsck.mode=force or tune2fs mount count > max mount count.

/run/initramfs/fsck-root would be created by initramfs running fsck: https://manpages.debian.org/jessie/initramfs-tools/initramfs-tools.8.en.html#FILES

The same as above, but does not depend on fstab flag.

On RPi, the initramfs seems to be not active by default or at least does not run fsck. Don't know much about how to use initramfs btw or where it is possible etc. This is the reason why RPi boot fsck strictly depends on fstab flag, why VM does not.

So what actually should work on every system, is the systemd-fsck-root, if not initramfs already checked and created /run/initramfs/fsck-root. The latter also seems to be not used any more on newest versions, but hmm on my Buster VM it is still created: https://github.com/systemd/systemd/commit/956eaf2b8d6c9999024705ddadc7393bc707de02

So to further investigate on other systems:

ls -al /run/initramfs/
systemctl status systemd-fsck-root
tune2fs -l /dev/sda1 | grep 'Last checked'

Maybe we can add the feature for x86_64 and RPi only first and add other SBCs as fast as we can find a way.

I would currently go this way, as long as there is no cleaner way via custom initramfs?

[ ] By default add $6 flag 1 to /etc/fstab on root device (enabling systemd-fsck-root and first priority)
[ ] To allow user control over tune2fs parameters (max mount count before fsck), save current maximum mount count to /DietPi/dietpi/.fsck-root, set it to 1 before reboot and check for /DietPi/dietpi/.fsck-root on boot to reset the max mount count. This could be done via postboot.d/ script which then removed itself as well, or just by default within DietPi-Boot.

About not scraping /etc/fstab:

Yeah I agree. At best we then use dietpi.txt settings/values to correctly add (or not) swapfile and ramlog.
This will not allow users to add custom ram disks, but strictly seen, we never supported this. Perhaps we can add this some time later to drive_manager 😉.
Network drives are still copied from old fstab. Maybe we can safe those info as well within separate config file, maybe dietpi.txt as well? Just difficult to handle, if more than one drive of same type is added...

What do you think about using lsblk -npao KNAME,TYPE,RO,SIZE,UUID,FSTYPE,MOUNTPOINT only to detect all drives at once?

Currently skip drives of type rom.
Then only add those with UUID to be selectable and those without UUID but SIZE (and TYPE=disk) are the drives (with dedicated partitions), thus can be used as headers/separators between the partitions. €: Ah a totally empty drive does not have UUID as well, just after mkfs has been applied. So there would be another trick necessary to (e.g. comparing /dev/_ names) to differentiate between totally empty drives (allow to edit/format) and those with existing partition table (don't allow to edit, or actually why not, to allow formatting whole drive there?).
- I tried to format empty /dev/sdb with drive_manager:

DietPi-Drive_Manager: mount /dev/sdb /mnt/fc2ac7c8-fab4-4c39-a391-6a1b1979e391
                                   │  - Exit code: 32
                                   │  - DietPi version: v6.10 | HW_MODEL:20 | HW_ARCH:10 | DISTRO:5
                                   │  - Image creator: 0
                                   │  - Pre-image: Debian netboot
                                   │
                                   │ Log file contents:
                                   │ mount: /mnt/fc2ac7c8-fab4-4c39-a391-6a1b1979e391: special device /dev/sdb does not
                                   │ exist.

Although the device is listed with lsblk, which shows UUID, type, size etc correctly, /dev/sdb does not exist any more. After reboot it was there again and worked as expected. Also another format then worked.
Ah I guess this is due to rm -R $FORMAT_PREVIOUS_MOUNT_TARGET and: https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-drive_manager#L356 (aDRIVE_MOUNT_TARGET[$INDEX_INIT_CURRENT_DEVICE]=${aDRIVE_MOUNT_SOURCE[$INDEX_INIT_CURRENT_DEVICE]})
The latter should be empty string for new devices, so that never a device node is removed.

x-systemd.automount doesn't work on VM at least, no matter with used with auto or noauto: umount drive unplug and plug does not lead to auto mount. Also on reboot, the drive is not mounted with noauto. But that's USB stick passed via host to VM, so this can be totally different than on real device. Might be an argument to skip this option on VMs.
Can't test with VM internal drive (not via host USB), since those cannot be unplugged live. But this also does not make any sense then to use automount for them 😉.

MichaIng on 29 Jun 2018

👍1

Generally about formatting:

Currently FORMAT_RECREATE_PARTITION_TABLE is always =1, if I didn't miss something.
On empty devices (just /dev/sdX exists, no partitions), leads leads to first a partition table is created on aDRIVE_SOURCE_DEVICE and afterwards this partition table is overwritten by mkfs on aDRIVE_MOUNT_SOURCE, as both variables are the same: /dev/sdX
We have to decide, whether we always want to create a partition table, even that we just create a single partition (then partition number needs to be added to aDRIVE_MOUNT_SOURCE) or we skip all the partitioning directly, which as said works great here. Seems to be only needed if more then one partition is wanted, a feature we anyway do not offer currently.

For drives with existing partition table, when formatting the partition [[ aDRIVE_SOURCE_DEVICE != aDRIVE_MOUNT_SOURCE ]] we just need to take care to not format the whole drive, but just the device. But as currently aDRIVE_MOUNT_SOURCE is used there, this is already the case.

MichaIng on 29 Jun 2018

👍1

tune2fs -C 2 -c 1

Odroid C2:

🈴 Not viable, only the boot.ini entry worked (which is default enabled)

NanoPC T4:

Appears boot code forces a check on this device. Unsure if its repairing?
I'll need to test another system.

root@DietPi:~# tune2fs -C 2 -c 1 /dev/sda
tune2fs 1.44.2 (14-May-2018)
Setting maximal mount count to 1
Setting current mount count to 2

#rootfs
root@DietPi:~# tune2fs -C 2 -c 1 /dev/mmcblk1p6
tune2fs 1.44.2 (14-May-2018)
Setting maximal mount count to 1
Setting current mount count to 2

#fstab
#Internal Drives---------------------------------------------------
UUID=ff313567-e9f1-5a5d-9895-3ba130b4a864 / auto rw,noatime 0 1
UUID=c976771f-1ec9-445b-949c-d6b2883e7860 /mnt/c976771f-1ec9-445b-949c-d6b2883e7860 auto rw,x-systemd.automount,nofail,noatime 0 2

#reboot

root@DietPi:~# tune2fs -l /dev/sda | grep 'Last checked'
Last checked:             Tue Jun 26 15:14:46 2018

root@DietPi:~# tune2fs -l /dev/mmcblk1p6 | grep 'Last checked'
Last checked:             Fri Jun 29 15:03:38 2018

#set fstab entries back to 0 0
#reboot

root@DietPi:~# tune2fs -l /dev/mmcblk1p6 | grep 'Last checked'
Last checked:             Fri Jun 29 15:04:59 2018

Interesting Lifetime writes looks useful, but only works since last formatted (as far as I can tell, not total lifetime)
My Odroid C2:

root@DietPi:~#  tune2fs -l /dev/mmcblk0p2 | grep writes
Lifetime writes:          123 GB

Recently formatted RPi SD card (1 year old card, heavly used)

Lifetime writes:          8190 MB

Fourdee on 29 Jun 2018

👍1

@Fourdee

Interesting Lifetime writes looks useful, but only works since last formatted (as far as I can tell, not total lifetime)

Jep, seems to be related/reset on format/fs creation. Nearly same writes here 😄:

Filesystem created:       Wed Apr 18 03:07:02 2018
Lifetime writes:          8104 MB

Did you check for /run/initramfs/fsck-root existence after reboot? As said this initramfs check doesn't show up on any log.

ls -al /run/initramfs/
systemctl status systemd-fsck-root

If this file doesn't exist, systemd-fsck-root should definitely kick in. It's a normal static unit that runs Before=local-fs.target and exist on every Raspbian + Debian systemd....
Ah found the reason: This service and systemd-fsck binary does NOT exist on Debian Jessie systemd version, just Stretch+: https://packages.debian.org/de/jessie/amd64/systemd/filelist
Hmm as > /forcefsck on the other hand does not work any more on newer kernel, does in instead work on Odroids? As at least the file is removed as well on Stretch+, we could just create it anyway as fail safe.

Okay, so I think on Stretch+ we can then rely on systemd-fsck. If I understand the man pages right, then the service itself understands fsck.mode=force and fsck.repair=yes from cmd line.

The first can be used as well of course to force fsck on next reboot, but as said I don't like that cmd line is located differently on every SBC. tune2fs seems to me the cleaner method then.
Default fsck.repair=preen repairs only what can be safely repaired. Looks actually as a good default to me. On RPi we already have fsck.repair=yes, I don't know if we should add this as default to all devices? Hmm if we anyway need to edit cmd lines, then we can also use it to force fsck on next reboot 🤔. Alternative would be to leave it as it is and show log on reboot to user (temporary bashrc.d script that also removes itself?):

2018-06-29 22:15:32 root@micha:~# echo "$(journalctl -t systemd-fsck)"
-- Logs begin at Thu 2018-06-28 01:05:01 CEST, end at Fri 2018-06-29 22:15:04 CEST. --
Jun 29 20:58:15 micha.gnoedi.org systemd-fsck[85]: e2fsck 1.44.2 (14-May-2018)
Jun 29 20:58:15 micha.gnoedi.org systemd-fsck[85]: rootfs: clean, 42911/3786016 files, 518504/15667200 blocks (check after next mount)

No idea if errors are logged here as well? There seems to be no other log location at least. Respectively, if fsck was performed by initramfs:

root@VM-Stretch:~# cat /run/initramfs/fsck.log
Log of fsck -C -a -V -t ext4 /dev/sda1
Fri Jun 29 20:22:02 2018

fsck from util-linux 2.29.2
[/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1
/dev/sda1: clean, 68439/524288 files, 635991/2096640 blocks

Fri Jun 29 20:22:02 2018
----------------

€: fsck.mode applies to all drives within fstab:

journalctl -t systemd-fsck
Jun 30 00:14:17 micha.gnoedi.org systemd-fsck[85]: e2fsck 1.44.2 (14-May-2018)
Jun 30 00:14:17 micha.gnoedi.org systemd-fsck[85]: Pass 1: Checking inodes, blocks, and sizes
Jun 30 00:14:56 micha.gnoedi.org systemd-fsck[85]: Pass 2: Checking directory structure
Jun 30 00:14:57 micha.gnoedi.org systemd-fsck[85]: Pass 3: Checking directory connectivity
Jun 30 00:14:57 micha.gnoedi.org systemd-fsck[85]: Pass 4: Checking reference counts
Jun 30 00:14:59 micha.gnoedi.org systemd-fsck[85]: Pass 5: Checking group summary information
Jun 30 00:15:01 micha.gnoedi.org systemd-fsck[85]: rootfs: 42590/3786016 files (0.2% non-contiguous), 518417/15667200 blocks
Jun 30 00:15:05 micha.gnoedi.org systemd-fsck[214]: fsck.fat 4.1 (2017-01-24)
Jun 30 00:15:05 micha.gnoedi.org systemd-fsck[214]: /dev/mmcblk0p1: 258 files, 46827/86872 clusters
Jun 30 00:15:05 micha.gnoedi.org systemd-fsck[235]: e2fsck 1.44.2 (14-May-2018)
Jun 30 00:15:19 micha.gnoedi.org systemd-fsck[235]: Pass 1: Checking inodes, blocks, and sizes
Jun 30 00:16:22 micha.gnoedi.org systemd-fsck[235]: Pass 2: Checking directory structure
Jun 30 00:16:46 micha.gnoedi.org systemd-fsck[235]: Pass 3: Checking directory connectivity
Jun 30 00:17:00 micha.gnoedi.org systemd-fsck[235]: Pass 4: Checking reference counts
Jun 30 00:17:59 micha.gnoedi.org systemd-fsck[235]: Pass 5: Checking group summary information
Jun 30 00:18:06 micha.gnoedi.org systemd-fsck[235]: /dev/sda: 979279/183148544 files (0.4% non-contiguous), 68303272/732566646 blocks

Just with tune2fs -c 1 /dev/mmcblk0p2:

journalctl -t systemd-fsck
Jun 30 00:57:29 micha.gnoedi.org systemd-fsck[87]: e2fsck 1.44.2 (14-May-2018)
Jun 30 00:57:29 micha.gnoedi.org systemd-fsck[87]: rootfs has been mounted 1 times without being checked, check forced.
Jun 30 00:57:29 micha.gnoedi.org systemd-fsck[87]: Pass 1: Checking inodes, blocks, and sizes
Jun 30 00:58:07 micha.gnoedi.org systemd-fsck[87]: Pass 2: Checking directory structure
Jun 30 00:58:09 micha.gnoedi.org systemd-fsck[87]: Pass 3: Checking directory connectivity
Jun 30 00:58:09 micha.gnoedi.org systemd-fsck[87]: Pass 4: Checking reference counts
Jun 30 00:58:10 micha.gnoedi.org systemd-fsck[87]: Pass 5: Checking group summary information
Jun 30 00:58:13 micha.gnoedi.org systemd-fsck[87]: rootfs: 42596/3786016 files (0.2% non-contiguous), 518425/15667200 blocks
Jun 30 00:58:15 micha.gnoedi.org systemd-fsck[200]: fsck.fat 4.1 (2017-01-24)
Jun 30 00:58:15 micha.gnoedi.org systemd-fsck[200]: /dev/mmcblk0p1: 258 files, 46827/86872 clusters
Jun 30 00:58:15 micha.gnoedi.org systemd-fsck[235]: e2fsck 1.44.2 (14-May-2018)
Jun 30 00:58:28 micha.gnoedi.org systemd-fsck[235]: /dev/sda: clean, 979279/183148544 files, 68303272/732566646 blocks

It seems to "touch" every disk, but just checks the ones, where mount count was reached. But as can be seen, this takes quite a lot of time, even on nearly empty 64 GB SDcard. Thus I agree this should not be done on every reboot. Default here is as well, that fsck is done, if flagged by kernel I/O error or unordered shutdown (powerloss).

Btw I found the reason why on RPi initramfs does no fsck:

2018-06-29 21:30:59 root@micha:~# cat /etc/default/raspberrypi-kernel
# Defaults for raspberrypi-kernel

# Uncomment the following line to enable generation of
# /boot/initrd.img-KVER files (requires initramfs-tools)

#INITRD=Yes

# Uncomment the following line to enable generation of
# /boot/initrd(7).img files (requires rpi-initramfs-tools)

#RPI_INITRD=Yes

Not sure if it is used somewhere else, but found this now: https://raspberrypi.stackexchange.com/a/26871
Looks like indeed RPi does not need initramfs-tools unless you want to do something special before mounting root. As well the config file indicates that rpi-initramfs-tools might offer an alternative. But I can't find this package on any repo, nor via web search anything about it.
€: Tested and works fine. kmod and udev need to be marked as manual then, all other packages can be safely removed, especially:
- plymouth for custom splash screen
- mountall is deprecated anyway, not available on Debian/Raspbian repo since Stretch, but preinstalled on Raspbian Stretch... no dependency of anything, but marked as "important", thus not autoremoved. Looks like Raspbian images are just ported from one version to another instead of build up fresh each time. If not taken care about obsolete packages, once marked as important, they will survive from one image to the next. This can be seen as well for several versions of gcc-x.y-base installed on Raspbian. All but the most current can be securely purged.
- linux-base seems to be used by linux-image-* package install scripts, but not by raspberrypi-firmware packages.
Might it be the case for other SBCs with non mainline kernel as well?

MichaIng on 30 Jun 2018

Just adding a note to say I experienced the same originally reported issue yesterday with an ODROID HC2 with the most recent download available on the website. I reported this in the forum via a post already about this (https://dietpi.com/phpbb/viewtopic.php?f=11&t=3908) but only found this bug report today.

I'm happy to help test fixes to sort this, but will need to read through this full post first.

pruperting on 30 Jun 2018

@MichaIng

https://github.com/Fourdee/DietPi/commit/f91bf4b44856d37ecb4dd6f6ddcbcaea835d8e0e
A few more fstab scrapes than i'd like, but this was the fastest and most efficient way to resolve RAMLOG ignored and /tmp issues.

Fourdee on 30 Jun 2018

Move RootFS on x86_64: https://unix.stackexchange.com/questions/5297/debian-grub2-moving-root-partition-to-new-drive

Needs testing => v6.11

MichaIng on 2 Jul 2018

Note to self:
I still need to do work on this:

Disallow userdata/swap if drive is RO
Disallow RO if userdata/swap

etc.

Menu_Drive_Manager_Edit_Drive getting messy, near impossible to control cat/shown options, rewrite?

local show_mount=0
local show_unmount=0
local show_format=0
local show_benchmark=0

if (( has_fs && mounted && read_write )); then

    show_benchmark=1
fi

#Then 

if (( $show_benchmark )); then

    G_WHIP_MENU+=('Benchmark' 'xxx')

fi

Then calculate?

Fourdee on 3 Jul 2018

Ok as it stands, this is completed.

Minor issue with: https://github.com/Fourdee/DietPi/issues/1858#issuecomment-401970424, however, does not affect functionality, only code layout. If it bugs me enough, and I get time, i'll tweak it post v6.10 release.

Marking as closed.

Fourdee on 3 Jul 2018

❤1 👍1

sounds great!
You guys are doing a great job!!!
i'm really looking forward to the update/image :))

butterbrot21 on 3 Jul 2018

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

DietPi-Software | nginx maximum upload size

pfeerick · 3Comments

DietPi-Drive_Manager | Why is it not supported on VM?

MichaIng · 3Comments

Info : In response to mayan-edms.com

Fourdee · 3Comments

DietPi Image | NanoPi M1+

Fourdee · 3Comments

Netdata 1.7

Kapot · 3Comments