DietPi-Drive_Manager | Root/BootFS fsck on reboot

[Fourdee]

https://dietpi.com/phpbb/viewtopic.php?f=12&t=3126&p=11924#p11917

[MichaIng]

@Fourdee
Jep, I like the idea. Although I am not sure, if forced fix is always the best, as it can also lead to data loss. Although, the data is anyway lost, if fix cannot be applied, right?

[Fourdee]

@MichaIng

Although, the data is anyway lost, if fix cannot be applied, right?

Pretty much, lost via known fixable means. Paid Data recovery business then required.

[MichaIng]

@Fourdee
Yeah, the question is if then the fsck fix attempt and removal/clearing makes things worse.
Maybe there is some level of errors where we can leave choice to user, informing in the first place about possible results of a forced fix/clearing.

[Fourdee]

@MichaIng

Maybe there is some level of errors where we can leave choice to user, informing in the first place about possible results of a forced fix/clearing.

Yep 👍

Dry run 1st, then repair option?

[Fourdee]

Finally fixed the incorrect scaling in G_WHIP while I was here 🤣 :
We now calculate character count per line against X, yep, its a little slower, but, required to prevent lost free space driving me up the wall 😃 :

Before

After

[Fourdee]

🈯️ Tested with EXT4 external drive

[Fourdee]

Ok so only downside, as we define drive must be unmounted for a check/repair to run, this removes the ability to do this on:

• /boot (which has umount disabled to prevent issues with ramdisk etc)
• RootFS (cannot/should not be done on a active filesystem)

[MichaIng]

@Fourdee
Jep, I guess this is known that rootfs can only be scanned and repaired on reboot (boot arguments) or on other system.

For /boot, as long as it's within single script on our control, I think it's okay to umount + check + remount it? Only important thing is it being mounted again on reboot/shutdown for ramdisk?
Actually as far as I can remember FAT fs can be scanned while mounted as well, but still to be on secure side I would always umount.

€: Ah maybe add scan on reboot option for (boot and) root via > /forcefsck? I know its deprecated but still works and is the only method I know to just scan a single time. On RPi we added fsck.repair=1 by default, which is okay since on ext4 by default just uncritical errors are repaired, other than for some other fs repair tools.

[MichaIng]

@Fourdee
Testing: We definitely need separate commands + options for supported FS types.

• 🈺 F2FS: -f for full check&repair, -a just for actually processed files? Dry-run is default.

root@DietPi:/var/log# fsck -h /dev/sdb1
fsck from util-linux 2.29.2
fsck.f2fs: invalid option -- 'h'
        Error: Unknown option ?

Usage: fsck.f2fs [options] device
[options]:
  -a check/fix potential corruption, reported by f2fs
  -d debug level [default:0]
  -f check/fix entire partition
  -p preen mode [default:0 the same as -a [0|1]]
  -t show directory tree [-d -1]

• 🈴 NTFS: fsck /dev/sdXX does not work. Needed to create fsck.ntfs manually: ln -s $(which ntfsfix) /bin/fsck.ntfs We could do this on DietPi-PREP or if $(which fsck.ntfs) is empty, on/before starting drive manager fs check.

root@DietPi:/var/log# fsck.ntfs --help
ntfsfix v2016.2.22AR.1 (libntfs-3g)

Usage: ntfsfix [options] device
    Attempt to fix an NTFS partition.

    -b, --clear-bad-sectors Clear the bad sector list
    -d, --clear-dirty       Clear the volume dirty flag
    -h, --help              Display this help
    -n, --no-action         Do not write anything
    -V, --version           Display version information

For example: ntfsfix /dev/hda6

Developers' email address: [email protected]
News, support and information:  http://tuxera.com

• 🈺 FAT32 (vfat) Supports -n but -f has different meaning.

root@DietPi:/var/log# fsck.fat --heko
fsck.fat: invalid option -- '-'
usage: fsck.fat [-aAbflrtvVwy] [-d path -d ...] [-u path -u ...]
               device
  -a       automatically repair the filesystem
  -A       toggle Atari filesystem format
  -b       make read-only boot sector check
  -c N     use DOS codepage N to decode short file names (default: 437)
  -d path  drop that file
  -f       salvage unused chains to files
  -l       list path names
  -n       no-op, check non-interactively without changing
  -p       same as -a, for compat with other *fsck
  -r       interactively repair the filesystem (default)
  -t       test for bad clusters
  -u path  try to undelete that (non-directory) file
  -v       verbose mode
  -V       perform a verification pass
  -w       write changes to disk immediately
  -y       same as -a, for compat with other *fsck

• 🈴 BTRFS: Even different command is needed

root@DietPi:/var/log# fsck.btrfs
If you wish to check the consistency of a BTRFS filesystem or
repair a damaged filesystem, see btrfs(8) subcommand 'check'.
root@DietPi:/var/log# btrfs check
btrfs check: too few arguments
usage: btrfs check [options] <device>

    Check structural integrity of a filesystem (unmounted).

    Check structural integrity of an unmounted filesystem. Verify internal
    trees' consistency and item connectivity. In the repair mode try to
    fix the problems found.
    WARNING: the repair mode is considered dangerous

    -s|--super <superblock>     use this superblock copy
    -b|--backup                 use the first valid backup root copy
    --repair                    try to repair the filesystem
    --readonly                  run in read-only mode (default)
    --init-csum-tree            create a new CRC tree
    --init-extent-tree          create a new extent tree
    --mode <MODE>               select mode, allows to make some memory/IO
                                trade-offs, where MODE is one of:
                                original - read inodes and extents to memory (requires
                                           more memory, does less IO)
                                lowmem   - try to use less memory but read blocks again
                                           when needed
    --check-data-csum           verify checksums of data blocks
    -Q|--qgroup-report           print a report on qgroup consistency
    -E|--subvol-extents <subvolid>
                                print subvolume extents and sharing state
    -r|--tree-root <bytenr>     use the given bytenr for the tree root
    --chunk-root <bytenr>       use the given bytenr for the chunk tree root
    -p|--progress               indicate progress

• 🈴 exFAT: No options available at all

root@DietPi:/var/log# fsck.exfat
exfatfsck 1.2.5
Usage: fsck.exfat [-V] <device>

• 🈯️ HFS+: Compatible with ext4 options.

root@DietPi:/var/log# fsck.hfs
fsck.hfs: missing special-device
usage: fsck.hfs [-c [size] dfl m [mode] npqruy] special-device
  c size = cache size (ex. 512m, 1g)
  d = output debugging info
  f = force fsck even if clean (preen only)
  l = live fsck (lock down and test-only)
  m arg = octal mode used when creating lost+found directory
  n = assume a no response
  p, a = just fix normal inconsistencies
  q = quick check returns clean, dirty, or failure
  r = rebuild catalog btree
  u = usage
  y = assume a yes response

[Fourdee]

@MichaIng

We definitely need separate commands + options for supported FS types.

Great work 👍

~I'll implement this today and send commit.~ 😃

[MichaIng]

@Fourdee
Waiiiit, I'm already done 😄! Currently testing with all FS and will push the commit to my opened PR.

€: PR is up: https://github.com/Fourdee/DietPi/pull/1763

[Fourdee]

@MichaIng

Great work on this PR https://github.com/Fourdee/DietPi/pull/1763, legend 👍

PS. Parcel sent today (signed for 1st class), should be with you next week (NanoPi K1+ and a special item 😃 )

[MichaIng]

ToDo (v6.9?)

• [x] Select root/boot device for scan (& repair) on next reboot.
  
  
  
  • With https://github.com/Fourdee/DietPi/commit/0eb0db0209985ff91fcd425bbd4ca4a4da6626d8 possible just for rootfs right now.
  
  
  • Live Check & Repair disabled for root and boot
  
  
  • Enable /boot scan on reboot and/or live? The latter would be possible, as long as re control remount for proper DietPi-RAMdisk sync, kernel/firmware updates etc.
  
  

Methods

• > /forcefsck deprecated but still works, echo y > /forcefsck does not work any more to force repair.
  
  
  
  • Temporary: File is removed after reboot!
  
  
• RPi only: Add fsck.mode=force to cmdline.txt, fsck.repair=yes to force repair. According to some web research, ext4 auto repair seems to be safe, as it just fixes errors, that are safe to repair and can not create additional damage.
  
  
  
  • Permanent, would need removal after reboot, at least if no already added manually by user.
  
  
• tune2fs -c 1 /dev/sdXX: Set automatic scan interval to every 1 boot.
  
  
  
  • Permanent, would need adjustment to default value after reboot.
  
  

[Fourdee]

@MichaIng

v6.9

Yep 👍

[Fourdee]

Interesting, i ran a check + repair on rootFS on the ASUS TB, it went through and ran the repair fine lol, however, its set rootFS to RO:

root@DietPi:~# mount
/dev/mmcblk0p2 on / type ext4 (ro,noatime,data=ordered)

[MichaIng]

Move scan on reboot to v6.10

• Concentrate on testing many deep changes done already.

€: Moved to v6.11

[MichaIng]

Not sure where I found/reported this:

• On x86, fsck is done via initramfs and on Jessie, the created flag (don't remember location of this currently) prevents other fsck methods (on later boot stage) from being done. Nothing shows up in boot log then (dmesg/journalctl) as well then.
• On Stretch, there is generally systemd-fsck available, which does rootfs check on reboot, if requested by one of the above methods. On Jessie, systemd-fsck does not yet exist, thus above methods do not work in every case, especially if no initramfs is used (e.g. RPi).
• On RPi fsck.mode=force does it on earliest boot stage, but need to be actively removed to prevent this on every boot.

So assuring rootfs fsck on reboot on all systems without further removal steps after reboot, only works on Stretch+, AFAIK, via > /forcefsck, done by either initramfs or systemd-fsck then.

• [x] Check if the flag file forces it on all partitions then or only on the partition where it is placed. Then enable this feature only on Streth+.
• [x] Check whether forcefsck works still on Buster, since it is already marked as deprecated for a while.

[Fourdee]

@MichaIng

The idea I had to cover all systems was:

• During boot at local-fs.target once rootfs is mounted
• Copy fsck binary somewhere off the rootfs (/tmp?)
• service which unmounts rootfs, runs /tmp/fsck, then remounts
• disable service

However, don't believe you can unmount rootfs once its mounted in any case.

[MichaIng]

@Fourdee
Would be elegant, but also intrusive. I am not sure if unmounting also affects other mounts somehow? And perhaps there can be some background processes active during boot that would fail then 🤔.

We could test it, but if > /forcefsck works reliable on all systems, I would stick with this proven method.

[MichaIng]

Retest

🈯️ VM Jessie:

> /forcefsck
reboot
root@VM-Jessie:~# cat /run/initramfs/fsck.log
Log of fsck -C -a -V -t ext4 /dev/sda1
Sat Dec  8 21:53:09 2018

fsck from util-linux 2.25.2
[/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1
/dev/sda1: clean, 15550/524288 files, 471578/2096640 blocks

Sat Dec  8 21:53:09 2018
----------------

🈯️ VM Stretch

root@VM-Stretch:~# cat /run/initramfs/fsck.log
Log of fsck -C -a -V -t ext4 /dev/sda1
Sat Dec  8 22:00:58 2018

fsck from util-linux 2.29.2
[/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1
/dev/sda1: clean, 15956/524288 files, 464742/2096640 blocks

Sat Dec  8 22:00:58 2018
----------------

🈯️ VM Buster

root@VM-Buster:~# cat /run/initramfs/fsck.log
Log of fsck -C -a -T -t ext4 /dev/sda1
Sat Dec  8 22:08:42 2018

/dev/sda1: clean, 18346/524288 files, 496129/2096640 blocks

Sat Dec  8 22:08:42 2018
----------------

• So on VM this works well. I guess it should be the same on all x86 machines via initramfs.

🈯️ Native x86 Stretch notebook

• Okay so the deprecated warning seems to occur just on RPi, where fsck.mode=force _can_ be applied to cmdline.

---

On ~Stretch+~ (€: Rechecked, seems to be implemented into Jessie as well via APT systemd update), systemd-fsck is available and would do the job, if not done by initramfs already, checked by flag file:

root@VM-Stretch:~# cat /lib/systemd/system/systemd-fsck-root.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=File System Check on Root Device
Documentation=man:systemd-fsck-root.service(8)
DefaultDependencies=no
Wants=systemd-fsckd.socket
Before=local-fs.target shutdown.target
After=systemd-fsckd.socket
ConditionPathIsReadWrite=!/
ConditionPathExists=!/run/initramfs/fsck-root

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-fsck
TimeoutSec=0
root@VM-Stretch:~# systemctl status systemd-fsck-root.service
● systemd-fsck-root.service - File System Check on Root Device
   Loaded: loaded (/lib/systemd/system/systemd-fsck-root.service; static; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Sat 2018-12-08 23:32:15 CET; 6min ago
           └─ ConditionPathExists=!/run/initramfs/fsck-root was not met
     Docs: man:systemd-fsck-root.service(8)

---

🈯️ Check what is done on RPi Buster with: > /forcefsck:

2018-12-08 23:54:08 root@micha:~# journalctl -t systemd-fsck
-- Logs begin at Thu 2018-12-06 22:05:01 CET, end at Sat 2018-12-08 23:51:17 CET. --
Dec 08 23:48:18 micha.gnoedi.org systemd-fsck[107]: Please pass 'fsck.mode=force' on the kernel command line rather than creating /forcefsck on the root file system.
Dec 08 23:48:18 micha.gnoedi.org systemd-fsck[107]: e2fsck 1.44.4 (18-Aug-2018)
Dec 08 23:48:18 micha.gnoedi.org systemd-fsck[107]: Pass 1: Checking inodes, blocks, and sizes
Dec 08 23:48:56 micha.gnoedi.org systemd-fsck[107]: Pass 2: Checking directory structure
Dec 08 23:48:58 micha.gnoedi.org systemd-fsck[107]: Pass 3: Checking directory connectivity
Dec 08 23:48:58 micha.gnoedi.org systemd-fsck[107]: Pass 4: Checking reference counts
Dec 08 23:49:00 micha.gnoedi.org systemd-fsck[107]: Pass 5: Checking group summary information
Dec 08 23:49:02 micha.gnoedi.org systemd-fsck[107]: rootfs: 41075/3786016 files (0.2% non-contiguous), 494822/15667200 blocks

• Ah deprecated comes from systemd-fsck, not from RPi. But since it still works even on Buster, we can implement. It has the clear advantage that the flag is removed automatically, so we don't need to adjust anything for a oneshot.

---

Okay since systemd is basis for all our images and systemd-fsck part of it even on Jessie, should work on all devices, with and without initramfs.

[MichaIng]

PR up: https://github.com/Fourdee/DietPi/pull/2335

[MichaIng]

Will mark as resolved.