Dietpi: Weird corrupption of dietpi commands

[samolego]

Creating a bug report/issue

I've encountered some strange behaviour; I've updated my system to debian buster and ran dietpi script (https://github.com/MichaIng/DietPi/issues/1285) for conversion. Everything went smoothly, but when I rebooted / shutted down and turned on the system, I got the following error;
-bash: /DietPi/dietpi/func/dietpi-globals: No such file or directory [FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

This was shown on every SSH instance I opened up, instead of "normal" dietpi info. I thought I've messed sth. up, so I reinstalled the system. Re-ran the dietpi conversion script. Shutted down the system and powered it back on. Same issue. I thouht I messed sth. again and did it the third time. This time I didn't shutdown the system for 3 days, everything was running smoothly. Today I decided to reboot it. Same problem.
Further checking shows that /DietPi folder is empty, leading to none of the commands working.

Required Information

• DietPi version | cannot run any dietpi cmd, but I'm sure it was latest, since I installed it freshly
• Distro version | 10.2
• Kernel version | Linux minecraft-server 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
• SBC device | Atomic Pi
• Power supply used | 5V 3A
• SDcard used | Has internal memory

Additional Information (if applicable)

• Was the software title installed freshly or updated/migrated?
  At first updated, then installed freshly twice.
• Can this issue be replicated on a fresh installation of DietPi?
  I've done fresh installation of debian buster twice and reproduced it both times.

Steps to reproduce

1. Install fresh debian buster, x86_64
2. Run script to convert to dietpi (https://github.com/MichaIng/DietPi/issues/1285)
3. Reboot / shutdown system - mine happened after rebooting for 2nd / 3rd time

Expected behaviour

Dietpi commands still existing

Actual behaviour

-bash: /DietPi/dietpi/func/dietpi-globals: No such file or directory [FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

[MichaIng]

@samolego
Many thanks for your report.

I guess the RAMdisk failed to load. Could you paste:

journalctl -u DietPi.mount
journalctl -u dietpi-ramdisk
cat /var/tmp/dietpi/logs/dietpi-ramdisk.log

[samolego]

Here's the output;
journalctl -u DietPi.mount
-- Logs begin at Tue 2019-12-17 16:21:04 GMT, end at Tue 2019-12-17 19:00:57 GMT. -- Dec 17 16:21:04 minecraft-server systemd[1]: Mounting /DietPi... Dec 17 16:21:04 minecraft-server systemd[1]: Mounted /DietPi.
journalctl -u dietpi-ramdisk
-- Logs begin at Tue 2019-12-17 16:21:04 GMT, end at Tue 2019-12-17 19:00:57 GMT. -- Dec 17 16:21:05 minecraft-server systemd[1]: Starting DietPi-RAMdisk... Dec 17 16:21:05 minecraft-server dash[365]: /bin/dash: 1: cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system Dec 17 16:21:05 minecraft-server systemd[1]: dietpi-ramdisk.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Dec 17 16:21:05 minecraft-server systemd[1]: dietpi-ramdisk.service: Failed with result 'exit-code'. Dec 17 16:21:05 minecraft-server systemd[1]: Failed to start DietPi-RAMdisk.
cat /var/tmp/dietpi/logs/dietpi-ramdisk.log
Sat Dec 14 09:56:45 UTC 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 09:56:46 UTC 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Sat Dec 14 10:09:34 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Sat Dec 14 10:09:34 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk. Sat Dec 14 10:09:35 GMT 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 10:09:35 GMT 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Sat Dec 14 10:11:49 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Sat Dec 14 10:11:49 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk. Sat Dec 14 10:11:50 GMT 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 10:11:50 GMT 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Tue Dec 17 16:21:04 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Tue Dec 17 16:21:04 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk.

[MichaIng]

@samolego

cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system

Strange, is /var or /var/tmp a dedicated mount on your system?

cat /etc/fstab

[samolego]

df

Filesystem     1K-blocks    Used Available Use% Mounted on
udev              959876       0    959876   0% /dev
tmpfs             195316    3156    192160   2% /run
/dev/mmcblk0p2  12470936 3638992   8178736  31% /
tmpfs             976580       0    976580   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             976580       0    976580   0% /sys/fs/cgroup
tmpfs            1047552       0   1047552   0% /tmp
tmpfs              10240       0     10240   0% /DietPi
tmpfs              51200       8     51192   1% /var/log
cat /etc/fstab
# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1023M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------

[MichaIng]

@samolego
Hmm, you run DietPi on an SDcard on x86_64? However it seems to be not inside the fstab, which is a bid strange. Also since it is partition two, did you setup the originating Debian with two partitions, e.g. first partition for kernel+bootloader?

Please try to start the dietpi-ramdisk.service now and check back if it succeeded:

systemctl start dietpi-ramdisk
ls -l /DietPi

[samolego]

No, I don't run DietPi on an external SD, it's installed on the built-in memory (Atomic Pi has 16 GB built-in). I sadly cannot remember how I set up the partitioning, but I'm pretty sure I left it to default configuration during installation process.

The starting of dietpi ramdisk hasn't succeeded;
systemctl start dietpi-ramdisk
Job for dietpi-ramdisk.service failed because the control process exited with error code. See "systemctl status dietpi-ramdisk.service" and "journalctl -xe" for details.

ls -l /DietPi
total 0

And systemctl status dietpi ramdisk.service shows:
``` dietpi-ramdisk.service - DietPi-RAMdisk
Loaded: loaded (/etc/systemd/system/dietpi-ramdisk.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-12-18 11:45:16 GMT;
7min ago
Process: 913 ExecStartPre=/bin/mkdir -p /var/tmp/dietpi/logs (code=exited, status=0/SUCCESS)
Process: 914 ExecStart=/bin/dash -c /boot/dietpi/func/dietpi-ramdisk 0 2>&1 >> /var/tmp/dietpi/logs/dietpi-ramdisk.log (code=exited, status=2)
Main PID: 914 (code=exited, status=2)

Dec 18 11:45:16 minecraft-server systemd[1]: Starting DietPi-RAMdisk...
Dec 18 11:45:16 minecraft-server dash[914]: /bin/dash: 1: cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system
Dec 18 11:45:16 minecraft-server systemd[1]: dietpi-ramdisk.
service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 18 11:45:16 minecraft-server systemd[1]: dietpi-ramdisk.
service: Failed with result 'exit-code'.
Dec 18 11:45:16 minecraft-server systemd[1]: Failed to start
DietPi-RAMdisk.
```

[MichaIng]

@samolego
Okay, so the root file system is indeed not R/W mounted. So you currently cannot edit /etc/fstab, right?

Not sure why the required fstab entry is missing, however lets try to fix this manually first:

mount -o rw,remount /dev/mmcblk0p2 /

If this works, then edit /etc/fstab, e.g. nano /etc/fstab and add the following last line:

/dev/mmcblk0p2 / ext4 noatime,lazytime 0 1

(I assume its ext4 file system?)
I guess /dev/mmcblk0p1 contains the boot partition? Check if /boot directory is empty: ls -Al /boot
If so, do: mount /dev/mmcblk0p1 /boot
Check now that /boot partition is not empty anymore and contains e.g. dietpi.txt and some other files. If so, add the related fstab entry as additional line:

/dev/mmcblk0p1 /boot auto noatime,lazytime 0 2

Now you should be able to start dietpi-ramdisk.service:

systemctl restart dietpi-ramdisk

This should lead to /DietPi directory being not empty anymore: ls -Al /DietPi

[samolego]

The service started without errors, however, when I rebooted the device, same thing happened as before.

/etc/fstab is indeed read only after booting up. The /boot was not empty, however, /DietPi was. So I did all the steps again (removed the lines in /etc/fstab, rebooted and re-added them). So rn I can start ramdisk service again. Any more clues?

Thanks for all the help.

[MichaIng]

@samolego
Wait, you added the lines to /etc/fstab and saved the file, which succeeded, but then after reboot the root file system was still mounted as read-only?
Is the root file system really ext4? journalctl -u -.mount should give a hint. To have fstab entry compatible with all file systems:

/dev/mmcblk0p2 / auto noatime,lazytime 0 1

File system can be checked via:

df -T

[samolego]

but then after reboot the root file system was still mounted as read-only?
Exactly.

If I do the steps from your previous comment and reopen SSH, system functions normally. But reboot mounts root fs as RO ...

df -T proves that root fs is ext4
```Filesystem Type 1K-blocks Used Available Use% Mounted on
udev devtmpfs 959876 0 959876 0% /dev
tmpfs tmpfs 195316 3156 192160 2% /run
/dev/mmcblk0p2 ext4 12470936 3638992 8178736 31% /
tmpfs tmpfs 976580 0 976580 0% /dev/shm
tmpfs tmpfs 5120 0 5120 0% /run/lock
tmpfs tmpfs 976580 0 976580 0% /sys/fs/cgroup
tmpfs tmpfs 1047552 0 1047552 0% /tmp
tmpfs tmpfs 10240 1360 8880 14% /DietPi
tmpfs tmpfs 51200 8 51192 1% /var/log

Oh, and
journalctl -u -.mount shows "no entries"

-- Logs begin at Wed 2019-12-18 16:29:49 GMT, end at Wed 2019-12-18 16:35:52 GMT. ---- No entries --
```

[MichaIng]

@samolego
After doing the changes, could you please paste again, to assure correct fstab entry:

cat /etc/fstab

[samolego]

Here it is

# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1023M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
#/dev/mmcblk0p2 / ext4 noatime,lazytime 0 1
/dev/mmcblk0p1 /boot auto noatime,lazytime 0 2
/dev/mmcblk0p2 / auto noatime,lazytime 0 1

[MichaIng]

@samolego
Can you run and paste:

systemctl daemon-reload
cat /run/systemd/generator/-.mount
systemctl status systemd-remount-fs

[samolego]

cat /run/systemd/generator/-.mount

# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/dev/mmcblk0p2
Options=noatime,lazytime

systemctl status systemd-remount-fs

● systemd-remount-fs.service - Remount Root and Kernel File Systems
   Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; static; vendor preset: enabled)
   Active: active (exited) since Wed 2019-12-18 16:29:49 GMT; 26min ago
     Docs: man:systemd-remount-fs.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 252 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 2155)
   Memory: 0B
   CGroup: /system.slice/systemd-remount-fs.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[MichaIng]

@samolego

16:29:49

Does this match the boot time or better suits to when you remounted the drive manually?
Also you can check journalctl and check for errors around the initial mount of this drive. systemd-remount-fs is the service that should remount the root file system with R/W access, at least when the fstab entry exists.

R/W should be default, but who knows, to assure it is set, you could adjust the fstab entries: rw,noatime,lazytime

[samolego]

Yeah, the time matches boot time, since I have wrong timezone set up rn :-P.

EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)

First error seems to be with swap:

Dec 18 16:29:49 minecraft-server swapon[263]: swapon: /var/swap: swapon failed: Read-only file system

There seem to be no systemd errors ... will try to reboot with added rw option to fstab
EDIT: problem isn't gone with rw added

[MichaIng]

@samolego
Jep, swap files of course cannot be created on R/O file systems. Okay so for some strange reason, rw seems to be not default in your case. This means, to assure that, we should add this as default to our fresh fstab.

Okay, DietPi boot scripts should then succeeds, including initial firstrun update+install prompts, right?
When done, could you do:

cp -a /etc/fstab /etc/fstab_bak
dietpi-drive_manager 4
cat /etc/fstab

This is what we call on DietPi-PREP to create a fresh fstab based on current attached drives. It "should" contain root and boot mount, however good to test since it was not the case when you ran the script.

If those are missing, revert:

mv /etc/fstab_bak /etc/fstab

[samolego]

So, if I understood correctly - should I reinstall the dietpi with PREP script and after installation procces is finished run those cmds?
Will do it tomorrowm since it's a bit late rn ;-)

[MichaIng]

@samolego
Nope no need to reinstall, if thinks work now. The commands are to test the fstab reset command only, to see if it now does what it should on your system. If not I'd do a closer look into the code.

[samolego]

I'm not sure if I presented it to you correctly - if I reboot the system, dietpi cmds die again, so not sure what you meant by DietPi boot scripts should then succeed

dietpi-drive_manager 4

[ INFO ] DietPi-Drive_Manager | Detecting drives, please wait...
/DietPi/dietpi/dietpi-drive_manager: line 207: blkid: command not found
/DietPi/dietpi/dietpi-drive_manager: line 208: blkid: command not found
/DietPi/dietpi/dietpi-drive_manager: line 213: blkid: command not found
[ INFO ] DietPi-Drive_Manager |  - Detected mounted drive: /dev/mmcblk0p2 > /
/DietPi/dietpi/dietpi-drive_manager: line 304: blkid: command not found
[ INFO ] DietPi-Drive_Manager | autofs4 module not available in kernel, x-systemd.automount has been disabled

cat /etc/fstab

# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs noatime,lazytime,nodev,nosuid,mode=1777
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs size=50m,noatime,lazytime,nodev,nosuid,mode=1777

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------


#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------

[MichaIng]

@samolego

blkid: command not found

There we have the issue, however this is possible. blkid is part of the essential util-linux, which is definitely installed, as otherwise the whole system would be unable to load. Please try to recover this binary: apt install --reinstall util-linux

I'm not sure if I presented it to you correctly - if I reboot the system, dietpi cmds die again, so not sure what you meant by DietPi boot scripts should then succeed

You mean when you add the /etc/fstab entries, after reboot rootfs is still mounted R/O?
After running dietpi-drive_manager 4 of course, since due to missing blkid it fails to add root and boot mounts to fresh fstab. Hence the backup of fstab to restore the entries: mv /etc/fstab_bak /etc/fstab
This was just for testing and now the reason is clear 😉.

[samolego]

You mean when you add the /etc/fstab entries, after reboot rootfs is still mounted R/O?

No, but I had to restart
systemctl restart dietpi-ramdisk

Anyways, seems to be working now. I will try to reboot a few times :-). Thanks so much!

[samolego]

Rebooted a few times successfully! I cannot believe it was a package failure. Thanks for all the help & quick responses. Keep being awesome :-P.

[MichaIng]

R/W flag is now added explicitly to fstab: https://github.com/MichaIng/DietPi/commit/c7b82ca81c380fdc423d342ed33cf487b8dec7dd
Changelog: https://github.com/MichaIng/DietPi/commit/c8243de0333ed9d1d1d81dec73c3da1c650f56e3