Dietpi: 0bytes free space, causes kernel clean/extraction to fail, results in no boot

[Invictaz]

Was trying to update from 146 to 149.
Gives "package.zip" cannot be written or some thing like that
I typed "reboot" after it said 149 was installed.

Now only gives 3x led blip and refuses to boot....

https://twitter.com/W00fer/status/858013761323409408

[Invictaz]

There is no serial output on 115200 so I cannot investigate what is wrong during boot.
Normally I have serial output there.

[Fourdee]

@Invictaz

Thanks for the report 👍

After Dietpi 147>148 it says disk full and refuses to reboot

It sounds like a possible filesystem corruption has occurred on the SD card. Resulting in a read only filesystem at some point to prevent further damage. This is usually caused by failing hardware.

Which:

• SD card used?
• PSU used?
• Any power cuts recently you are aware of?

The 3x LED blips are caused by: http://elinux.org/R-Pi_Troubleshooting#Green_LED_blinks_in_a_specific_pattern

3 flashes: start.elf not found

Which is most likely due to corrupt filesystem on SD card.

I believe the quickest fix is a fresh image write and install. However, if you are able and have time, any chance you are able to read this current SD card to a image file, then upload so I can run some tests?

[Invictaz]

Sdcard used is ADATA 16GB microsd
Psu is Samsung tablet adapter 2 amps.

[Invictaz]

@Fourdee how to make an img of the sdcard. I have various raw img backup programs but don't know which you like best (Windows 7)

[FireHelmet]

@Invictaz, you can use https://sourceforge.net/projects/win32diskimager/ , insert your sd card in your PC and click on read and save your image in your PC.

@Fourdee, Hello Dan, I'm Jérôme (haproxy). I have the same problem on my odroid c2, after the update in 149 from 148, the odroid can't to boot and I have a emmc.

Thank for your support

[WolfganP]

Is this the return of the empty Dietpi folder or the FAT32 partition? https://github.com/Fourdee/DietPi/issues/719
Or is the issue a corrupted /boot partition?
I also have issues on the FAT32 filesystem again (just a few dietpi- commands present) in my RPi2, noticed when trying to upgrade to v149, but I can't say the upgrade is the cause or it came from earlier.

[Fourdee]

@FireHelmet @WolfganP
Hi guys,

Thanks for the report 👍

I'd like to investigate your current installations and try and find out whats triggering this. Any chance both of you could read your SD > image, compress and upload?

Is this the return of the empty Dietpi folder or the FAT32 partition?

We've done everything at our end to prevent this from re-occurring, improving our image creation process. However, in all honesty, i've never experienced any update (or filesystem corruption) issues, on all the boards we support.

I still believe this to be a hardware PSU/SD card issue. But, if we can get hold of your current installations, it may shed some light on another cause.

[Fourdee]

@Invictaz

Sdcard used is ADATA 16GB microsd
Psu is Samsung tablet adapter 2 amps.

Both of those are cause for concern. I very much doubt the tablet PSU will provide a stable 5V under load, if at all.

We recommend SanDisk + Official RPi PSU for ensured system stability and reliability.

[WolfganP]

We've done everything at our end to prevent this from re-occurring, improving our image creation process. However, in all honesty, i've never experienced any update (or filesystem corruption) issues, on all the boards we support.

I know you do everything you can to help users and improve Dietpi, and we're all grateful for it :-)

I still believe this to be a hardware PSU/SD card issue. But, if we can get hold of your current installations, it may shed some light on another cause.

I did believe that as well, but even if I checked my PSU and SDcards, I changed them anyways and similar issues still remain. As for the DietPi folder, I CRC checked all remaining files on the card and the SD for errors and nothing seems wrong (in fact it's the same folder that get's cleaned somehow, but I can't figure out how as this last reinstall I just didn't include any custom app, just proftp, samba and plex)
BTW, any preference on where to share the SDcard image with you?

[Fourdee]

@WolfganP

BTW, any preference on where to share the SDcard image with you?

Anywhere is fine 👍

Might be an idea to password the zip file (if you have personal data) and email link to me directly daniel.[email protected]

[FireHelmet]

@Fourdee , I think it's not related because my emmc is not detected via windows and via sdcard adapter :(. Sorry

[FireHelmet]

So my emmc is dead :(

[Invictaz]

@Fourdee I very much doubt the SD as the system was running fine since v12x (somewhere in the twenties). And yes one day it fails but that could also happen with A-brand sdcards.

As for the PSU where is a report on this? I have seen underpowered Pi's where the accessoires attached to the Pi drew too much power (more than 1Ah). But my Pi was running fine until the update....

Not excessively warm, PSU also fairly cool.

Will make it into an IMG asap.

[DarkElvenAngel]

I had the same issue.
During the update the kernel files are deleted and if the package.zip doesn't fully download ( as was my case ) the update doesn't fail it continues to the end and reports all was well and reboot to complete.

If you don't have enough space to download the kernel update this will cause your issue. I was able to save my install by copying kernel files from another SD card.

The whole cause of this was not lack of space on my SD card. I was in a folder mounted to a ram drive and wget downloads to the current folder.

[Fourdee]

@FireHelmet

So my emmc is dead :(

Gutted, does a full format bring the EMMC back? If so, could be a partition table corruption.

[Fourdee]

@DarkElvenAngel

During the update the kernel files are deleted and if the package.zip doesn't fully download ( as was my case ) the update doesn't fail it continues to the end and reports all was well and reboot to complete.
The whole cause of this was not lack of space on my SD card. I was in a folder mounted to a ram drive and wget downloads to the current folder.

Great find and thanks for letting us know. Think your onto something 👍

We do check connection to the kernel download, prior to starting it. Else, it will abort:
https://github.com/Fourdee/DietPi/blob/master/dietpi/func/dietpi-set_hardware#L434-L461

We do not check for free space during updates.

I believe we could simply cd "$HOME", prior to the update (or place in our scripts by default) to prevent this.

[Fourdee]

@Invictaz

As for the PSU where is a report on this?

Personal experience/testing with Samsung 2A phone charger giving out <4.7V under load:
https://www.amazon.co.uk/d/Electrical-Tools-Testers/Voltmeter-Ammeter-Capacity-Current-Tester/B00NPVZHAO/ref=sr_1_3?ie=UTF8&qid=1493548393&sr=8-3&keywords=usb+power+meter

[WolfganP]

@DarkElvenAngel Great find. Exactly which files where erased from the SDcard on your case?

@Fourdee I'm still puzzled by the /boot/Dietpi files being erased, as this seems not related to upgrade operations. Is there any way to monitor that folder somehow?
In any case, shouldn't a "dietpi-update 1" (force the update) solve any issue?

[DarkElvenAngel]

@WolfganP
/boot/kernel.img
/boot/kernel7.img
/lib/firmware/*
/lib/modules/*
As I mentioned basically the entire kernel. I only assume that the system deletes the old kernel at some point in the upgrade. If this is the case maybe waiting for a successful reboot before clearing these files and renaming the kernel images with .bak would be safer and make recovery faster.

Not 100% sure how your hardware works @FireHelmet but most EEMC require a working kernel module to access them via USB OTG if you can boot your device with an SD card check you should be able to confirm whether or not it's dead

[Fourdee]

@WolfganP

I'm still puzzled by the /boot/Dietpi files being erased, as this seems not related to upgrade operations.

Me and you both. I've never been able to replicate this.

Is there any way to monitor that folder somehow?

Yep, we generate a perm log for RAMdisk:

cat /etc/dietpi/logs/dietpi-ramdisk.log

In any case, shouldn't a "dietpi-update 1" (force the update) solve any issue?

A standard update will pull in the latest scripts and replace all missing in boot partition, however, if you are missing any /boot/dietpi/.* settings/state files, DietPi will most likely reset to a state of "fresh install". It needs those files for installed state, dietpi-software installed/options etc.

[WolfganP]

Thanks @Fourdee

Me and you both. I've never been able to replicate this.

I strongly think is not a filesystem corruption of sorts, the deletion seems to be surgical as not all files are erased inside the /boot/dietpi folder, and I didn't find any files missing outside that folder nor FS/media errors on SD integrity checks on FAT32 nor ext4 partitions.

A standard update will pull in the latest scripts and replace all missing in boot partition, however, if you are missing any /boot/dietpi/.* settings/state files, DietPi will most likely reset to a state of "fresh install". It needs those files for installed state, dietpi-software installed/options etc.

As those files are so important in terms of system start, maybe is worthy to incorporate a rolling backup of them in a separate folder after the 1st start configuration takes place? (as you said, dietpi executable scripts can be replaced by a forced update or wget anyways; ie dietpi-sysbackup)

[Fourdee]

@DarkElvenAngel

As I mentioned basically the entire kernel. I only assume that the system deletes the old kernel at some point in the upgrade. If this is the case maybe waiting for a successful reboot before clearing these files and renaming the kernel images with .bak would be safer and make recovery faster.

Yep, we clean current kernel from system, before replace. Ensures stuff like /lib/modules dont get left behind.

Kernel is now overwritten without cleanup, to prevent failed unzip rendering system unbootable.

[WolfganP]

Glad to read you liked the idea @Fourdee. Mulling further on it, I lean towards a /boot/dietpi.sysbackup folder where the state files are stored in separate directories (ie config1, config2, ...) acting as restore points of sorts (as everything else boot related can be reloaded even manually)

BTW, /etc/dietpi/logs/dietpi-ramdisk.log doesn't provides too much information, as the entries aren't timestamped (excerpt below).

DietPi-Ramdisk: Completed
DietPi-Ramdisk: Stopping
DietPi-Ramdisk: Error - /DietPi/dietpi/boot is missing, aborting...
DietPi-Ramdisk: Completed
DietPi-Ramdisk: Starting

[Fourdee]

@WolfganP

Ok, so the missing boot files may be due to:

• reboot
• DietPi-RAMdisk then removes existing /boot/dietpi/* scripts, so we dont need to manually rm deprecated scripts, before copying over the current ones from /DietPi to storage.
• sync is called to ensure disk data is saved, but on your recent installation, some files didn't copy successfully for unknown reasons.

So, lets disable clearing of /boot/dietpi/* prior to DietPi-RAMdisk shutdown. I think this may prevent the loss of files, but we'll need to see over time.
Then if the same occurs, when you reboot, your system "should" be in its previous state, before update.

I'll also add a check for failed unzip on dietpi-update.

BTW, /etc/dietpi/logs/dietpi-ramdisk.log doesn't provides too much information, as the entries aren't timestamped (excerpt below).

We like minimal logging, where possible. But i'll add time-stamps in aswell.

[Invictaz]

@Fourdee My mistake it's an Optima 4GB class 4 card.

Since it's 4GB how to compress it?
Even after compression with Winrar it's still 945 mb. It takes an absolute age to upload that over DSL to the filesharing sites or my FTP host.

Checkdisk on Windows did not find any errors.

[Fourdee]

@WolfganP

I lean towards a /boot/dietpi.sysbackup folder where the state files are stored in separate directories (ie config1, config2, ...) acting as restore points of sorts (as everything else boot related can be reloaded even manually)

Yep its a good idea, but I think rather try and prevent the issue occurring, instead of relying on undocumented, additional backups in case of failure.

A current solution would be to use dietpi-backup, creating a full system backup that could be restored from. Boot + scripts are then in:

/mnt/dietpi-backup/dietpi-backup_system/boot/

Either way, once you've mulled over it :) , let me know and we'll discuss.

[Invictaz]

@Fourdee I have a backup but since I cannot boot into the thing I cannot recover that...
And backups are automatically stored on the same SD card so yeah.
The filesystem is readable in Windows though so I have no idea what is wrong...

[Fourdee]

@Invictaz

Even after compression with Winrar it's still 945 mb. It takes an absolute age to upload that over DSL.

7zip would give you the best compression, after that threes not much else we could do outside the image file itself.

I have a backup but since I cannot boot into the thing I cannot recover that...

If you can upload image, i'll see if we can recover.

[ghost]

yep, same issue here, even tried a new image write to sd card, but after initial boot, update and restart, nothing happened. can't wait for v150 :)

[FireHelmet]

@DarkElvenAngel and @Fourdee ,

After some investigation at my side, my emmc is readable (boot partition is present) only on another PC in my home, strange issue because I used the same sd-card adapter to read a sd-card without problem on my personal PC...

But I'm still with the same error like described in this issue. I can't upload my image because I have some password not encrypted in my conf....xD...this is my next enhancement after new install of v150 :p

[Invictaz]

@Fourdee

Here is the image I recovered from the SDcard that won't boot. It's 834 MB in 7ZIP (LZMA2)

https://w00fer.stackstorage.com/s/dn36KoGhgL4Aw0G

[Fourdee]

@Invictaz

Here is the image I recovered from the SDcard that won't boot. It's 834 MB in 7ZIP (LZMA2)

Thanks 👍

DietPi scripts/dir looks intact, seems like missing kernel as per @DarkElvenAngel. Missing is:

• /lib/modules
• /lib/firmware
• kernel.img + 7.img

🈯️ Cause is due to 0bytes free space remaining on disk (rootfs), means kernel extraction failed.

@DarkElvenAngel was spot on with this 👍
This wont occur after v150 (Even if you have 0 free space) as we no longer clean existing kernel, prior to extraction.

@Invictaz
edit:
now boots, lemme zip this up for you. 700KB/s upload 👎 10~ mins

Please look into upgrading your SD card size, and/or maintaining total disk usage on your system.
You can use df -h to quickly check free space.

[Fourdee]

Notes, whats done for v150:

• 🈯️ Kernel upgrades and dietpi-update no longer clean existing kernel/scripts prior to unzip/replace of latest. Even if 0free space, system will continue to be "operational" after a reboot.
• 🈯️ All scripts cd into home directory, to prevent installs/unzips etc on tmfps.
• 🈯️ dietpi-update dietpi-software check free space, prior to allowing update/installs: https://github.com/Fourdee/DietPi/issues/905#issuecomment-298612799

[Fourdee]

@Invictaz

Your fixed image:
http://dietpi.com/downloads/testing/failed_fixed.7z

[FireHelmet]

@Fourdee,

Do you have a "howto" to solve the issue ?

Thank you !

[Invictaz]

@Fourdee
Thanks for fixing it quite fast!

5 questions:
-Can I simply write that image back do SD?
-Is it updated to v149 now?
-If you don't update the kernel when size is 0 how can it say update complete then?
-Is a 4GB SD card not enough?! I had almost nothing installed :(
-Can you make external backup location available i.e. other usb stick or clouddrive?

Will check with Voltcraft Multimeter the power under load.

@DarkElvenAngel thanks for replying in this issue. The newest fixes prevent this from happening again.

[Fourdee]

@Invictaz

Can I simply write that image back do SD?

Yes, although, i'd use a spare SD just incase.

Is it updated to v149 now?

Untouched.

-Is a 4GB SD card not enough?! I had almost nothing installed :(

DietPi will run on a 1GB card, but, the user must maintain the system and filesystem usage.

Can you make external backup location available i.e. other usb stick or clouddrive?

You can, in dietpi-drive_manager setup the drive, then select list location from dietpi-backup

If you don't update the kernel when size is 0 how can it say update complete then?

I'll get that fixed 👍

[Invictaz]

So if I try to update again it fails again? Then I'm quite stuck. Better load this, export the backup and then refresh this SD and start with a clean 149.

[FireHelmet]

@Fourdee ,

Dan, sorry to insist, could you publish a fix like a procedure to execute to solve this issue ? Or is it necessary to start from scratch with the next v150 ?

Thanks

[ghost]

kind of stuck here too. Cannot seem to install the current download version, it auto updates and then doesnt reboot. Or did I miss a fix for that?

[Invictaz]

Seems this issue is bigger than a quick fix. Waiting for v150 isn't going to help some users as there devices are crashing now.

@FireHelmet replacing the files above (kernel etc) onto the SD would solve it?

[FireHelmet]

@Invictaz ,

I will backup my installation and I will test to transfert all files in boot partition of a new install to my boot parition of my emmc.

Thanks !

[ghost]

after a couple of times of reinstalling on a smaller SD card (1GB, I normally use 2GB) it suddenly worked again. I installed Pihole and PiVPN and kept working altogether. Only the card is almost fully used, at 99%. I will try to create an image of it, and write the image to a bigger SD card later today.

[Fourdee]

@FireHelmet

I will backup my installation and I will test to transfert all files in boot partition of a new install to my boot parition of my emmc.

If this is missing kernel files only, do not overwrite DietPi scripts in these locations, else, effective reset of DietPi install state and settings:

• /boot/dietpi.txt
• /boot/dietpi/*

[Fourdee]

@motoboon

I installed Pihole and PiVPN and kept working altogether. Only the card is almost fully used, at 99%

We upped the swapfile to "1GB - total RAM" recently. To prevent 256/512MB RAM users getting out of memory errors.

You can check/change size in:

• dietpi-config > advanced options > swapfile

[Fourdee]

Will change to 500MB min limit:
🈺 This will render DietPi not install-able on 1GB cards. 2GB min now required.

[Invictaz]

@Fourdee

I still don't see why my 4GB card is too small when 2GB is already suitable. I have installed only LXDE and some other small things...

[ghost]

I copied the image from 1GB to 16GB card, then expanded the partition. Works fine and have 15GB free space left.

[Fourdee]

@Invictaz

I still don't see why my 4GB card is too small when 2GB is already suitable. I have installed only LXDE and some other small things...

A dietpi-backup will double the size of used space, as this is a duplicate of your current install + user data (if selected). Try backing up to a USB drive instead. Aside from that you will need to find out what is using up your drive space.

To list used space by file/folders in the current directory, try:

cd /
treesize

Then to check /etc

cd /etc
treesize

[Invictaz]

Is it possible to retry updating if there is enough space available?
After check of https://github.com/Fourdee/DietPi/issues/913

[Fourdee]

@Invictaz

After check of #913

That wont be available until v150 is completed and released: https://github.com/Fourdee/DietPi/milestone/52

Either way, as the original update failed, you'll need to roll back your version to ensure the update is completed successfully this time for v149.

Do the following:

echo 148 > /DietPi/dietpi/.version
dietpi-update

[Fourdee]

I believe we can now mark this as resolved for v150.

The changes in https://github.com/Fourdee/DietPi/issues/905#issuecomment-298343932 will prevent loss of kernel in 0byte free space situations, and, now prevent dietpi-update and dietpi-software if less than 500MB of free space is available.

@WolfganP
These changes should finally resolve the missing DietPi script files aswell, as we no longer clean them prior to extraction 😄

Thanks to everyone for their input on this one. Especially @DarkElvenAngel for spotting the main cause.

[Invictaz]

@Fourdee Will rollback first. When is v150 due?

[Fourdee]

@Invictaz

When is v150 due?

1/2 weeks. Depends on how much available time I have, and how long outstanding jobs take: https://github.com/Fourdee/DietPi/milestone/52

[Invictaz]

@Fourdee sorry to interrupt again but it says 2.4G used and thus more than 900mb free. Still unclear why it cannot update the kernel

[Fourdee]

@Invictaz

it says 2.4G used and thus more than 900mb free. Still unclear why it cannot update the kernel

Did you run the update as per: https://github.com/Fourdee/DietPi/issues/905#issuecomment-298900557 ?

If so, the update should of been completed successfully, check kernel version with:

uname -a

And DietPi version should be v149 after the update.

[Invictaz]

Kernel is 4.4.50 not 4.9. Will that be possible for PiZero? Even after failed update it says 149...

[Fourdee]

@Invictaz

DietPi RPi kernel current we use is 4.4. We'll update to 4.9 sometime in the future.