Dietpi: backup dietpi system as a .img file

[msongz]

so far the dietpi-backup will create a data folder for restore.
I will prefer it could create a .img file and I can copy it out and restore to SD cards with etcher tools.

[Fourdee]

@msongz

Hi,

Thanks for the suggestion. 👍

backup dietpi system as a .img file

Love the idea, but its not possible on a live rootFS system.

The solution is to use a program such as win32diskimager (or linux dd) and read the SD card to image file.

[msongz]

I am finding the best backup solution.
now I am using a card reader, read the sd card on Mac or Linux, and dd it out.
but the .img file was so large, the size depends on my sd card capacity.
and I using https://github.com/Drewsif/PiShrink to shrink down the size, for example, 7GB -> 2GB
but it is a boring job.
has not found a better way.
:(

[MichaIng]

@msongz
You can reduce the image size manually from the linux system, before running dd via zerofree package.

apt install zerofree
zerofree -v /dev/mmcblk1

This will fill empty space with zeros. I think the tool you used does nothing different.

But note that this are much writes to the SDcard, which you usually want to avoid to not reduce life time. I am not sure if zerofree can also be applied to the already created .img to keep those writes on HDD?

[Fourdee]

@msongz

I use this, requires a linux desktop, but allows to shrink a saved .img
Change these accordingly:

    IMAGE_FP='/mnt/samba/#Backups/_Daniel/Projects/DietPi'

    IMAGE_NAME='DietPi_v6.12_Rock64-ARMv8-Stretch'

Script

#!/bin/bash
{

    #check packages are installed
    if (( ! $(dpkg -l | grep -ci -m1 'gparted') )); then

        apt-get install gparted gpart zerofree -y

    fi

    #-------------------------------------------------------------------------------
    #MUST LEAVE 50MB+ space or automation autologin may fail due to 0 free space!!!
    #-------------------------------------------------------------------------------

    RK_SINGLE_IMG=0 #if 1, disable #IMAGE_NAME as needed.

    IMAGE_FP='/mnt/samba/#Backups/_Daniel/Projects/DietPi'

    IMAGE_NAME='DietPi_v6.12_Rock64-ARMv8-Stretch'
    IMAGE_NAME+='.img'

    INDEX_BOOT_PART=1 #rock64 = 6 # comment out for 1 FS images and set INDEX_ROOTFS_PART=1
    INDEX_ROOTFS_PART=2 #rock64 = 7

#ROCK 64 after reading the image AND again after shrinking
#to fix:
#   GPT PMBR size mismatch (4458495 != 15523839) will be corrected by w(rite).
#   15523806
#RUN
# gdisk "$IMAGE_FP/$IMAGE_NAME"
#   w | y
#

    if [[ ! -f $IMAGE_FP'/'$IMAGE_NAME ]]; then

        read -p "No file found..."
        exit

    fi

    #resize2fs /dev/loop2p7 500M
    #dumpe2fs -h /dev/loop2p7 | grep 'Block count'
    #Block count * Block size / 1024 / 1024 =MB?

    modprobe loop
    losetup -f
    losetup /dev/loop2 "$IMAGE_FP/$IMAGE_NAME"
    partprobe /dev/loop2

    if (( $RK_SINGLE_IMG )); then

        e2fsck -f /dev/loop2

    else

        if [ -n "$INDEX_BOOT_PART" ]; then

            e2fsck -f /dev/loop2p$INDEX_BOOT_PART

        fi
        e2fsck -f /dev/loop2p$INDEX_ROOTFS_PART

    fi

    #Optional, copy
    # mkdir -p /mnt/loop2p1
    # mkdir -p /mnt/loop2p2
    # mount /dev/loop2p$INDEX_BOOT_PART /mnt/loop2p1
    # if (( $RK_SINGLE_IMG )); then
    #mount /dev/loop2 /mnt/loop2p1
    # fi
    # mount /dev/loop2p$INDEX_ROOTFS_PART /mnt/loop2p2

    # echo -e "\ncopying data to $HOME\n"
    # rm -R $HOME/loop2p1
    # rm -R $HOME/loop2p2

    # cp -R /mnt/loop2p1 $HOME/
    # cp -R /mnt/loop2p2 $HOME/
    # rm -R $HOME/loop2p1/lost+found
    # rm -R $HOME/loop2p2/lost+found

    sync

    umount /mnt/loop2p1
    umount /mnt/loop2p2
    umount /mnt/loop2

    #Resize partitions
    echo -e "\Resize partitions\n"
    gparted /dev/loop2

    sync

    #Mount for file changes, if required
    mkdir -p /mnt/loop2p1
    mkdir -p /mnt/loop2p2
    if (( $RK_SINGLE_IMG )); then

        mount /dev/loop2 /mnt/loop2p1

    else

        if [ -n "$INDEX_BOOT_PART" ]; then

            mount /dev/loop2p$INDEX_BOOT_PART /mnt/loop2p1

        fi
        mount /dev/loop2p$INDEX_ROOTFS_PART /mnt/loop2p2

    fi

    # echo -e "\nCopying data back\n"
    # cp -R $HOME/loop2p1/* /mnt/loop2p1/
    # cp -R $HOME/loop2p2/* /mnt/loop2p2/

    read -p "Partitions mounted to /mnt/loop2px, make changes to files if required..."

    sync

    umount /mnt/loop2p1
    umount /mnt/loop2p2

    #Resize partitions
    # echo -e "Resize partitions"
    # gparted /dev/loop2

    # sync

    read -p "If failed press CTRL+C to exit and prevent further action, else, press any key to continue..."

    if (( $RK_SINGLE_IMG )); then

        zerofree -v /dev/loop2

    else

        if [ -n "$INDEX_BOOT_PART" ]; then

            zerofree -v /dev/loop2p$INDEX_BOOT_PART

        fi
        zerofree -v /dev/loop2p$INDEX_ROOTFS_PART

    fi

    losetup -d /dev/loop2

    if (( $RK_SINGLE_IMG )); then

        END_PARTITION='450M'

    else

        END_PARTITION=$(( $(fdisk -l "$IMAGE_FP/$IMAGE_NAME" | grep ".img$INDEX_ROOTFS_PART" | awk '{print $3}') + 1 ))
        END_PARTITION=$(( 512 * ( $END_PARTITION + 128 ) )) #34 bytes for secondary GPT + safety net

    fi

    truncate --size=$END_PARTITION "$IMAGE_FP/$IMAGE_NAME"

    echo -e "\nCleaning up $HOME/loop2px\n"
    rm -R $HOME/loop2p1
    rm -R $HOME/loop2p2

    read -p "Done, press any key to continue..."

    exit

}

[msongz]

@Fourdee nice script!
but unfortunately, I did not have a Linux desktop to test.
thanks you so much~
hope dietpi-backup will support backup .img file in the future.

[MichaIng]

@msongz
As explained, you can never create a real .img file from within the running system. A disk image means, that the raw physical disk content (inodes, free space, ...) is part by part copied and written into the .img. On a running system, there will be always changes. Imagine some background job adds a log entry to the disk. The related log file then is rewritten and by times moved from one physical location on the disk to another. If some script is reading from physical start of disk to physical end, it could read thus the same inode (file) two times. It is impossible to create a consistent image by this, AFAIK. For this reason image creation tool will never allow to run on a r/w mounted disk.

What dietpi-backup instead does, is reading the actual hard links of the files to the file system structure, which is the usual way you access files. The physical file (inode) location on the underlying disk space is not recognized by this. Accessing the files on the file system means that you navigate through a "virtual" file structure, that has nothing to do with the actual physical location of the inodes on the disk.

This is also the reason, why the .img file you created is much larger than the dietpi-backup. Physically on the disk, when removing a file, only the hardlink to the file system is removed, thus it disappears from file manager. But the underlying inode will stay (to only do actual read/writes when free space is really needed). zerofree is a tool that clears that unlinked inodes on the raw disk, thus reduces image size. As this is again raw disk writing, zerofree denies to run on a mounted drive.

Long story short, dietpi-backup will never be able to create image files from it's own running system. For this you will always need to attach the disk/sdcard to another machine and create the image there while it's not mounted. Hope I could make it a bid clear? 😃

[msongz]

@MichaIng man! thanks for your detail explanation!
I am clear now and will do the backup using zerofree!

[echable9999]

Interesting reading this, I had been hoping DietPi would incorporate the creation of a .img file running from within it's own menus, with programs installed etc., but you are saying it is impossible because the file system is too dynamic and needs to be no R/W while the process of cloning the SD card is being done.

I have been wanting to clone my configured sd-card with an exact image on another card so if something corrupts all I have to do is switch to the other card. It is a system where I have only a music server setup and there are not many changes except new files in database updates. I managed in the end creating an exact .img, I think I used Win32Imager, and BalenaEtcher. See my thread on Dietpi forums.

When Balena Etcher flashed the cloned .img onto the backup SD card, it took much longer than usual for a DietPi SD, because it wrote the full 32GB of the card (including free space, being written and counted). However, after I started using the cloned card, it reported disk usage of something like 7GB total, identical to the original card.

[MichaIng]

@echable9999
You have to understand the difference between a file/data backup/copy and a drive clone.

A file/data backup copies only the contained data files, which can be done relatively safely from a running system when write-intense services/processes are stopped as much as possible, and which is what dietpi-backup and dietpi-sync do. But for a data recovery, you need a running system as well or you would need to carefully prepare a fresh partition table and partitions on the drive first.

However when you want a compete ready-to-boot drive/SDcard, you need a full drive clone which contains the partition table, partitions and file system definitions as well. Since one usually does not know the physical location of the data on the disk, this means that the whole drive, including empty space, needs to be cloned block-by-block (bit-by-bit). So a clone will at first always have the same size as the whole drive. One could stop cloning at the block where all data files have been handled but due to usual empty space between the data this is difficult to know.

To cover this, for our own images, we created the dietpi-imager script (see .meta/dietpi-imager on Git code). This can be used to create an img file from a pluggrd drive/SDcard (on external DietPi system), and it at first reduces root partition and file system size to the possible minimum which still contains all data files. It then uses dd to write the img file but stops with this after the root partition end has been reached. It also creates a ready to upload 7zip achieve with DietPi image naming, which then consumes much less space again.

When using this script one should enable the dietpi-fs_partition_resize service (on the still running system, before), so partition and file system are automatically expended when flashing the image back on first boot.

This also has the advantage that you can flash the image to smaller SDcards/drives.