Dietpi: Support pre-configuration customisation script

[Jaffa]

Is your feature request related to a problem? Please describe:

I'd like to create a series of appliances based on DietPi. These would have the same dietpi.txt, and ideally be based on flashing the same image.

However, there are some changes that are node-specific. For example, hostname. Changing the hostname in Automation_Custom_Script.sh, even using /DietPi/dietpi/func/change_hostname, doesn't change the hostname for the first DHCP request.

This can mean that the DHCP server can cache the hostname of DietPi (or whatever default is specified in dietpi.txt and result in confusion when its later changed).

Describe the solution you'd like:

It would be useful if there was a way of hooking a script in to the first run, when /DietPi/dietpi/.install_stage was -1 and before boot#Apply_DietPi_FirstRun_Settings is run.

The script can then make device-specific changes to dietpi.txt (based on hardware-specific values, such as serial numbers, MAC addresses, random values) and update dietpi.txt.

Describe alternatives you've considered:

Theoretically, I could make changes to dietpi.txt, reset .install_stage and restart the whole cycle; but that seems inefficient (and fragile).

Alternatively, I could try and see if the DHCP client has got a cached copy of the hostname; but this is just one example. Others might be changing the root password to be based off a serial number printed on a case (if that's readable from the OS).

[MichaIng]

@Jaffa
Thanks for your request.

Jep, the automation custom script is applied after dietpi.txt settings on first run and /DietPi/dietpi/func/change_hostname just has affect after reboot.
And everything is applied after first DHCP/network connection, which would be needed differently in your case. A bid difficult to implement a reasonable method here, since some custom setup steps might require network, some require even more, yours requires especially DHCP/network to be not yet applied 🤔.

I am thinking if a dietpi.txt.d/ folder for overwriting some settings individually would be helpful, like the typical conf.d folders work.

But more thinking...
Why is it actually easier to manually create a script, that changes dietpi.txt entries, than manually changing the entries directly 🤔?

---

But as said, all current DietPi setup steps are anyway applied after first network load/DHCP lease anyway. In my case e.g. this is no issue. The router/DHCP server saves/caches MAC+hostname+IP, but this gets overwritten, as fast as the hostname of the client changes (e.g. after next DHCP lease/reboot).

To solve your specific issue, applying new hostname before first network/DHCP lease, it would require a systemd unit that changes /etc/hostname and /etc/hosts accordingly, e.g.

cat << _EOF_ > /etc/systemd/system/change_hostname.service
[Unit]
Description=Changing hostname before network
After=local-fs.target
Before=network.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo "NewHostname" > /etc/hostname; sed -i "s/DietPi/NewHostname/g /etc/hosts; systemctl disable change_hostname; rm /etc/systemd/system/change_hostname.service"'

[Install]
WantedBy=local-fs.target
_EOF_
systemctl enable change_hostname

Would need to be done during image preparation, the service then could read the chosen hostname from another file or dietpi.txt.
Not sure if using /DietPi/dietpi/func/change_hostname (after dietpi-ramdisk) also works reliable. Would make the service load later, but Before=network.target should still assure that it loads before network/DHCP lease.

[Jaffa]

Why is it actually easier to manually create a script, that changes dietpi.txt entries, than manually changing the entries directly?

The script could be the same on every card, but make changes unique to the device. For example, it could be based on CPU ID, MAC address, board serial number or even a random number. This means I can start off with the same SD card image, but still have each device self-customise at start up.

Thanks for the change_hostname.service. /DietPi/dietpi/func/change_hostname has changed the hostname read by the system, but not on the DHCP lease. Maybe that's my DHCP lease timing options.

A bid difficult to implement a reasonable method here, since some custom setup steps might require network, some require even more, yours requires especially DHCP/network to be not yet applied

One thought to make it more of an extensible approach, would be to do something like:

PRE_BOOT_SCRIPT=/DietPi/Automation_Custom_Preboot_$G_DIETPI_INSTALL_STAGE.sh
[ -f "$PRE_BOOT_SCRIPT" ] && . "$PRE_BOOT_SCRIPT"

Then, in my case, I'd create Automation_Custom_Preboot_-1.sh.

I am thinking if a dietpi.txt.d/ folder for overwriting some settings individually would be helpful, like the typical conf.d folders work.

This'd be really useful separately for upgrading DietPi: there are sometimes changes in the base dietpi.txt that make sense to take, but I want to overlay my custom wifi settings or similar. Having dietpi.txt.d/wifi.txt would mean I can keep the default dietpi.txt.

[MichaIng]

@Jaffa

The script could be the same on every card, but make changes unique to the device. For example, it could be based on CPU ID, MAC address, board serial number or even a random number.

Ah yes that makes sense, thanks for clarification.

Thanks for the change_hostname.service. /DietPi/dietpi/func/change_hostname has changed the hostname read by the system, but not on the DHCP lease. Maybe that's my DHCP lease timing options.

Jep as said, all DietPi scripts currently run after first network access, at least if there is a DHCP server in the network and that is not too slow.
https://github.com/Fourdee/DietPi/blob/master/dietpi/boot does first setup steps (including hostname change as very first step) on first boot and is initiated by: https://github.com/Fourdee/DietPi/blob/master/rootfs/etc/systemd/system/dietpi-boot.service
As this starts After= all network related services, DHCP lease is most likely already done.

The only script that runs before all of this, is the file system partition resize on very early first boot stage: https://github.com/Fourdee/DietPi/blob/master/rootfs/etc/systemd/system/dietpi-fs_partition_resize.service
This is most likely as well earlier than network and we could actively assure it via Before=. But we don't want to touch anything else before the drive is completely ready.

I am thinking if it is possible to allow DietPi-Boot running earlier (explicitly Before= all network related services). We actively stop networking, drop and restart interfaces within the script due to eth/wifi setup. Not sure if this then throws errors, or the other way round if we could skip those steps and only need to edit the related settings files to make networking.service etc. read directly the correct ones.
On first dietpi-software run (runs after first DietPi-Boot execution), we could then adjust the Before=/After= entries to run after network, as it is now and wanted for regular boot.
This would allow as well to implement pre-setup scripts is requests in this topic.

• So finally we would have optional pre-setup scripts, executed within DietPi-Boot on very early boot stage (just rootfs/bootfs/dietpi-ramdisk/ramlog required, explicitly before network), which also allow to adjust dietpi.txt before it is used, based on e.g. hardware values.
• And we already have support for custom post-setup script.
• Sounds round to me, but the former needs careful testing.

[Jaffa]

Ah, I hadn't spotted that dietpi/boot was started by systemd. I hadn't done enough backtracking; I'd _assumed_ that dietpi/boot was the init script (but hadn't checked). OK, hmm, makes sense.

On first dietpi-software run (runs after first DietPi-Boot execution), we could then adjust the Before=/After= entries to run after network, as it is now and wanted for regular boot.

Or, split dietpi/boot into a service for the network setup stages, and one for after. Then, instead of rewriting the service you remove/disable it once you get past that point (or make it exit as a no-op).

Alternatively, is there a way of overlaying new services from the boot partition? (Though, I guess there's no reason I couldn't write to the rootfs too, but that _seems_ messier).

[MichaIng]

@Jaffa
Jep we do not reinvent the wheel and rely on systemd as init system and service management.

Or, split dietpi/boot into a service for the network setup stages, and one for after. Then, instead of rewriting the service you remove/disable it once you get past that point (or make it exit as a no-op).

Jep, has several advantages. Generally I thought about separating first run setup/automation from DietPi-Boot and DietPi-Software, also to reduce the amount of code that is loaded after all setup steps are done. But this means a deeper rewrite and careful testing.

---

About dietpi.txt.d/ also would need some some rewrite, not only for reading settings (would include a global G_READ_DIETPI_TXT function) but also for writing those, what we usually do when user changes them via dietpi-config/software.
But it would allow some cleaner adjustments and separation of settings blocks (e.g. separate wifi/network, automation and e.g. settings related to specific software titles).

Let's see what @Fourdee thinks about all of this.

[Fourdee]

Ok, so 1 issue and 1 feature:

• 🈯️ https://github.com/Fourdee/DietPi/commit/1d714ac6926f200401163a155b20e6c20252f4d3 Issue: Hostname change applied after network online, causes cache on network. Needs to be before
• 🈯️ https://github.com/Fourdee/DietPi/commit/1d714ac6926f200401163a155b20e6c20252f4d3 Feature: Add support for a automated script which runs at install stage -1 (after fs expand, before networking, after initial automation set)

NB: Above changes will not take effect until we recreate the images with updated code, or, user runs PREP on existing installation.

---

Notes:

• Create dietpi-pre-boot which runs before dietpi-boot and networking
  move Apply_DietPi_FirstRun_Settings there
• Add support for running custom script in pre-boot with install stage -1
• We need to leave as much as possible in dietpi-software, as preboot and boot scripts won't be patched until dietpi-software is run. And, we would need to re-do the images each time if an automated/early issue occurs with those scripts.

[Fourdee]

Testing required:

PREP

• 🈯️ /boot/Automation_Custom_PreScript.sh | Run prior to networking
• 🈯️ AUTO_SETUP_TIMEZONE=America/New_York | Run prior to networking
• 🈯️ AUTO_SETUP_LOCALE=en_US.UTF-8 | Run prior to networking
• 🈯️ AUTO_SETUP_KEYBOARD_LAYOUT=us | Run prior to networking
• :u6307: Hostname | Run prior to networking
• :u6307: WiFi automated | Run prior to networking
• 🈯️ Static IP automated | Run prior to networking

[Fourdee]

@Jaffa

Ok, testing passed (https://github.com/Fourdee/DietPi/issues/2050#issuecomment-418973044)

Please let me know which device you intend to use, as, I'll need to update the image to use the latest code.

[Jaffa]

Raspberry Pi (Zero W)

On 7 September 2018 at 10:09, Dan [e-mail] wrote:

@Jaffa https://github.com/Jaffa

Ok, testing passed (#2050 (comment)
https://github.com/Fourdee/DietPi/issues/2050#issuecomment-418973044)

Please let me know which device you intend to use, as, I'll need to update
the image to use the latest code.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Fourdee/DietPi/issues/2050#issuecomment-419375342,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASNBYr3rpLbTlP-BdJ8-2-CaZk_S6SBks5uYjevgaJpZM4WWNfk
.

--
Andrew Flegg -- mailto:[email protected] | http://www.bleb.org/

[Fourdee]

@Jaffa

Image updated, please download:

• https://dietpi.com/downloads/images/DietPi_RPi-ARMv6-Stretch.7z

Changes:

• Hostname applied before any network up which resolves the initial issue
• Few minor tweaks to ordering of pre-network configurations
• Added support for /boot/Automation_Custom_PreScript.sh, which runs pre-network and before DietPi runs setup. Simply enter a bash script there, DietPi will run it pre-boot.

Local testing passed my end, so i'll mark this as closed. However @Jaffa, please let us know if this resolves the issues you are experiencing.

[Jaffa]

Still doing some testing. The script is definitely invoked :+1:

There's a warning on the console, as line 342 of preboot uses local outside of a function :-1:

And booting seems to hang - with some warnings so far about waiting for serial console. I don't know if this is a symptom of my script (don't think so); a symptom of 6.15; or a symptom of something else. Unfortunately, this isn't allowing me to test whether or not the script allows me to achieve what I wanted :-/

I'll be a bit more patient and do so more investigation...

[Jaffa]

[Fourdee]

@Jaffa

There's a warning on the console, as line 342 of preboot uses local outside of a function

Whoopsie, good spot. I'll get that resolved and redo the image today 👍

[Fourdee]

@Jaffa

Can we also have a copy of the /boot/Automation_Custom_PreScript.sh script you are running? Will allow us to duplicate testing.

[Fourdee]

@Jaffa

Image updated, removes the local issue:
https://dietpi.com/downloads/images/DietPi_RPi-ARMv6-Stretch.7z

Confirmed testing passed:

• 🈯️ /boot/Automation_Custom_PreScript.sh

#!/bin/bash
echo 1 > /root/test.txt

• 🈯️ No hang during boot

@Jaffa

If problems persist, please attach the /boot/Automation_Custom_PreScript.sh script which you are using so we can assist with debugging.

[Jaffa]

Watching the full boot, the log runs:

         Starting DietPi-PreBoot...
/DietPi/dietpi/func/dietpi-obtain_hw_model: line 109: cd: HOME not set
[ INFO ] DietPi-PreBoot | Running custom script, please wait...

Looking at preboot, it does:

/boot/Automation_Custom_PreScript.sh | tee $fp_dietpiautomation_custom_prescript_log

This risks losing stderr, so I suggest we want a 2>&1 in there, so that stderr gets captured by tee into the log as well.

I've re-flashed the image and tried it without my Automation_Custom_PreScript.sh and it doesn't hang waiting for a serial console. So it _is_ something my script is doing.

I narrowed it down: my script was accessing /dev/urandom: we've just booted and won't have much in the way of an entropy pool. Changing it to mktemp -u random bytes for a hostname worked better :+1: Sorry for the confusion! The serial console messages must have just have been coming up in the background.

[Fourdee]

@Jaffa

Good stuff, thanks for testing 👍

Starting DietPi-PreBoot...
/DietPi/dietpi/func/dietpi-obtain_hw_model: line 109: cd: HOME not set
[ INFO ] DietPi-PreBoot | Running custom script, please wait...

I'll check this.

[Fourdee]

HOME unset resolved with: https://github.com/Fourdee/DietPi/commit/7e1fcf864d88af3547357e92cccf94e988ebb659

Completed.

@Jaffa

Many thanks for testing, if any further issues should appear in the future, please reopen as required.