DietPi-PREP | Install required APT packages first, autoremove afterwards

[pitsi]

Required Information

• DietPi version: 6.22... I assume
• Distro version: debian buster (debian 10)
• Kernel version: Linux testvm 4.19.0-4-amd64 #1 SMP Debian 4.19.28-2 (2019-03-15) x86_64 GNU/Linux
• SBC device: Virtual machine on vmware workstation 15.x with 2 cores, 20GB of space, 2GB of ram
• Power supply used: Not applicable
• SDcard used: Not applicable

Please check the first image below on the scripts info about the system, in case the above info is incorrect.

Additional Information (if applicable)

• Was the software title installed freshly or updated/migrated? Installed freshly.
• Can this issue be replicated on a fresh installation of DietPi? Yes.

Steps to reproduce

1. Install debian 10 (installer is on the rc1 version at the time of writing) on a regular system or on a virtual machine.
2. Log to it locally, not via ssh, and preferably as root.
3. Follow the instructions described on #1285
4. Unrelated to the overall behavior, but select wireless support when asked by the script for wired only or wired+wireless support.

Expected behaviour

The script is supposed to finish and give me a dietpi system.

Actual behaviour

The script fails to finish because halfway through, after the removal of some packages, because network connection drops completely, fails to install some new packages because it can not download them, and exits with an error at the end

Extra details

Some pictures

and several pages later

I am 99% sure it happens because the script "changes" apt's manually installed packages to auto installed ones, so it can remove them later on. And when the removal is done, essential packages like iproute2 and net-tools are removed, so networking stops.
It does not come up after a reboot too.

[MichaIng]

@pitsi
Many thanks for your report. Hmm I just created two Debian 10 (Buster) VM images without any issue. But perhaps some dependencies changed meanwhile. We mark all packages auto, but we mark especially resolvconf as manually before doing autoremoval. resolvconf depends on ifupdown which again depends on iproute2, so all relevant network packages should stay in place. I just checked, nothing changed about this: https://packages.debian.org/buster/resolvconf

If you still have the terminal output of the DietPi-PREP run, it would be interesting to see which packages really got autoremoved.

---

Btw:

• Which base image did you use? Debian CD/DVD or the installer (e.g. mini.iso, which we use)?
• Also if you want to use VM on Buster, you can use our Buster images. For VM those exist on download page, marked as experimental. Or see: https://dietpi.com/downloads/images/
• And actually why do you select WiFi support? All VMs have virtual Ethernet adapters only, or is there a way to add a virtual WiFi adapter to VMware? Would be actually nice for testing purpose, not useful for production of course. (EDIT: Ah I see you only tried both, because of connection loss, right?)

[pitsi]

First things first, I tested it on a vm because I want to try it on a regular one with limited storage sometime soon, i.e. when debian 10 (buster) reaches stable. And if possible, port some of the script's parts to i686, for the same reason, since i686 is not supported officially.
That is why I installed it from scratch on a vm and did not download the premade vmware disk and that is also why I enabled wifi support.

For the installation, I used the netinstall iso from here, as I always do when installing debian. I will try mini iso now that I know you use it.
https://cdimage.debian.org/cdimage/buster_di_rc1/amd64/iso-cd/

You can add an external wifi adapter to a vm, provided that the adapter is usb connected to the host os.

As for the log of dietpi-prep, do you mean the stuff that is shown on the pics? How can I get the full output of it?

p.s. I won't be having access to the virtual machine until Tuesday, if not later, due to the Easter holidays here in Greece, so please be patient.

[pitsi]

Instead of letting the days pass until Tuesday, would it help if I made a debian 10 (or even 9) installation in a vm and just run the
apt-mark auto $(apt-mark showmanual)
command there and post what it marks for removal?
It will be an i686, however, because my home pc can not do 64bit vms.

[MichaIng]

@pitsi

apt-mark auto $(apt-mark showmanual)

Of course when only running this, it will purge everything but "essential"/"required" marked packages (marked by the Debian APT system itself, outside of DietPi), so including all network related things.

The important step of our script that should prevent removal of any required (required by DietPi + base system + network as well) is: https://github.com/MichaIng/DietPi/blob/dev/PREP_SYSTEM_FOR_DIETPI.sh#L854

apt-mark manual ${aPACKAGES_REQUIRED_INSTALL[@]}

This includes resolvconf (https://github.com/MichaIng/DietPi/blob/dev/PREP_SYSTEM_FOR_DIETPI.sh#L668) which again includes ifupdown and iproute2 as of above mentioned APT dependencies.

You can add an external wifi adapter to a vm, provided that the adapter is usb connected to the host os.

Ah that makes sense. Indeed interesting for testing reasons.

As for the log of dietpi-prep, do you mean the stuff that is shown on the pics? How can I get the full output of it?

Not the whiptail prompts, but the terminal output itself. Easiest is most properly to access the VM via SSH and run DietPi-PREP via SSH session. Then copy&paste all SSH terminal output.

---

And if possible, port some of the script's parts to i686, for the same reason, since i686 is not supported officially.

Yeah an old topic to allow x86_32 systems. In general it should work when adding the related architecture output (uname -m) to three core scripts. The larger task is to go through the DietPi-Software titles and enable/disable them accordingly for this architecture. This is important since not all APT packages nor all custom downloads/binaries are available for i386. Adding correct links, where available to download/install correct i386 binaries/archives then the last step.

[pitsi]

Well, I just learned something I did not know (the packages in the "PACKAGES_REQUIRED_INSTALL" set)...

Because ssh access requires some extra setup, which is tedious on a windows pc, and it will probably stop working once the network drops, can I somehow store the output of the script somewhere?

[MichaIng]

@pitsi
You mean using a Windows system as SSH client is tedious?
Use PuTTY: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

• Start putty.exe enter the local IP of the DietPi or hostname and clock Open. Very easy 😄.
  _- In case not yet installed, on the server install Dropbear or OpenSSH is not yet installed: apt install dropbear-run_

The SSH connection will drop of course if network on the server drops, but the output messages until then should already give a good hint.

From local terminal you could do something like: ./PREP_SYSTEM_FOR_DIETPI.sh | tee dietpi-prep.log
But the colour codes look ugly in the log file.

[pitsi]

There you go, output from the ssh client
https://pastebin.com/CVrsqjXb

As I expected, the connection was dropped right after the removal of ifupdown.
The paste will be available for 30 days and, if needed, I can replicate it with a debian 9 installation, tomorrow.

As for my previous comment. Maybe "tedious" is not the right word for what I wanted to say. The only tedious part so far is that I have to make a fresh debian installation every time and partialy destroy it 2 minutes later.
Plus, the owner of the windows pc that hosts the vms that I do the testing has asked me not to install extra apps on windows, and I can't blame him for that.

[MichaIng]

@pitsi
Okay thanks for this. Jep I have an idea how this happens. Please manually add ifupdown to the script:
sed -i "/\'resolvconf\'/a\'ifupdown\'" diet.sh

I anyway planned to switch how the script handles APT packages:

• Installing our required packages first.
• Auto-removing afterwards.
• By this it is assured that dependencies are not autoremoved first and installed afterwards again or break things like in your case.

[pitsi]

All this command does is change 'resolvconf' on line 651 to 'ifupdown' and nothing more, right? I will manually edit the file, because I think I will mess something up with all those apostrophes and slashes of sed. I will test it in a few hours and report back.

[MichaIng]

@pitsi
It does not change/replace the line, but adds 'ifupdown' as a new entry.

Yeah without SSH (copy & paste) please add it manually. You see the format of the other array entries there.

[pitsi]

Adding ifupdown there made the script end successfully. Thank you for your support :)
And if I understand correctly, the script first installs that is mentioned below line 643 and then removes what is marked as manually installed, from line 612?

[MichaIng]

@pitsi

And if I understand correctly, the script first installs that is mentioned below line 643 and then removes what is marked as manually installed, from line 612?

• apt-mark auto marked packages are autoremoved from APT via apt-get autoremove, if no manually marked package is installed the depends on it.
• apt-mark manual marked packages are never autoremoved as well as all their dependencies.
• And there are some system core packages ("essential" and "required") that are never autoremoved.

To clean up installed APT packages from the system, we first mark all packages via apt-mark auto, then mark only the ones we require via apt-mark manual and then run apt-get --purge autoremove.
The problem is that if not yet all packages that we require are installed, their dependencies are autoremoved as well (resolvconf => ifupdown in your case).

Currently we install missing required packages after the autoremoval. But we need to do that first to avoid autoremoval of any dependencies. I already observed earlier that in cases packages are autoremoved first and reinstalled afterwards again. This of course is an unnecessary overhead and additional disk I/O that we should avoid.

[pitsi]

I just noticed that v6.23 was released, so I tested it on a fresh vm. However, nothing has changed yet, the issue is still present, exactly as I described it at the beginning, so I am leaving it open.

[MichaIng]

@pitsi
Jep, I moved this to v6.24 milestone.

[pitsi]

I was on my way home when 6.24 was relased, so please wait until tomorrow (if not the day after tomorrow) for me to test it again.

[MichaIng]

@pitsi
v6.24 is not yet released, v6.23 was released just on Sunday. Nothing there to test yet. I will see if I can do the rework next weekend and inform you. Thanks for offering testing 🙂.

[pitsi]

I won't be able to test v6.25 (or whichever gets the fix) once it comes out. I no longer have access to the vm I had used so far, because... I lost my job and I had to delete that vm, along with all my files.
On top of that, the guy that lend me a hand by letting me use his pc the first time will be away for the entire summer, so any testing on his pc can not be done by September or October.

Sorry for the bad news guys :(

[MichaIng]

@pitsi
Sorry to read that. No worries about DietPi testing, I hope you find a new, even better job soon.

Topic still on the ToDo and I hope we can make it for v6.25.

[MichaIng]

I just fixed the ifupdown autoremoval (release later today), but rework as topic header is still on ToDo for v6.26.

[Snurring]

Hi,
I've made some attempts installing DietPi on my Beaglebone Black SBC using the prep script.
The installation failed in a similar fashion described by pitsi.

Required Information

DietPi version: 6.25.3
Distro version: Debian 9.5 2018-10-07 4GB SD IoT (Latest image from beagleboard.org)
Kernel version: 4.14.71-ti-r80
SBC device: BeagleBone Black
Power supply used: 5V/15W
SDcard used: 4GB

Please check the first image below on the scripts info about the system, in case the above info is incorrect.
Additional Information (if applicable)

Was the software title installed freshly or updated/migrated? Installed freshly.
Can this issue be replicated on a fresh installation of DietPi? N/A.

Steps to reproduce

Log to it locally, as root.
Follow the instructions described on #1285
Modify the prep-script by adding 'ifupdown' before running the script

Expected behaviour

The script is supposed to finish and give me a dietpi system.
Actual behaviour

The script fails on step 4 - APT installation, after the removal of packages and purging of configuration files, because network connection drops completely.

Extra details
I could not get the suggested sed command to work until I removed the double quotes.
This also was the case on my freshly installed DietPi (v6.25.3) on a RPi 3 which i used for modifying the prep-script. (I also had troubles installing the lighttpd web-server on it with the DietPi installer, but this went ok using apt-get.)

I've attached the log file (tee output) and the modified script ('ifupdown', L749) for reference.

dpi-prep.log

dpi-prep.sh.txt

[MichaIng]

@Snurring

EDIT, at assure that the interface is indeed called eth0: ip a On modern systems it's often enp1s1 or similar. In case replace eth0 below with the actual name.

Please try the following prior to running the script:

apt install resolvconf ifupdown isc-dhcp-client
cat << _EOF_ > /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
_EOF_
systemctl enable networking
ifup -a
apt purge dhcpcd5
systemctl disable connman
reboot

• After reboot connman should not establish network, since it's disabled, but networking.service (ifupdown) should instead establish it via DHCP which also sets the nameserver through resolvconf hooks.

I hope that does the trick. Same hopefully works for netman (systemctl disable NetworkManager): https://github.com/MichaIng/DietPi/issues/2054

[Snurring]

Tanks. I'll make an attemt next week (not at home atm.).
The beagleboard software uses the legacy udev/systemd naming scheme, so the ethernet if is eth0.
Originally, the boards were shipped with the Ångstrøm Linux distro, but today the beagleboard.org provided image downloads are debian based. Included is a cloud based development software suite, beaglebone 101.

[Snurring]

Ok. I have done some testing and finally got the PREP-script working.
What you suggested didn't quite work, but I was able to make a script that works based on it.
BBB/Debian login is as regular user (sudo su for root) so I decided to stay in user mode and sudo my commands.

First i had to do a repo upgrade as apt reported:
"Package 'resolvconf' has no installation canidate".
'ifupdown' and 'isc-dhcp-client' was already installed in my system so I skipped them.
'dhcpcd5' was not installed so no need for purging.
I made the /etc/network/interfaces file edit "scriptable" by using sed.

The script I made seemed working, but the PREP-script still failed on step 4.
After the script failed, the network connection was ok, but dns was not resolved.
It appeared, that dns resolving stopped working after rebooting between scripts, causing the PREP-script to fail. The problem was caused by the 'connman' package.
In order to get 'resolvconf' working 'connman' had to be purged and then the symlink for /etc/resolv.conf had to be reestablished. This made the dns resolver work after reboot.
I also choose to purge 'dnsmasq' and 'udhcpd'. It seemed to have no effect, but they were due to removal anyway, so I did´t change script.

By running the 'prep-bbb.sh' prior to the PREP-script, DietPi should now install on the BeagleBoneBlack running the Debian 9.5 iot image.

I noticed that the files in the home directory for the debian user was missing after the script had completed. Is there a way for preventing this?
I like to make a log file for the PREP-script and keep it after the installation is finished.

prep-bbb.sh.txt

[MichaIng]

@Snurring
Good point about /etc/resolv.conf, so we need to redo the symlink after purging non-required packages, to be failsafe, or already after resolvconf has been installed.
I am still thinking if there is any way to have all this done in one session, or if purging connman, when it's still active, in every case breaks network interfaces. EDIT: Checking your script, actually it seems that it can be purged in the same session?

So what should then work:

1. Install required APT packages
2. Configure /etc/network/interfaces so that the current active default interface (ip r s 0.0.0.0/0) will be enabled with DHCP.
3. Disable all known network managers (netman + connman currently)
4. Refresh /etc/resolv.conf symlink + ifup -a
5. EDIT: Could be skipped! If any known network manager process is active, advise user to reboot and rerun the script, to allow networking.service + resolvconf controlling the interfaces.
6. Purge all non-required packages

Ah I see you purge connman before rebooting? So then after /etc/network/interfaces and /etc/resolv.conf config + ifup -a it seems that active network connections are not dropped by this uninstall. That is nice, since reboot most likely can then be skipped.

[Snurring]

@MichaIng
The network is up until connman is purged. After purging it goes down and a reboot is needed.

[MichaIng]

@Snurring
Ah okay, even after disabling it's service and rebooting?

systemctl disable connman
reboot

How were you able to continue the uninstall and reboot then?

Would be nasty if dropping network interfaces would be somehow included in the pre/post/uninstall script, regardless what configured it. Actually no package should ever de-configure any physical network connection or DNS nameserver entry on uninstall. Leaving it up until another implementation is taking care does not hurt... However out of our control.

[Snurring]

Yes. It did not go down until I purged connman.

1. I changed the script to only do the /etc/network/interface modification and ran it first.
2. Installed resolvconf, enabled network and activated interfaces.
3. Disabled connman.
4. Purged connman.
5. Made symlink
6. Reboot.

Checked status with 'ip addr' after every step. Down after step 5 (Ifup -a will not work).
After reboot (step 6) network is up and the prep-script will work.

[MichaIng]

@Snurring
Ah, the reboot needs to be done BEFORE purging connman. As long as you do not reboot, connman is still initiator of the active network interfaces, thus when purging it, those interfaces are dropped. Hence the idea is:

1. Disable connman.service, so it will not take care network interfaces on next reboot.
2. Install and configure ifupdown + resolvconf + /etc/network/interface + resolv.conf symlink, so those are able and will take care network on next reboot.
3. Reboot, so network management is handed over from connman to networking.service.
4. Purge connman, since it should not be active anymore, hence purging it should not affect active network.

So the main issue here on headless devices is that you cannot switch the process that handles the network connection outside of a reboot, since dropping the connection makes you loose SSH which breaks the shell and any script with ongoing steps, e.g. those that would re-initiate the network interfaces. Only screen would allows such, to drop and re-start network connection since the shell+script will continue in background and you can re-attach the session once network is up again.

[Snurring]

@MichaIng
Ok. My bbb is not headless (got a 7" touch lcd attached) so I did not think of that.
I'll do a test with this in mind.

[Snurring]

With connman installed it will overwrite the symlink at reboot even if the service is disabled.
I tested this and the installation failed because it couldn't resolve when reaching PREP step 4.
In this test I let the PREP-script do the purging of connman.
My script handled the following: modified network interface settings - installed resolvconf - enabled network service - activated interfaces - disabled connman - make new symlink.
After reboot, PREP-script was started.
Also, when connman is purged, it simply removes the symlink. It doesn't care if it's made for another service.

I also did the same test, except purging connman after reboot and make the symlink before running the PREP-script. This worked fine.
What I don't understand is why the system is resolving after reboot when the symlink is tied to connman, which has been disabled.

[MichaIng]

@Snurring
Okay, nasty uninstall script then. But at least something we can handle like you did by re-creating the symlink right after purging connman.

What I don't understand is why the system is resolving after reboot when the symlink is tied to connman, which has been disabled.

Not sure, either the symlink already points to /etc/resolvconf/run/resolv.conf, due to resolvconf install, and the connman uninstall scripts always places a non-symlink default /etc/resolv.conf, regardless where it currently links to, or the connmans link target still contains the last valid nameserver entry, even after the service has been stopped, like /etc/resolvconf/run/resolv.conf as well does. In case of resolvconf there is not even a service but it's binary is a passive one, being called by other network related scripts and interface configure/de-configure hooks to update or provide nameserver info. So the last valid nameserver stays available and is used by the system as long as nothing touches the symlink or target file.

[Snurring]

I successfully did a remote install from ssh.

1. Run the prep-bbb.sh script prep-bbb.sh.txt
2. Reboot
3. Purge connman (sudo apt-get -y purge connman)
4. Restore resolvconf symlink (sudo ln -sf /etc/resolvconf/run/resolv.conf /etc/resolv.conf)
5. Run the PREP-script (./PREP_SYSTEM_FOR_DIETPI.sh)

[MichaIng]

@Snurring
Many thanks for sharing the final working procedure.

I hope I find time the next days to implement the APT reorder and would then add these steps into DietPi-PREP. Basically and additional step that checks for any high level network tools being active and in case applies the switch to networking.service handling, then ask the user to reboot.
Additionally after purging/autoremoving non-required packages, always re-create the resolv.conf symlink.

[pitsi]

The issue seems to have been resolved in v6.26, so this report can be closed,
As I had promised, I tested the script as soon as I had the chance to make a fresh vm on my friend's pc. It now completes the procedure as it should, without removing ifupdown and breaking the network connection.

Thank you :)

[MichaIng]

@pitsi
I placed a workaround for the initial issue with resolvconf/ifupdown packages, however the full idea has not yet been implemented. Hence I reopen to do this with v6.27.

[pitsi]

It seems I have lost some episodes... What is the "full idea" you want to implement?
I will probably give no further feedback for this issue because I am really unable to test it more.

[MichaIng]

@pitsi
We want to install all packages, required for the DietPi system tho run, first, and remove the non-required packages afterwards.
This resolves potential issues like the one with ifupdown being purged first and reinstalled afterwards. This is since we do not mark all packages as required, but skip those which are dependencies anyway. So such dependencies (like ifupdown) in cases are currently purged first and reinstalled afterwards, which not only can cause issues (network loss in this case), but as well causes additional non-required disk writes and install time.

Only downside is, that after installing required packages and before removing non-required ones, there is a higher maximum disk usage. This as well leads to larger data holes on the disk and the resulting image being created. However this effect should not be large and can be solved again by the DietPi-Imager script.