Dietpi: Dietpi-Software | Split script for better readability

[Bridouz]

Hi,

Juste throwing an idea for DietPi. The distribution is a reamly Nice piece of code but I fond myself often lost in that huge file that dietpi-software is.

Could it be more friendly to split dietpi-software with :

• One main file which brings the menu
• Category Folder containing individual files for software (both installation ans optimisation)

I know it's a lot of work but I think it would be easier for people to contribute.

[MichaIng]

I had moreless the same idea (https://github.com/Fourdee/DietPi/pull/1142#discussion_r139130084), but:
As the "DietPi" folder is written into ramfs, it should be generally kept small. Splitting everything into individual files brings the problem that every files takes at least a 4k block, so several individual files will most likely take more memory.

But for me the benefit would outweigh the larger memory usage, as even hundreds of 4k blocks sum up to a quite small overall memory usage, even to 256er systems. The new default swap settings provide 2 GB anyway.
The next is, that I would find (middle or long term or step by step from now on) another location for e.g. software related config files and in your case the individual software installation+optimization files outside of ram.

[sytone]

I have a issue open on this with a proposal. I have played around and have a model I’ll push to my fork over th weekend.

[MichaIng]

Easy solution would be to move installation, configuration and uninstallation of every software into an own subfolder script and source that script at the right position of software id loop.

By sourcing instead of just running these scripts, it will be executed within the same terminal context, thus global variables and functions will be shipped as well.

nano ~/test

#!/bin/bash
TEST="success"
echo "$TEST test"
. ~/subtest

nano ~/sub/test

#!/bin/bash
echo "$TEST subtest"

~/test

success test
success subtest

The dot . with space afterwards btw is an abbreviation of source.

To solve it more elegant, we could just move an index through all subfolders, check for some .installed flag there and in case run the related scripts:

#!/bin/bash
for i in /etc/*; do echo "$i"; done

We could do even more nice things by this:

• Dependencies/influences of software between each other (configs that depend on appearance of other software etc) could be just solved by e.g. subfolders with the dependencies name/the name of the software, which config needs to be adjusted (if installed) and place the scripts to do this right into the subfolders.
• Generally, I would like to see a dependency system, where individual other software can be chosen instead of the fixed aSOFTWARE_REQUIRES_* solution.

But yeah step by step and definitely low priority if any 😉. At least config files btw. could be well moved into /opt/dietpi e.g. The /opt folder at least seems to be for configs and other related stuff for software that was installed manually, not via package manager. /etc on the other hand is intended for apt/repo/package manager installations.

[Fourdee]

I did a initial prototype for this a while back, but lacked the time to pull it off, putting it here for reference:

dietpi-software_splitfiles.zip

[MichaIng]

@Fourdee
Okay, so you prefer to run this scripts as independent environments instead of sourcing? Needs loading of globals for every software title again, as well as special variable determination, such as PHP version/base path and install state of other software. This also increases the effort, if methods about this change. (€: such vars can simply be exported!) On the other hand, it allows better/specific error handling/investigation, minimizes possible unwanted interaction and allows separate execution of single software installation/uninstallation, if needed for dev, fix or on update.

You did not include configuration yet. Include it into same file via argument "2", or include configuration into installation? Latter would need some testing/investigation, that no configuration steps are based on other installation steps.

Why not also include software definitions, so definition: 0, installation: 1, configuration: 2 (or directly after installation), uninstallation -1? Ah just see the issue: aSOFTWARE would need to be global, if subscripts are running instead of sourced.

[MichaIng]

@Fourdee
While at first I had the very same idea I am not to sure anymore whether I would like dietpi-software split. The effort to search and change install methods would massively increase, if one has to open single files for each software title.

But I have some other ideas about decreasing the script size:

• Adding more install methods to Download_Install(), e.g. direct script download + execution, Git clone.
• Handle existing install dir within Download_Install() based on option variables (EXIST=skip/merge/clean, BACKUP='file1.conf file2.conf dir1') and otherwise whiptail prompt to let user decide:
  
  
  
  • a) Skip download + install
  
  
  • b) Merge new dir into existing
  
  
  • c) Remove old dir, optionally preserve (backup/restore) old config files/data dirs
  
  
• Merge DietPi software and Linux software into one loop, simply do Linux software first.
• Merge software install and config step. Actually I don't know any case, where for configuration of two software titles, for each, the install of both is required. The dependencies are in one direction only, so install + configure the dependencies first, then install + configure the dependants should work fine and reduces the required up/down scrolling and separation of both steps.

All low priority, but something I would like to work on by times and like to hear your opinion.

[Fourdee]

@MichaIng

While at first I had the very same idea I am not to sure anymore whether I would like dietpi-software split. The effort to search and change install methods would massively increase, if one has to open single files for each software title.

Agree, each file would need to load globals, download the file during install. Generally will slow the installation down.

I also prefer it all in dietpi-software, runs well currently. The only reason we might need to split to install files, is if we reach a significant software number (eg: 200+), in which performance should be equal.

Adding more install methods to Download_Install(), e.g. direct script download + execution, Git clone

Yep, good idea to simplify code 👍

Merge DietPi software and Linux software into one loop, simply do Linux software first.

Makes sense. Probably keep the menus separate (as it is currently)?

Merge software install and config step.

Makes sense 👍

[MichaIng]

@Fourdee

Agree, each file would need to load globals, download the file during install. Generally will slow the installation down.

This could be resolved by sourcing the files instead of calling them as sub shell. This also prevents us from the need to estimate PHP version and such dietpi-software wide variables.

Makes sense. Probably keep the menus separate (as it is currently)?

Jep, code and menu can be handled individually.
On the long term, we could merge the menus as well and perhaps remove some very basic software titles (libraries, very simple dependencies) from having an own ID and simply install via DEPS_LIST (in case of APT package) during software installs that require them. On the other hand, individual software ID prevents reinstall on every dependent install, so not yet sure. Or we split differently into libraries/runtime engines/deps, that usually one does not want to install as single software without related "frontent" and those that have a use on their own.
But nothing urgent to think about now, so far it's about keeping the code size in ranges.