Dietpi: Design: Using Zip based storage for multiple files.

[sytone]

Breaking this out to have a discussion on this as I mentioned it in a review which is not really the right place.

Idea: Reduce file count while making separation of scripts for software packages more manageable.
Assumptions: zip/unzip available on all platforms.
Proposal
In GitHub make a /dietpi/software folder. For each software package make a {id} file (could be name for clarity as well, just has to match in the dietpi-software file. ID makes picking the next one easy :)

The {id} file would have a structure internally like the blow, optionally we can have start service/stop service in here as well so each package can control or check if they should restart/stop during a change or update.

if [ "$1" = "install" ]  ; then
  # Install Logic
elif [ "$1" = "uninstall" ]  ; then
  # Uninstall Logic
elif [ "$1" = "reinstall" ]  ; then

elif [ "$1" = "setpermissions" ]  ; then

elif [ "$1" = "setconfiguration" ]  ; then

else
  /DietPi/dietpi/func/dietpi-notify 2 "Unknown command $1"
fi

The image creation process would be:

• Copy all the bits amd make image.
• run zip to put all the software packages in a zip file.
• delete the software directory.

During execution of dietpi-software:

• make menu from internal array (or JSON ;) )
• When user selects package unzip the script via the ID number.
• source it as part of the software script to execute the required commands.
  
  
  
  • It could alos call the zip api and pull out files like the service conf or other conf files removing them from scripts and the EOF piping logic to clean it up. Then Software would be /dietpi/software/{id} and /dietpi/software/{id}/service.conf etc.
  
  

[Fourdee]

@sytone

Then Software would be /dietpi/software/{id} and /dietpi/software/{id}/service.conf etc.

Love where your going with this 👍

ref: https://github.com/Fourdee/DietPi/pull/1087#issuecomment-331666708

[sytone]

The approach I am going with is sourcing in bAsh files like libraries. Any issues with this? This way common variables can be shared as bAsh cannot handle objects so stuck with string manipulation

[Fourdee]

@sytone

bAsh

Hi,

Is this some new type of Bash that i'am unaware of, or, file name extension you plan to use? Or something else, unsure what you mean 😕

[sytone]

Typo. :)

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Dan [e-mail]
Sent: Thursday, October 26, 2017 11:40:21 AM
To: Fourdee/DietPi
Cc: Sytone; Mention
Subject: Re: [Fourdee/DietPi] Design: Using Zip based storage for multiple files. (#1157)

@sytonehttps://github.com/sytone

bAsh

Hi,

Is this some new type of Bash that i'am unaware of, or, file name extension you plan to use? Or something else, unsure what you mean 😕

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Fourdee/DietPi/issues/1157#issuecomment-339760329, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABVakzqiWabUQm1NIva8c4jxQQeGPdWpks5swNIVgaJpZM4PhoRS.

[MichaIng]

Have the same idea, first to separate software installation from dietpi-software, but would skip the zip. DietPi files are small anyway, so zip is not necessary. But removing those conf files from ram would be a goal I am looking for. /opt/ would be an idea....

[sytone]

So played a bit with this, working with scripts is slowly sending me mad as I have to pass strings around. Considering the following approach.

In git we have all the files broken out and as part of the release/packaging process we take these and push them together into single files to reduce the foot print, this is effectively building the dietpi-software on the fly or including a library in which just contains the pre-generated arrays it uses as a single file for all scripts to use. That way each software package can have separate scripts for install, uninstall, configure, start, stop, etc and would be easier to manage and simpler to redouble shoot as you can run them independently to validate and then together as part of the full system.

Leaning towards including using the full path for ease, for example each file would have a standard header like this so all constants are in one place and not spread out. Then files could include other files as needed to increase re-use and reduce any duplication. Looked around and this seems to be a valid approach for all bash files working with libraries of functions given there Is not a native thing like this in the scripts.

#!/bin/bash
. /DietPi/dietpi/lib/constants.sh

Example constants file.

#!/bin/bash
#//////////////////////////////////////////////////////////////////////
# DietPi Script constants and override helpers
#
#//////////////////////////////////////////////////////////////////////
# Created by Daniel Knight / [email protected] / dietpi.com
# Created by Jon Bullen / [email protected] / sytone.com
#
#//////////////////////////////////////////////////////////////////////
#
# Info:
# - filename /DietPi/dietpi/lib/constants.sh
#//////////////////////////////////////////////////////////////////////

# Core variables
DIETPI_ROOT="/DietPi"
DIETPI_SCRIPT_ROOT="$DIETPI_ROOT/dietpi"

# This should be removed / not pulled when clients update as the generated scripts should
# be used. However this can stay for testing, just be aware of the file count overhead. 
DIETPI_SOFTWARE_PATH="$DIETPI_SCRIPT_ROOT/software"

# Error codes
E_NOARGS=75

##################################################################
# Used for diagnostics, prints known constants and overrides. 
# Arguments:
#   None
# Returns:
#   None
##################################################################
dietpi_print_settings() {
    echo "$DIETPI_ROOT"
    echo "$DIETPI_SCRIPT_ROOT"
}

@Fourdee throughts?

[MichaIng]

@sytone
Nice that you went on thinking about this. It actually also still spook through my mind, but as the image preparation automation https://github.com/Fourdee/DietPi/issues/1285 and dietpi-globals https://github.com/Fourdee/DietPi/issues/1311 are already two fundamental changes, I wanted to wait with the next until v160 is out.
Anyway good to bring the ideas forward.

There were already some other topics about separating the dietpi-software script for better readability, coding and simply browsing performance especially on mobile devices (as I often use spare bus/train travel times to have a look into DietPi issues 😉.
My ideas are to start with simply separating installation+configuration+uninstallation of every piece of software into their own files and source them _(€: Ah, just see you had the same idea)_ from dietpi-software, if chosen to handle: https://github.com/Fourdee/DietPi/issues/1163#issuecomment-346912863
Now, that all important variables+functions are globally accessible (okay, the G_AGX and AGX_ERROR_HANDLED are different), we could even just run the scripts without sourcing, thus local variables and stuff will not disturb within the main/next script any more.
After this is done, we could go on thinking for other, more elegant ways, to go through the software scripts.

You are thinking now about merging them together again into on script on the devices? I am just thinking what actually are the benefits of having everything in one file, instead of leaving it separated? One argument of Fourdee was, that single files take more space, because of minimum block sizes, which is especially important, as we mount all DietPi scripts as ramdisk. On the other hand we are talking about ~5 MB overall size, so the difference will be hardly noticeable. And if we already separate out the individual software scripts, we could also save/move them e.g. to the newly used /var/lib/dietpi-software: https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-software#L10286 where they are out of ramdisk.

[Fourdee]

Marking as closed, in favor of: https://github.com/Fourdee/DietPi/issues/1163