DietPi-Globals | Replace duplicate function/var code in our scripts

[Fourdee]

By simply loading this script on demand to export global vars.

Also allows for logged in bash user session to use our commands (eg: AGI) etc.

[Fourdee]

@MichaIng

root@DietPi:~# AGI thispackagedoesnotexist_Test
 [Info] APT is processing, please wait...

E: Unable to locate package thispackagedoesnotexist_Test

---

[MichaIng]

Nice! 👍
Just some minor questions/hints:

• Why not include the error check/output to base AGI etc. directly? Would be helpful in every script and terminal, I think.
• Since -qq includes -y, we can remove the latter: https://wiki.ubuntuusers.de/apt/apt-get/ Why ubuntu wiki is so much better/more detailed than debian ones? Totally missing complete command overviews with all sub commands and options there! 😛.

[Fourdee]

@MichaIng

Why not include the error check/output to base AGI etc. directly? Would be helpful in every script and terminal, I think.

Makes sense, i'll take a look. 👍

Since -qq includes -y, we can remove the latter

Its a habit thing more than anything, although, having -y in there, ensures if we (or apt) change any options in the future, the -y should always ensure start of install. I'll leave it in for now.

[Fourdee]

ToDo:

• 🈯️ Global dietpi-notify
• 🈯️ Replace existing /DietPi/dietpi/dietpi-notify commands in dietpi scripts with dietpi-notify etc
• 🈯️ Replace existing apt-get commands in dietpi scripts with AGI etc
• 🈯️ Send my commit
• 🈯️ Manually fix conflicts on outstanding PR's
  -- 🈯️+ using global dietpi-notify
  -- 🈯️+ using global AGX commands

---

• dietpi-drive_manager not listing USB drives, list mode is 1, caused by:
  -- Print_Input_String() in dietpi-notify in dietpi-global.
  --- 🈯️ Fixed, seems i was not local to dietpi-global and being passed back to script loading dietpi-notify
  --- confirm defined i is not changed by dietpi-notify

root@DietPi:~# i=1230
root@DietPi:~# . /DietPi/dietpi/func/dietpi-globals
root@DietPi:~# dietpi-notify 0 Test program
 [Ok] Test program
root@DietPi:~# echo $i
1230

#and visa versa
root@DietPi:~# unset i
root@DietPi:~# . /DietPi/dietpi/func/dietpi-globals
root@DietPi:~# dietpi-notify 0 Test program
 [Ok] Test program
root@DietPi:~# echo $i

[Fourdee]

@MichaIng

Why not include the error check/output to base AGI etc. directly? Would be helpful in every script and terminal, I think.

We can provide a notify for result of the apt-get command, but if we want to handle the end result, we'll need to do this outside the global script.
eg: in dietpi-software, we need to handle the exit code, then we can match it to software title etc if its a failure. If we simply exit in dietpi-global AGX, we lose the ability to do that.

[Fourdee]

This is now completed as intended, however, we'll continue to add/improve it as needed in the future.

[Fourdee]

dietpi-drive_manager move userdata:

• 🈯️ Resolved: https://github.com/Fourdee/DietPi/commit/63bf1a2d50b9b1ae818853f7cc53cc3c080ed5b8

root@DietPi:~# readlink -f /mnt/dietpi_userdata
/mnt/00293bdc-f422-41a3-99f9-7bfd50eeb266/dietpi_userdata
root@DietPi:~# readlink -f /mnt/dietpi_userdata
/mnt/dietpi_userdata

---

root@DietPi:~# echo $(/DietPi/dietpi/func/dietpi-set_userdata return_source)
[Info] DietPi-Globals loaded  [Info] Checking for (elevated) root access, please wait..  [Ok] Root access verified. /mnt/dietpi_userdata

Ok,

• 🈴 Disable any print for a default load of globals?, nope, we need error information if rootfs rw, root user access fails etc.
  

[Fourdee]

Completed (again).

[Fourdee]

• use of G_ at start of all global commands/vars to clearly define global status, and, avoid variable interaction.

root@DietPi:~# G_
G_AGA              G_AGI              G_AGUP             G_CHECK_URL
G_AGDUG            G_AGP              G_CHECK_ROOTFS_RW  G_DIETPI-NOTIFY
G_AGF              G_AGUG             G_CHECK_ROOT_USER

[Fourdee]

@MichaIng

Globals are now finalized, using G_format https://github.com/Fourdee/DietPi/issues/1311#issuecomment-352999485

Supported use in PREP_SYSTEM: https://github.com/Fourdee/DietPi/issues/1311#issuecomment-353039600

Completed (for now) 😄

[Fourdee]

Check and automatically install required packages:
NB: renamed to G_AG_CHECK_INSTALL_PREREQ

[Fourdee]

Note to self:

• Allo GUI replace $(/DietPi/dietpi/dietpi-cpuinfo 1) with G_OBTAIN_CPU_TEMP
  -- + /DietPi/dietpi/dietpi-cpuinfo 3, with G_OBTAIN_CPU_USAGE

[Fourdee]

Improved error handling:

Notes:

• G_AGx commands are automatically error handled

G_RUN_CMD mkdir /never/gonna/work

G_AGI nevergonnawork

Disable whiptail (eg: headless)

export G_USER_INPUTS=0; G_AGI nevergonnawork

root@DietPi:~# G_AGI nevergonnawork
 [Info] APT installation for: nevergonnawork
 [Info] APT is processing, please wait...

E: Unable to locate package nevergonnawork

 [Error] G_AGI: nevergonnawork
 [Info] exit_code = 100
 [Info] HW_MODEL:12 | HW_ARCH:3 | DISTRO:5
 [Info] Logfile contents:
E: Unable to locate package nevergonnawork
 [Info] Unable to continue, the program will now terminate.
 [Error] Unable to continue, this program will now terminate.

---

DietPi-Software:

[MichaIng]

@Fourdee
Maybe we can include the option the directly send the error as dietpi-bugreport (together with all the other system data)?

---

About bug reporting generally, although this would belong into an own topic, it would be nice to have an option or by default make an output that one can copy&paste directly into github preformatted.
Nextcloud e.g. offers this via an app and by this all information is together directly: https://apps.nextcloud.com/apps/issuetemplate

[MichaIng]

I played around a bid with global error handling and info echo during installation. It would be nice to reduce the overall amount of echo lines without reducing the information content.

I loved the installation script of PiHole and had a look into their method:

• echo -n 'test...' skips new line
• "\\r\\033[K" at the beginning of string, will overwrite existing line.
• Thus you can use:

echo -n 'Command is running...'
if ( command succeded ); then
  echo -e "\\r\\033[KCommand succeded"
else
  echo -e "\\r\\033[KCommand failed"
end

... to just show the final result of a command, after it's finished.

• Maybe we can use this for G_RUN_CMD?

Another issue I found with G_RUN_CMD is, if you want to forward output into a file, e.g. G_RUN_CMD echo "blabla" > /path/to/file, the echo output of error handler will also be found in the file. I'm not sure how in this case it is set, where the input for G_RUN_CMD ends and in case it's output could be treated. Is it possible to somehow parenthesise the error handlers input?

[Fourdee]

@MichaIng

Maybe we can include the option the directly send the error as dietpi-bugreport (together with all the other system data)?

Love the idea 👍

The main issue we have is structure, currently bug reports are simply uploaded to an FTP server. Its really not that much fun to work with bug reports in that format.

Maybe we could add a simple error send on failures, however, we'd need a better system to log them than FTP. SQL db on server, some kind of API on github that allows us to automatically generate a ticket? All options need research.

copy&paste directly into github preformatted.

Might work, not a bad idea 👍

[Fourdee]

@MichaIng

Another issue I found with G_RUN_CMD is, if you want to forward output into a file, e.g. G_RUN_CMD echo "blabla" > /path/to/file, the echo output of error handler will also be found in the file. I'm not sure how in this case it is set, where the input for G_RUN_CMD ends and in case it's output could be treated. Is it possible to somehow parenthesise the error handlers input?

Yep, had the same issue with cat << _EOF_ > file. I'll take a look, but we may need another command for this.

edit:
cant see a way to achieve redirect handles with the command:

root@DietPi:~# G_ERROR_HANDLER_COMMAND='echo 1 > test'
root@DietPi:~# $G_ERROR_HANDLER_COMMAND
1 > test

We'll need to avoid using G_RUN_CMD if it contains a redirect. Then we can add a global for G_FILE_EXISTS to check for file creation afterwards.

• 🈯️ G_FILE_EXISTS https://github.com/Fourdee/DietPi/commit/c54c657b4b9a1aefc0b26c5bd13e56e8b37b44f4
  

[Fourdee]

@MichaIng

I loved the installation script of PiHole and had a look into their method:

Yep, i agree, very clean.

Currently, i've simply enabled as much info as possible during dev, using existing dietpi-notify. We can always trim/clean/refine it at a later date.
Notify may have an issue with echo -n 'test...` || "\\r\\033[K", would need testing.

[Fourdee]

@MichaIng
having a little play with your idea 👍
This will only print the github info on G_ERROR_HANDLER_ONERROR_EXIT=1 (default)

---

Results in:

Required Information:

• DietPi Version | v160
• SBC Device | oDroid C2 (aarch64) (index=12)
• Distro | buster (index=5)
• Command | G_AGI: nevergonnawork
• Error Handler | G_ERROR_HANDLER

Additional Information (if applicable):

• Software title | DietPi-Software

Expected behaviour:

Actual behaviour:

Steps to reproduce:

Additional logs:

Log file contents:
E: Unable to locate package nevergonnawork

[MichaIng]

@Fourdee
Nice so far 👍.

• On $G_USER_INPUT => whiptail, we could add an user choice at the bottom to create the guthub ready output on demand (if user has/agrees to create github account e.g.).
• If it is possible to then create this within whiptail and automark the lines (at least on SSH it marks all parts from outside the whiptail "window" as well), we could easily add more information to the output and replace the existing dietpi-bugtracker partly with this. But yeah it should stay for exceptional hard issues, as I know there are muuuch files/configs/logs gathered and copied to your server.

[MichaIng]

@Fourdee
I played around also a bid with process messages and error handling. Basically I merged some status messages to reduce the output lines and established a new G_DIETPI-NOTIFY option "-2" == process. This process messages will not produce a newline and will be overwritten by "ok" and "failure" outputs. Thus in the end you will only see the final results and still perhaps interesting/important "2" info messages in between.

I also made the brackets all same size. As "Processing" is a long word, this might need tuning, anyway is just an offer/idea, how we could handle our output:

After adding some "2" notifications to ownCloud installation and running some of it's commands as "G_RUN_CMD" it looks like this:

During (longer) G_AGX/G_CMD_RUN commands you will see:

[Fourdee]

@MichaIng

During (longer) G_AGX/G_CMD_RUN commands you will see:

Love it, looks brilliant, very nice work! 👍

EDIT:
Once you've finished this, send the PR and i'll take a look.
We can also remove the dietpi-funtime 😉 (it looked fun, but did absolutely nothing lol)

[k-plan]

MichaIng commented:
But hmm, we are sort of in the wrong issue here @Fourdee is it possible for you to copy the last comments over to https://github.com/Fourdee/DietPi/issues/1285?

Yes, sorry, it was my fault. Wrong issue here. Done!

_(have deleted double comments. Sorry for the mess.)_

[Fourdee]

Testing/notes: https://github.com/Fourdee/DietPi/pull/1364

• Processing only gets replaced by ok/failure, with G_RUN_CMD and few others. Ah i see, NOTIFY -2 rollout.
• G_CHECK_URL does not replace
• 🈯️ https://github.com/Fourdee/DietPi/commit/d5b7c01d0c95ce9ac846d44adf02a6ae0f9ab8cd G_CHECK_URL hangs on a failed connection. No result. Due to error output being printed in background. Due escaping overwrite string, not needed.
• 🈺 Replace does not replace, if output hits screen width :
  -- Maybe another idea is to use echo -en "\033[1A 1 \033[0m" which will print on the previous line?

root@DietPi:~# G_RUN_CMD systemctl enable dietpi-fs_partition_resize.service
 [Processing] Running command: systemctl enable dietpi-fs_partition_resize.servi [    Ok    ] systemctl enable dietpi-fs_partition_resize.service

root@DietPi:~# G_RUN_CMD echo 1
 [    Ok    ] echo 1

[Fourdee]

Notes: https://github.com/Fourdee/DietPi/pull/1365

• @MichaIng Looks much better 👍
• 🈯️ Fixed with log file generation. ~G_CHECK_URL http://google.comxxxx hangs, stuck in BG. kill tail process, whip appears.~
  

[Fourdee]

Notes:

• Incorrect usermsg (not cleared from previous)
• Missing dietpi-software info

 [  Error   ] DietPi-Software: G_AGI: mosquitto
 [   Info   ] exit_code = 100
 [   Info   ] VERSION:v6.-1 | HW_MODEL:3 | HW_ARCH:2 | DISTRO:4
 [   Info   ] Log file contents:
E: Failed to fetch http://repo.mosquitto.org/debian/pool/main/m/mosquitto/mosquitto_1.4.12-0mosquitto1_armhf.deb  404  Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
 [   Info   ] User message:
Connection failed while trying to access: http://dietpi.com/downloads/binaries/all/libwebsockets3_1.2.2-1_armhf.deb. The URL may be offline, or invalid.
 [   Info   ] If problems persist, please report this to DietPi for investigation, including a screenshot of this error! (https://github.com/Fourdee/DietPi/issues).

---

G_ERROR_HANDLER_ONERROR_USERMSG must never be defined changed in dietpi-globals additional functions. It must only be exported by the sourcing script, to ensure it never gets overwritten.

The downside:

• G_ERROR_HANDLER_ONERROR_USERMSG will be displayed for any future errors, regardless if its outdated. Unless we manually add unset G_ERROR_HANDLER_ONERROR_USERMSG in the sourcing script after each event (messy)

Solution for now, remove G_ERROR_HANDLER_ONERROR_USERMSG from scripts. See if we can implement this better in the future.

[Fourdee]

I'll mark this as completed. Will reopen if required.