Dietpi: General | Using e.g. /tmp as ramdisk download location

[MichaIng]

To reduce SD writes, increase unpacking and installation times, shouldn't we use a ramdisk location, most properly /tmp or /tmp/dietpi-downloads to download achieves and packages to, unpack them and install from? Also for DietPi-Update this would be beneficial I think.

As DietPi-Software installations and DietPi-Update first stop services, there should be enough free RAM available and we recently increased /tmp size to 500M for some other reason.

In case of very large achieves, we could leave it as is or decide based on overall or free ram. Need to keep in mind, that with unpacking, we should have ~3 times the download size available. But for the vast majority of our offered software titles this should be absolutely no problem.

[Fourdee]

@MichaIng

Love the idea 👍

Yep 500MB should fit all our downloaded software one at a time, as we already remove the file afterwards, should be fine.

This will also speed up G_THREAD_START.

Yep, lets make it happen:

• variable in dietpi-software for filepath of /tmp/dietpi-downloads?
• then use $DP_DL_TMP/$INSTALL_INDEX.ext? (DP= directory path)

[MichaIng]

@Fourdee
Jep, sounds good. Hmm new prefix DP_? I think I would stay with FP_, as we use this for other directory paths as well. "_directory_ <= file" and "_file_ == regular file <= file" or something like that 🤣. But in the end it doesn't matter much.

Nice to have more and more consistency with dietpi-software downloads 👍.

[MichaIng]

Started to implement this in combination with an unified download + install function for non-APT installs: https://github.com/Fourdee/DietPi/pull/1906

[Fourdee]

Ok, looks good, nice work, very impressive 👍

I see what your trying to do with URL_LIST, allowing check of all URL's prior to install, however, I'd personally remove this to keep the code simple, and simply run a command for each item, eg:

Download_Install deb "url1"
Download_Install deb "url2"
Download_Install deb "url3"

Reasons:

• Even if we pre-check all URLS's at the start, it may fail when it comes to actually downloading it. Ideally, check straight before downloading.
• Simplifies the code a little, less global script VAR's to worry about

---

Maybe we should make Download_Install a Global?

I'll have a little play my end, see what I come up with.

[Fourdee]

Notes:

Some issues with G_THREAD_WAIT:

[FAILED] DietPi-Software | unzip -o 114.zip -d /var/www



       ┌───────────────────────────────┤ DietPi Error Handler: ├────────────────────────────────┐
       │ DietPi-Software: unzip -o 114.zip -d /var/www
       │  - Exit code: 9
       │  - DietPi version: v6.13 | HW_MODEL:12 | HW_ARCH:3 | DISTRO:4
       │  - Image creator: DietPi Core Team
       │  - Pre-image: Meveric
       │
       │ Log file contents:
       │ Archive:  114.zip
       │   End-of-central-directory signature not found.  Either this file is not
       │   a zipfile, or it constitutes one disk of a multi-part archive.  In the
       │   latter case the central directory and zipfile comment will be found on
       │   the last disk(s) of this archive.
       │ unzip:  cannot find zipfile directory in one of 114.zip or
       │         114.zip.zip, and cannot find 114.zip.ZIP, period.
       │
       │
       │                 Retry         Re-run the last command that failed

Works on retry.

Seems is not pulling the PID correctly.

[ INFO ] DietPi-Software | G_THREAD_START_1 (PID:19186) | wget https://download.nextcloud.com/server/releases/latest.zip -O 114.zip
[ INFO ] DietPi-Software | G_THREAD_WAIT_0 (PID:) | No PID for thread. Command may of finished, before PID could be pulled.

1st thread is fine, additional threads fail, var not cleared?

root@DietPi:~# G_THREAD_START wget https://download.nextcloud.com/server/releases/latest.zip -O test.zip; G_THREAD_WAIT
[ INFO ] G_THREAD_START_1 (PID:26028) | wget https://download.nextcloud.com/server/releases/latest.zip -O test.zip
[ INFO ] G_THREAD_WAIT_0 (PID:) | No PID for thread. Command may of finished, before PID could be pulled.
[  OK  ] G_THREAD: All threads finished

[MichaIng]

@Fourdee
Jep, this is also why I marked URL_LIST as optional first. I just saw some installs checking all related URLs first and though that the idea is actually good, so before any download and installation is started, it first checks for all URLs still up-to-date/available. Otherwise you might have a messy half completed installation that could have been avoided.
Download can still fail, but it will definitely, if pre-check failed due to obsolete URL/server outage.
But yeah, if it's worth to have it inside the function, is another question. We can also check URLs manually, if we think it is helpful in a certain case. And yeah in case of long time installations, a check right in front of download might be helpful again.
Finally for me either way is okay, just decide how you like it more 😃.

But the ability skip check within the script (no_check_url=1) is still needed for those cases, where wget --spider fails but download works and we thus need to check a shortened URL.

As global function we could use it within dietpi-update and dietpi-prep so far, but I think it is not (yet) enough benefit to compensate the additional RAM usage?

---

Hmm never faced the issue with G_THREAD_WAIT during my tests. Do you have a clue what might have gone wrong?

[Fourdee]

@MichaIng

Hmm never faced the issue with G_THREAD_WAIT during my tests. Do you have a clue what might have gone wrong?

Fixed with https://github.com/Fourdee/DietPi/commit/f161b6fed6fa85aa5cf43b6ed92837c46fcf3cc9

root@DietPi:~# G_THREAD_START wget https://download.nextcloud.com/server/releases/latest.zip; G_THREAD_WAIT
[ INFO ] G_THREAD_START_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26248) | wget https://download.nextcloud.com/server/releases/latest.zip
[  OK  ] G_THREAD: All threads finished
root@DietPi:~# G_THREAD_START wget https://download.nextcloud.com/server/releases/latest.zip; G_THREAD_WAIT
[ INFO ] G_THREAD_START_0 (PID:26261) | wget https://download.nextcloud.com/server/releases/latest.zip
[ INFO ] G_THREAD_WAIT_0 (PID:26261) | wget https://download.nextcloud.com/server/releases/latest.zip

[Fourdee]

Notes:

• fixed RPi failed install of WiringPi due to downloading web page, instead of file, now hosted on DietPi.com
• Autodetect file extension type with Download_Install. Only downside is all links must be full links, no redirects. I've checked/updated all items in script and we should be fine.
• use G_THREAD_START for all downloads in Download_Install, regardless if there is no DEPS_LIST

---

issue with nextcloud install

─────────────────────────────────────────────────────
 Mode: Configuring PHP: webserver

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pdo_sqlite.so' - /usr/lib/php/20151012/pdo_sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/sqlite3.so' - /usr/lib/php/20151012/sqlite3.so: cannot open shared object file: No such file or directory in Unknown on line 0
WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available
WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available
WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available

[Fourdee]

package_list+=" $PHP_APT_PACKAGE_NAME-sqlite*" #wildcard for version (eg:3)
~Doesnt like the wildcard? does not get installed. Looking into it now.~

Hmm has something changed, php7.0-sqlite3 not getting installed unless sqlite is installed. Breaks most php-sql related programs:

Koel

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pdo_sqlite.so' - /usr/lib/php/20151012/pdo_sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0

---

Node still installing x86 on ARMv8.

/usr/local/bin/yarn -> /usr/local/lib/node_modules/yarn/bin/yarn.js
+ [email protected]
+ [email protected]
added 2 packages from 4 contributors in 2.065s

     install : node-v8.11.3
       mkdir : /usr/local/n/versions/node/8.11.3
       fetch : https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x86.tar.gz
######################################################################### 100.0%
/usr/local/bin/n: line 590: /usr/local/bin/node: cannot execute binary file: Exec format error
   installed :

arm64 binaries exist (https://nodejs.org/dist/v8.11.3/), appears external bug with npm code

Initial install is fine v10, appears v8 downgrade is the issue:

Node Linux Installer by www.github.com/taaem
Running as Root User
Get Latest Version Number...
Done
Downloading latest stable Version node-v10.7.0-linux-arm64.tar.gz...
Done
Installing...
Finished installing!

---

─────────────────────────────────────────────────────
 Mode: Configuring PHP: webserver

WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available
WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available
WARNING: Module apcu.dpkg-dist ini file doesn't exist under /etc/php/7.0/mods-available

🈯️ https://github.com/Fourdee/DietPi/commit/4e0efa706ef4ea10330f96255ea433c7f9a36010

Jul 20 21:22:16 DietPi php[11727]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/wddx.so' - /usr/lib/php/20151012/wddx.so: cannot open shared object file: Permission denied in Unknown on line 0
Jul 20 21:22:16 DietPi php[11727]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/xmlreader.so' - /usr/lib/php/20151012/xmlreader.so: cannot open shared object file: Permission denied in Unknown on line 0
Jul 20 21:22:16 DietPi php[11727]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/xmlwriter.so' - /usr/lib/php/20151012/xmlwriter.so: cannot open shared object file: Permission denied in Unknown on line 0
Jul 20 21:22:16 DietPi php[11727]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/xsl.so' - /usr/lib/php/20151012/xsl.so: cannot open shared object file: Permission denied in Unknown on line 0
Jul 20 21:22:16 DietPi php[11727]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/zip.so' - /usr/lib/php/20151012/zip.so: cannot open shared object file: Permission denied in Unknown on line 0
Jul 20 21:22:16 DietPi php[11727]: PHP Fatal error:  Interface 'JsonSerializable' not found in /var/www/koel/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php on line 22
Jul 20 21:22:16 DietPi php[11727]: PHP Fatal error:  Interface 'JsonSerializable' not found in /var/www/koel/vendor/laravel/framework/src/Illuminate/Support/Collection.php on line 17

root@DietPi:~# ls -lha /usr/lib/php/20151012/zip.so
-rw-r--r-- 1 root root 55K Jun 14 14:50 /usr/lib/php/20151012/zip.so

Permissions issue running under non-root, didn't exist in v6.12, why now? (i've probably broke something 🤣 )

Strange, live (v6.12) in fine.

[MichaIng]

I mark this as closed. Implementation done via:

• https://github.com/Fourdee/DietPi/pull/1906
• https://github.com/Fourdee/DietPi/pull/1945
• https://github.com/Fourdee/DietPi/pull/1961

Further implementation into all scripts, that produce tmp files, can be done, but the most urgent ones are covered.