DietPi PREP_SYSTEM_FOR_DIETPI.sh (dev branch) fails on Pi 4B 8GB with 64bit raspios

[StephanStS]

Details:

• Date | Thu Jun 4 20:09:16 BST 2020
• Bug report |
• DietPi version | v6.30.0 (MichaIng/dev)
• Image creator |
• Pre-image |
• Hardware | (ID=0)
• Kernel version | Linux raspberrypi 5.4.42-v8+ 1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux
• Distro | buster (ID=5)
• Command | apt-get -q update
• Exit code | 100

• Software title | DietPi-PREP

• SBC: New Raspberry Pi 4B with 8 GB RAM and OS 2020-05-27-raspios-buster-arm64

Steps to reproduce:

1. Flash and start 2020-05-27-raspios-buster-arm64 on Pi4
2. Login via SSH, su to root user
3. apt update && apt upgrade (runs without problems, see below: Running apt update via DietPi-PREP fails)
4. reboot and login again, su root user

5. Run dietpi stuff:
   apt update
   apt install -y systemd-sysv ca-certificates sudo wget locales --reinstall
   wget https://raw.githubusercontent.com/MichaIng/DietPi/dev/PREP_SYSTEM_FOR_DIETPI.sh -O PREP_SYSTEM_FOR_DIETPI.sh
   chmod +x PREP_SYSTEM_FOR_DIETPI.sh
   ./PREP_SYSTEM_FOR_DIETPI.sh

6. Run Dev version of DietPi-PREP

Expected behaviour:

System shall run DietPi-PREP without error.

Actual behaviour:

At the beginning of the script the apt update fails.

Extra details:

Details 1:
Contents of /etc/apt/sources.list before DietPi-PREP is:

deb http://deb.debian.org/debian buster main contrib non-free
deb http://deb.debian.org/debian-security/ buster/updates main contrib non-free
deb http://deb.debian.org/debian buster-updates main contrib non-free
# Uncomment deb-src lines below then 'apt-get update' to enable 'apt-get source'
#deb-src http://deb.debian.org/debian buster main contrib non-free
#deb-src http://deb.debian.org/debian-security/ buster/updates main contrib non-free
#deb-src http://deb.debian.org/debian buster-updates main contrib non-free

Details 2:
At the point of the problem:
The contents of /etc/apt/sources.list is
deb http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free

The contents of /etc/apt/sources.list.d/raspi.list is
deb https://archive.raspberrypi.org/debian/ buster main

Details 3:
Also a test without steps 3. and 4. fails (i.e. direct run of DietPi stuff in step 5.).

Details 4:
If

• going into a subshell at the point of the problem,
• re-editing /etc/apt/sources.list to the contents given above,
• exiting the subshell and rerun the failed command
  the DietPi-PREP runs much longer, but fails also later.

Additional logs:

Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Hit:2 https://archive.raspberrypi.org/debian buster InRelease
Err:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9165938D90FDDD2E
Reading package lists...
W: GPG error: http://raspbian.raspberrypi.org/raspbian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9165938D90FDDD2E
E: The repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' is not signed.

Additional question

Is there a logfile of the DietPi-PREP script present which can be investigated?

[Joulinar]

Hi,

you are missing the correct signatures for the package.

Can you try following on a sub shell

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9165938D90FDDD2E

EDIT:

Ah you are using the 64bit BETA image from Raspi OS. And this is pointing to Debian packages (http://deb.debian.org/debian). While the DietPi PREP script is not yet ready for the 64Bit version and revert all source back to Raspbian (http://raspbian.raspberrypi.org)

[MichaIng]

Ayayay, that is why I was arguing that RPi foundation should make somehow clear that Raspberry Pi OS (64 bit) is Debian (!) and not Raspbian!

DietPi does not support is hence yet. DietPi-PREP will add the Raspbian repo, while it needs to add/keep the Debian repo, and there are several other cases where our code expects Raspbian when getting an RPi G_HW_MODEL.

So please do not try to work around the issues, you will only run into more, especially since Raspbian and Debian are simply not compatible, when its about mixing core packages/libraries with the to different architectures (Raspbian = armv6hf, RPi OS 64 bit is arm64) 😉.

---

Basically duplicate of: https://github.com/MichaIng/DietPi/issues/3570

[Joulinar]

Well workaround is to use the official 32bit Raspberry OS Image instead

https://downloads.raspberrypi.org/raspios_lite_armhf_latest

[StephanStS]

@Joulinar, @MichaIng: Thanks for the explanation, so I can stay at raspbios (w/o DietPi) or use the actual 32 bit DietPi image.
So, for a 64 bit DietPi I will wait for your announcement or a trigger from you to do some DietPi-Tests with a development version when someone works on the 64 bit issue.

[MichaIng]

Yes indeed, the newest kernel also supports RPi4 8 GiB, which requires hence an image upgrade from our side. But I guess 8 GiB RAM will not be fully usable.

Until DietPi fully supports and handles correctly RPis on Debian (basically), probably we should add an exit path within DietPi-PREP 🤔.

So, for a 64 bit DietPi I will wait for your announcement or a trigger from you to do some DietPi-Tests with a development version when someone works on the 64 bit issue.

👍 I'll start working on it during the weekend, but honestly I am not sure how deep the RPi <> Raspbian equivalence assumption is coded into DietPi. In theory it is easy (check /etc/os-release for "raspbian" or "debian" and store this info in /boot/dietpi/.hw_model), but I'm afraid the devil is in the detail 😅.

[Joulinar]

or use the actual 32 bit DietPi image.

Nope you would need to start with the 32bit Raspberry OS Image and run the PREP script. The actual available DietPi Image (v6.28) will not work as the kernel used on the image will not support the 8GB version.

[MichaIng]

Yes indeed, the RPi 8 GiB model seems to require the newest RPi kernel version to boot. This is the first I'll repack tomorrow 👍.

[StephanStS]

If it helps, I could generate the image like Julinar mentioned and do some short tests on my PI4 8GB with it.
Tests: Local HMI, ssh shell, WiFi, LAN, some basic installations from dietpi-software (nextcloud, Webmin, X11 xfce).
The PREP script runs with the actual Raspbian Lite (kernel 4.19), the system reboots and installs as a DietPi minimal image without an error.

[MichaIng]

Many thanks for testing, good to know that there are really no other changes required.

How about free -m the physical RAM being reported? I'm wondering what happens if one reaches not the physical RAM limit but the limit that the 32 bit OS can handle 🤔. If that is somehow done non-gracefully, we might be able to add some warning and size /tmps smaller then the usual 50% of physical RAM. When 64 bit OS is supported, of course this is an edge case, but we cannot prevent users from flashing the 32 bit Raspbian-based image on their 8 GiB RPi4.

[StephanStS]

root@DietPi:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           8016          71        7859          16          84        7754
Swap:             0           0           0

Are there any tests useful to try to crash the system accessing > 4GB to get an idea about the reaction for a "known issue"?

BTW:
After the PREP script and rebootet PI there was only one strange thing: Activating the WiFi was o.k. at the first step (changing WiFi country to DE, activating WiFi, scanning SSID, entering password, getting DHCP addresses, etc.). All seemed to be fine. Then, after a reboot there was no WiFi hardware present reported (or similar). After an additional reboot all is running fine.

[MichaIng]

Then, after a reboot there was no WiFi hardware present reported (or similar).

Hmm, either insufficient power or indeed something went wrong so that the dtoverlay was loaded that disables the internal WiFi adapter.

grep 'disable-wifi' /boot/dietpi.txt

This is not present on first boot and should of course also not be added when WiFi is enabled. But it will be added when disabling WiFi.

Are there any tests useful to try to crash the system accessing > 4GB to get an idea about the reaction for a "known issue"?

mount -o 'size=7G,remount' /tmp

Then you are able to copy/create large files in /tmp, e.g.:

cd /tmp
dd if=/dev/zero of=./testfile1 bs=1M count=2048
dd if=/dev/zero of=./testfile1 bs=2M count=2048
# If this work, >4 GiB RAM can be used obviously. To check for any limit:
dd if=/dev/zero of=./testfile2 bs=3M count=2048
dd if=/dev/zero of=./testfile4 bs=4M count=2048
rm ./testfile*

2 GiB chunks to rule out issues with 4 GiB file sizes.

[StephanStS]

df before mount:

root@DietPi:~# df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/root       30483740 3246032  25967488   12% /
devtmpfs         4051140       0   4051140    0% /dev
tmpfs            4084932       0   4084932    0% /dev/shm
tmpfs            4084932   10800   4074132    1% /run
tmpfs               5120       4      5116    1% /run/lock
tmpfs            4084932       0   4084932    0% /sys/fs/cgroup
tmpfs              51200      68     51132    1% /var/log
tmpfs            4104192       0   4104192    0% /tmp
/dev/mmcblk0p1    258095   54241    203854   22% /boot
tmpfs             816984       0    816984    0% /run/user/0

and after mount:

root@DietPi:/tmp# df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/root       30483740 3247820  25965700   12% /
devtmpfs         4051140       0   4051140    0% /dev
tmpfs            4084932       0   4084932    0% /dev/shm
tmpfs            4084932   10800   4074132    1% /run
tmpfs               5120       4      5116    1% /run/lock
tmpfs            4084932       0   4084932    0% /sys/fs/cgroup
tmpfs              51200      68     51132    1% /var/log
tmpfs            7340032       0   7340032    0% /tmp
/dev/mmcblk0p1    258095   54241    203854   22% /boot
tmpfs             816984       0    816984    0% /run/user/0

I.e. /tmp is increased in size (like wanted).

Then I did:

root@DietPi:/tmp# dd if=/dev/zero of=./testfile1 bs=1M count=2048
2048+0 Datensätze ein
2048+0 Datensätze aus
2147483648 bytes (2,1 GB, 2,0 GiB) copied, 5,85736 s, 367 MB/s

root@DietPi:/tmp# ls -l
insgesamt 2097152
drwx------ 3 root root         60 Jun  4 23:53 systemd-private-dc846a1735e74ad2bd3c35f6372f2782-redis-server.service-OusAE0
-rw-r--r-- 1 root root 2147483648 Jun  4 23:57 testfile1

root@DietPi:/tmp# dd if=/dev/zero of=./testfile1 bs=2M count=2048
2048+0 Datensätze ein
2048+0 Datensätze aus
4294967296 bytes (4,3 GB, 4,0 GiB) copied, 11,8302 s, 363 MB/s

root@DietPi:/tmp# ls -l
insgesamt 4194304
drwx------ 3 root root         60 Jun  4 23:53 systemd-private-dc846a1735e74ad2bd3c35f6372f2782-redis-server.service-OusAE0
-rw-r--r-- 1 root root 4294967296 Jun  4 23:58 testfile1

root@DietPi:/tmp# dd if=/dev/zero of=./testfile2 bs=3M count=2048
dd: Fehler beim Schreiben von './testfile2': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
1025+0 Datensätze ein
1024+0 Datensätze aus
3221225472 bytes (3,2 GB, 3,0 GiB) copied, 8,42517 s, 382 MB/s

root@DietPi:/tmp# ls -l
insgesamt 7340032
drwx------ 3 root root         60 Jun  4 23:53 systemd-private-dc846a1735e74ad2bd3c35f6372f2782-redis-server.service-OusAE0
-rw-r--r-- 1 root root 4294967296 Jun  4 23:58 testfile1
-rw-r--r-- 1 root root 3221225472 Jun  4 23:58 testfile2

root@DietPi:/tmp# df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/root       30483740 3247872  25965648   12% /
devtmpfs         4051140       0   4051140    0% /dev
tmpfs            4084932       0   4084932    0% /dev/shm
tmpfs            4084932   10800   4074132    1% /run
tmpfs               5120       4      5116    1% /run/lock
tmpfs            4084932       0   4084932    0% /sys/fs/cgroup
tmpfs              51200      68     51132    1% /var/log
tmpfs            7340032 7340032         0  100% /tmp
/dev/mmcblk0p1    258095   54241    203854   22% /boot
tmpfs             816984       0    816984    0% /run/user/0

root@DietPi:/tmp# dd if=/dev/zero of=./testfile4 bs=4M count=2048
dd: Fehler beim Schreiben von './testfile4': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
1+0 Datensätze ein
0+0 Datensätze aus
0 bytes copied, 0,0110152 s, 0,0 kB/s
root@DietPi:/tmp#

[Joulinar]

well it would be needed to remove the files after each try :) because the file system is just 7GB and after the 2nd run you already block 4GB due to the test file. For sure a lager file 3 will not fit and you will run out of space 😉

But I guess that was the purpose of the test. Anyway it might be a difference if you use tmpfs or if you have a process that tries to allocate the ram

[MichaIng]

well it would be needed to remove the files after each try :) because the file system is just 7GB and after the 2nd run you already block 4GB due to the test file. For sure a lager file 3 will not fit and you will run out of space 😉

The idea was to fill /tmp, se to keep the files 😉. I just wanted to do it in chunks to rule out any limitation with the per-file size.

So the result is great, somehow the 32 bit Raspbian is able to handle more then 4 GiB used RAM. I guess there might be limitations in other scenarios but I lack the detailed knowledge on how/where the 2³² byte = 4 GiB addressing limitation is worked around usually or where it might still become apparent. However for now it seems we do not need to take action on default /tmp size 😃.

[StephanStS]

One more test was to have a nearly full filesystem with 4M x 1500:

root@DietPi:/tmp# dd if=/dev/zero of=./testfile4 bs=4M count=1500
1500+0 Datensätze ein
1500+0 Datensätze aus
6291456000 bytes (6,3 GB, 5,9 GiB) copied, 16,8635 s, 373 MB/s

root@DietPi:/tmp# ls -l
insgesamt 6144000
drwx------ 3 root root         60 Jun  4 23:53 systemd-private-dc846a1735e74ad2bd3c35f6372f2782-redis-server.service-OusAE0
-rw-r--r-- 1 root root 6291456000 Jun  5 00:25 testfile4

root@DietPi:/tmp# df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/root       30483740 3333052  25880468   12% /
devtmpfs         4051140       0   4051140    0% /dev
tmpfs            4084932       0   4084932    0% /dev/shm
tmpfs            4084932   10800   4074132    1% /run
tmpfs               5120       4      5116    1% /run/lock
tmpfs            4084932       0   4084932    0% /sys/fs/cgroup
tmpfs              51200       8     51192    1% /var/log
tmpfs            7340032 6144000   1196032   84% /tmp
/dev/mmcblk0p1    258095   54241    203854   22% /boot
tmpfs             816984       0    816984    0% /run/user/0
root@DietPi:/tmp#

This shows that a single file with > 6 GB works.

[StephanStS]

@MichaIng: WiFi investigation is done tomorrow with a fresh install.
Now: One small beer, then go to sleep. ;-)

[StephanStS]

BTW: A short test was done: Nextcloud started, Webmin started, phpmyadmin started, mopidy started, phpsysinfo worked. No detailed tests were made, only installation and starting test.
I.e. no obvious issues detected.

[Joulinar]

to simulate the ram usage, you could install stress and try to use as much as possible ram on a single process 😃

apt install stress
stress --vm 1 --vm-bytes 1024M

on my RPi4B it's failing with 3072. Will try to install Raspi OS 64Bit Beta to verify this.

[MichaIng]

Good point that actually using (instead of just occupying) that mount of RAM might make a difference. Nice program stress, didn't know that 🙂 👍.

[Joulinar]

I continue to play around. Found some nice command on the web

stress --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.98;}' < /proc/meminfo)k --vm-keep -m 1 --timeout 20

It will try to load the mem up-to 98% (if possible) for 20sec. I did the test on my RPi4B (4GB) with Raspi OS 64bit as well as 32bit.

• 64bit: I was able to load the full memory + SWAP
• 32bit: Max load was 79% - around 2.9G

[MichaIng]

Hah, I didn't think about the swapfile. Might this mean that on 32-bit devices it doesn't make sense to create a swapfile if physical RAM is 4 GiB already, respectively that RAM + swap have a shared limit of 4 GiB? I'm still wondering why it is then able to create such large files. Probably the bs=1M (1 MiB block size) for dd worked around the 2³² addressing issue.

I'll ask this on stackexchange (respectively the correct child community), makes sense to get some more insights and I am simply interested as well. I.e. I have 4 GiB swapfile on my RPi, would be interesting to know if this is nonsense 😄.

[Joulinar]

well ist depends on the amount of processes. I mean just because a single process can't allocate more than 3G ram doesn't mean that you could not run 3 processes using 2G ram each 😉

EDIT:
this should force your system swapping. 4 processes, each try to allocate 25%
stress --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.25;}' < /proc/meminfo)k --vm-keep -m 4 --timeout 20

Bottom line: it depends on the usage of your system. If you have just a few applications with less mem usage, it's indeed nonsense as it will never be used. However if you have some heavy applications that are going to trans-code or render or huge db apps, it might make sense to avoid memory shortage. Anyway best is to never use swap as it will slow down your system.

[MichaIng]

Found it btw: https://askubuntu.com/questions/488828
PAE allows to use more then 4 GiB RAM on 32 bit OS.
The question is now it the official RPi kernel has PAE enabled, respectively the CPU even support it. I didn't find some clear answer about this yet.

😄 the 5 GiB RAM + swap are never used on my RPi, that's true. It's more like a "why not" thing, the swapfile is on a stable external drive. When creating RPi and ARMv7 images and compiling I ran into the situation where I needed a swapfile, so I just created it very large as it doesn't matter and is not used under normal circumstances anyway. I was just wondering if it can even be used correctly or not. Would be bad if e.g. it would be used on another heavy build and then it is not killed "regularly" due to OOM but somehow creates some bad corruption or crash due to > 4 GiB usage.

[Joulinar]

ok the system is able to operate with the 4 or 8gb ram, even if it is a 32bit OS. However a single process is not able to use it. Anyway I'm still lacking the fantasy what the hell process you are going to run on a SBC that will allocate more than 3gb ram alone 🤔

Ok maybe if you are going to compile huge stuff probably

[MichaIng]

That's true: https://stackoverflow.com/questions/33582419
So yeah still nothing to do our side then, since we have not really a chance to know how (with how many processes or threads) the memory is used but must not limit the overall RAM or swapfile size only to prevent >4 GiB single process usage.

[StephanStS]

@MichaIng: Regarding the WiFi problem mentioned above, testing with grep 'disable-wifi' /boot/dietpi.txt did never show a disabled WiFi.
Then I did the same procedure described above and the problem did not occur any more. No matter, why.
Therefore, we can omit this point. The issue stays closed and should be finished from my view.

[MichaIng]

New RPi image ready which should work on RPi4 8 GiB as well: https://dietpi.com/downloads/images/testing/DietPi_RPi-ARMv6-Buster.7z

[Joulinar]

@MichaIng
would it be possible to save the image as zip format? I'm playing with some remote image installation over the web but it seems it require zip file instead of 7z 😄 Just for testing 😉

[Joulinar]

So finished installation on my RPi3B+. Seems to be working fine. Image was flashed from local network to SD card without issues. Attached the log from ssh cli

install.txt

[MichaIng]

Ah indeed 7z has no data stream functionality, xz has but cannot extract 7z, does it? So this requires download, extract and dd in three separate steps 😉.

Actually, I was thinking to switch to tar.xz for all your downloads by times. 7zip for Windows can handle it, hence no difference from that point of view. Benefit is that tar container can hold UNIX meta data (owner, modes, symlinks, ...) and xz allows data streaming while using the same lzma(2) compression algorithm. Other benefit is that p7zip basically is dead since 4 years.

Many thanks for testing, I hope an RPi4 8 GiB user will test it as well before I move it to stable, just to be sure 😉.

[Joulinar]

Maybe we would need rethinking the test farm to be able testing as much as possible ourselves. :thinking:

[MichaIng]

Yes indeed a test farm would be great, however I have not the possibilities to run one currently. As fast as I can, I guess Fourdee would send me most of the supported SBC models. And regardless I'll order an RPi4 8 GiB once the most relevant RPi feature have been ported to arm64. Practically impossible else to finally add 4k support and do the display options rework. Also it will greatly speed up ARM image building including ARMv8.

[MichaIng]

Support for Raspberry Pi OS (64-bit) has been added and first alpha image ready: https://github.com/MichaIng/DietPi/issues/3570