Dietpi: Dietpi-software/apt-get crashes

[mukamuk1]

Details:

• Date | Fri 13 Jul 09:56:39 BST 2018
• Bug report | N/A
• DietPi version | v6.11
• Img creator | mukamuk1
• Pre-image | Rasbian 5.41 kernel 4.4.119
• SBC device | Asus Tinker Board (armv7l) (index=52)
• Kernel version | #2 SMP Thu Mar 8 07:04:46 CET 2018
• Distro | stretch (index=4)
• Command | G_AGI: bash-doc -s
• Exit code | 100
• Software title | DietPi-Software

ALSO:

• Date | Fri 13 Jul 12:12:32 EEST 2018
• Bug report | N/A
• DietPi version | v6.10
• Img creator | n/a
• Pre-image | n/a
• SBC device | RPi 2 Model B (armv7l) (index=2)
• Kernel version | #1106 SMP Wed Apr 4 18:19:23 BST 2018
• Distro | stretch (index=4)
• Command | G_AGI: bash-doc -s
• Exit code | 100
• Software title | DietPi-Software

Steps to reproduce:

1a. start dietpi-software
2a. try to install ANY software

ALSO:
1b. apt-get install freeradius

Expected behaviour:

• software should install normally

Actual behaviour:

• dietpi-software/apt-get crashes due to write error

Extra details:

Behavior started after update to new version of dietpi

root@DietPi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            939M     0  939M   0% /dev
tmpfs           202M   21M  181M  11% /run
/dev/mmcblk0p1   29G  4.8G   24G  17% /
tmpfs          1006M  4.0K 1006M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1006M     0 1006M   0% /sys/fs/cgroup
tmpfs            10M  1.5M  8.6M  15% /DietPi
tmpfs           2.0G     0  2.0G   0% /tmp
tmpfs            20M   20M     0 100% /var/log

root@DietPi:~# sudo df -i /
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/mmcblk0p1 1908736 165110 1743626    9% /

Additional logs:

Log file contents:
Inst bash-doc (4.4-5 Debian:9.4/stable [all])
Conf bash-doc (4.4-5 Debian:9.4/stable [all])
E: Write error - ~LZMAFILE (28: No space left on device)

[gittusmaximus]

Sounds like the same problem here https://github.com/Fourdee/DietPi/issues/1915

[MichaIng]

@mukamuk1
Thanks for your report.

tmpfs 20M 20M 0 100% /var/log

Your log directory is full. That should usually not occur with dietpi-ramlog enabled, as it clears the log directory every hour.

Can you paste result of

G_TREESIZE /var/log
grep '=2' /DietPi/dietpi/.installed

to check which log takes so much space and which software (e.g. the related one) you installed via dietpi-software.

---

Quick solution would be to manually do the dietpi-ramlog clean step
dietpi-logclear 1
or uninstall dietpi-ramlog/choose logging mode "none" (fallback to journalctl only) or "full" (full rsyslog + logrotate) to allow log to disk instead of ramdisk.

But if really something produces 20M logs within less than an hour, then you should identify and fix or remove the related software.

[mukamuk1]

root@DietPi:/var/log# G_TREESIZE /var/log 20.0 MB /var/log 132.0 KB /var/log/nginx 64.0 KB /var/log/redis 8.0 KB /var/log/letsencrypt 4.0 KB /var/log/postgresql 0.0 KB /var/log/samba 0.0 KB /var/log/ntpstats 0.0 KB /var/log/news 0.0 KB /var/log/fsck 0.0 KB /var/log/apt 0.0 KB /var/log/apache2
and
root@DietPi:/var/log# grep '=2' /DietPi/dietpi/.installed aSOFTWARE_INSTALL_STATE[7]=2 aSOFTWARE_INSTALL_STATE[9]=2 aSOFTWARE_INSTALL_STATE[16]=2 aSOFTWARE_INSTALL_STATE[17]=2 aSOFTWARE_INSTALL_STATE[73]=2 aSOFTWARE_INSTALL_STATE[102]=2 aSOFTWARE_INSTALL_STATE[103]=2 aSOFTWARE_INSTALL_STATE[104]=2 aSOFTWARE_INSTALL_STATE[126]=2 aSOFTWARE_INSTALL_STATE[130]=2 aSOFTWARE_INSTALL_STATE[140]=2
What makes log so long is motioneye and mastodon BUT there is log entries worth of few days. Ramlog broken down July 9th and log slowly filled tmpfs. It seems that there has been problem in ramlog after 6.10 update.

[Fourdee]

@mukamuk1

root@DietPi:/var/log# G_TREESIZE /var/log 20.0 MB /var/log 132.0 KB /var/log/nginx 64.0 KB /var/log/redis 8.0 KB /var/log/letsencrypt 4.0 KB /var/log/postgresql 0.0 KB /var/log/samba 0.0 KB /var/log/ntpstats 0.0 KB /var/log/news 0.0 KB /var/log/fsck 0.0 KB /var/log/apt 0.0 KB /var/log/apache2

Strange, this does not add up. Are we missing a few entries? (eg: apache2). Or unstable system/memory corrupting tmpfs?

What makes log so long is motioneye and mastodon BUT there is log entries worth of few days

Ideally, we need to see the contents of the logs, however, sending a bug report may fail due to 0 free space on /var/log.

Please run the following:

dietpi-logclear 2

Then upload the log files from /root/logfile_storage/ somewhere that we can view.

Once done, please send a bug report, so we can verify system settings:

dietpi-bugreport

[MichaIng]

@mukamuk1 @gittusmaximus
Ah G_TREESIZE only shows folder sizes, not file sizes, sorry please show instead (before doing dietpi-logclear):
ls -al /var/log

dietpi-bugreport fails because of this large /var/log, as we allow just 10M maximum upload size. Hmm we should actually increase this, if we allow /var/log size of 20M?

What makes log so long is motioneye and mastodon BUT there is log entries worth of few days. Ramlog broken down July 9th and log slowly filled tmpfs. It seems that there has been problem in ramlog after 6.10 update.

Hmm, okay makes sense. /var/log should be not mounted from tmpfs without the related cron job (/etc/cron.hourly/dietpi) regularly calling dietpi-logclear to clear logs.

If somehow this job does not work, please try to switch to logging mode "none" and back to either "clear hourly" or "clear hourly + save".
Then run-parts /etc/cron.hourly to check if the cron job then successfully clear logs.

We had other cron job failure reports which should not be related, but just in case, this is the related fix: https://github.com/Fourdee/DietPi/commit/82ac7b32d32dca9db4fdb824c7ead80174844090

[mukamuk1]

Sorry, I cleared logs already. So no logs to send to you :-(

Big logfile was syslog and daemon.log in Dietpi1 and it was full of errors from my mastodon installation and motioneye (motion is not working on Asus TB). Both logs contains basically same errors from same sources. Dietpi2 had big pihole.log.

Remember that same situation was in 2 dietpi installations with totally different software mix and hardware. Same situation still persists as I can see that dietpi-ramlog logfile clearing is not working (in Dietpi1) because entries in logs are older than 1 hour. I will let logfiles grow for a while and send bug report before any of /var/log files are 10M. I rebooted Dietpi2 to see if dietpi-ramlog logfile clearing functions after reboot. I will send more info tomorrow.

[MichaIng]

@mukamuk1
Okay, it seems indeed that your issue is related to this: https://github.com/Fourdee/DietPi/issues/1901#issuecomment-405075590

• Please try: wget https://raw.githubusercontent.com/Fourdee/DietPi/82ac7b32d32dca9db4fdb824c7ead80174844090/dietpi/func/dietpi-globals -O /DietPi/dietpi/func/dietpi-globals
• Afterwards the cron jobs should succeed in cleaning the log directory regularly.

[MichaIng]

@mukamuk1
To clean up a bid I opened a new issue, please try to apply the fix mentioned there and report back, if it solves all issues: https://github.com/Fourdee/DietPi/issues/1923

I will close this issue in favour of the new one.