netdata/error.log: postdrop: warning: mail_queue_enter: create file maildrop/615227.4142: Read-only file system

[araemo]

Bug report summary

Lots of errors in error.log:

postdrop: warning: mail_queue_enter: create file maildrop/615227.4142: Read-only file system

(the numbers change each time). I believe this started after I configured postfix on my system (mail was completely unconfigured originally)

I am using Ubuntu Server 19.10. I found a related item: #1582 , but it has been closed and someone asked us to open a new issue.

I also noticed that with this issue, netdata never fully loads. The service is running, and it accepts HTTP connections, but the page never displays, and it doesn't not report to the nodes page (nodes page lists no address for the host, and cannot connect).

Once I fixed this (via the suggestion in this comment: https://github.com/netdata/netdata/issues/1582#issuecomment-549714472 ), everything is working again.

I copied /lib/systemd/system/netdata.service to /etc/systemd/system/netdata.service so I could override the default, and added the ReadWriteDirectories entry:

ReadWriteDirectories=/var/spool/postfix/maildrop

As an additional point of note, I did su to the netdata user and ran /usr/lib/netdata/plugins.d/alarm-notify.sh, which completed successfully and I received the 3 test emails, so it isn't a literal permissions issue, but seems to be a systemd sandbox/restrictions issue.

OS / Environment

Linux smaug 5.3.0-48-generic #41-Ubuntu SMP Fri Apr 10 06:59:18 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=19.10
/etc/lsb-release:DISTRIB_CODENAME=eoan
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 19.10"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION="19.10 (Eoan Ermine)"
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:PRETTY_NAME="Ubuntu 19.10"
/etc/os-release:VERSION_ID="19.10"
/etc/os-release:HOME_URL="https://www.ubuntu.com/"
/etc/os-release:SUPPORT_URL="https://help.ubuntu.com/"
/etc/os-release:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
/etc/os-release:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
/etc/os-release:VERSION_CODENAME=eoan
/etc/os-release:UBUNTU_CODENAME=eoan

Netdata is running locally:

Netdata version

netdata v1.16.1

Component Name

Steps To Reproduce

1. Install netdata on ubuntu 19.10 with apt
2. Configure postfix
3. Netdata stops responding and starts spamming the log with the error above.

Expected behavior

Netdata should continue to collect and display data, and send email alerts.

[mologie]

This also affects the netdata release distributed with Ubuntu 20.04 LTS.

A less intrusive workaround is to add a systemd override file via systemctl edit netdata.service with the following content:

[Service]
ReadWriteDirectories=/var/spool/postfix/maildrop

Note: systemctl edit merely writes your changes to /etc/systemd/system/netdata.service.d/override.conf and reloads the service file.

Afterwards, restart netdata via systemctl restart netdata.