Packages: syslog-ng: clash with ubox logd when both are required

[muddyfeet]

Maintainer: @BKPepe
Environment: Linksys WRT1200ac trunk

Commit fe165eeb94d2e7fb35fee93e64336afa024a1283 prevents both ubox logd and syslog-ng coexisting in the same builds. Previously, I was using the vanilla openwrt logd to forward events to syslog-ng at 127.0.0.1 which allowed both to run on the one device. This allowed you to display the syslog in Luci and also take advantage of the extra capability of syslog-ng (I note that this change allows you to achieve this with just syslog-ng now).

Happy to delete the ubox logd from my builds and rework the logging if that is required - but I'm not sure this was intended. The make -j1 V=s output is shown below:

Collected errors:
* check_data_file_clashes: Package syslog-ng wants to install file /home/muddyfeet/openwrt-trunk/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/root-mvebu/sbin/logread
But that file is already provided by package * logd
* opkg_install_cmd: Cannot install package syslog-ng.
package/Makefile:65: recipe for target 'package/install' failed
make[2]: *** [package/install] Error 255
make[2]: Leaving directory '/home/muddyfeet/openwrt-trunk'
package/Makefile:108: recipe for target '/home/muddyfeet/openwrt-trunk/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/stamp/.package_install' failed
make[1]: *** [/home/muddyfeet/openwrt-trunk/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/stamp/.package_install] Error 2
make[1]: Leaving directory '/home/muddyfeet/openwrt-trunk'
/home/muddyfeet/openwrt-trunk/include/toplevel.mk:225: recipe for target 'world' failed
make: *** [world] Error 2

[muddyfeet]

Further information:

Building the firmware without logd and only syslog-ng completes successfully.

I can run only syslog-ng to collect the logs but it presents a new problem. My syslog-ng files are being created on an attached USB disk with a new file created per day (e.g. /mnt/sda1/logs/syslog-2020-03-11.log). However, logread is hard coded to use the RAM based syslog at /var/log/messages - so this prevents Luci from displaying the logs in my config.

As I want to keep the logs between restarts of the router (so they need to go to a permanent disk file) and not have them fill up all available ram between restarts, I need both a circular log in /var/logs/messages and a permanent log elsewhere. I can get syslog-ng to create both files but haven't been able to figure out how to get syslog-ng to create a circular logfile for the ram based /var/log/messages file.

My solution so far is to run a cron job every night linking the latest disk log to the /var/log/messages file which at least allows logread to run:

for file in $( find /mnt/sda1/logs -type f ) ; do
    [ -z "$newest" -o "$newest" -ot "$file" ] && newest=$file
done
ln -sf "$newest" /var/log/messages 

For reference here is my syslog-ng.conf

@version:3.17

options {
    chain_hostnames(no);
    create_dirs(yes);
    flush_lines(0);
    keep_hostname(yes);
    log_fifo_size(256);
    log_msg_size(8192);
    stats_freq(0);
    flush_lines(0);
    use_fqdn(no);
    keep-timestamp(no);
    frac-digits(2);
    ts-format(iso);
};

source net {
    udp(ip(0.0.0.0) port(514));
    tcp(ip(0.0.0.0) port(514));
};

source src {
        internal();
        unix-dgram("/dev/log");
};

source kernel {
        file("/proc/kmsg" program_override("kernel"));
};

destination messages {
        # Log to a file on the USB filesystem, with a new file every day
        file("/mnt/sda1/logs/syslog-$YEAR-$MONTH-$DAY.log");
};

log {
        source(src);
    source(net);
        source(kernel);
    destination(messages);
};

# @include "/etc/custom-logs.conf"

[Catfriend1]

Could I please have the old behaviour that the syslog-ng installs its version of logread to /usr/sbin ? I have scripts that depend on /sbin/logread and other depend on /usr/sbin/logread.

[muddyfeet]

A good logic would be to revert the installed location of syslog-ng's version of logread to /usr/sbin and link /sbin/logread to /usr/sbin/logread only if logd is not installed.

[dibdot]

Maybe we should make LuCI more flexible to accept both logread locations.
@BKPepe should I prepare a PR in the LuCI repo?

[muddyfeet]

@dibdot great idea - that would work for me. @Catfriend1 said they have scripts that use both locations - may need to deal with that somehow.

[BKPepe]

Could I please have the old behaviour that the syslog-ng installs its version of logread to /usr/sbin

I responded in the commit, which was mention in the OP and, by the way, it was on the same day. And I still find it as low priority currently for me. I am fully aware that I break it for someone and I am responsible for it, but I decided to go this way of changing the location of logread. This was less pain for other people, who are using syslog-ng by default and they don't use OpenWrt system log implementation. Because without it, Luci was not cooperating and it was unable to display system log and this was the reason, which I will mention further.

My comment below the commit:

Thanks for reporting. I wonder, why you would like to have installed logd and syslog-ng at the same time. I have installed syslog-ng and I don't use logd at all that's why I haven't met such issue so far. I was forced to move the logread script different path, which expects LuCI as it couldn't view the system/kernel log in it as LuCI does not find it. It is hard-coded here.

I am sorry that you can not install syslog-ng for now. I will look at it, but with the low priority for me currently. I'm thinking that I will introduce ALTERNATIVES for syslog-ng and as well logd. I can appreciate if you can create an issue and assign it to me instead of commenting it below commit.

Hard-coded paths in LuCI: https://github.com/openwrt/luci/commit/939b371bc72f4c020c499c815f636dafc296829a#diff-e32045eab7d6e4701afc02a6fa707e8dR47 and we discussed it here as well https://github.com/openwrt/packages/pull/11523#pullrequestreview-370802123. I see in the LuCI pull request that @jow- suggested the better solution would be to use $PATH instead of using hard-coding, which is appreciated.

@BKPepe should I prepare a PR in the LuCI repo?

I think that this should be most likely handled by alternatives. You don't want to have two files with the same name located in different paths. This could lead to several issues or misunderstandings. And I shouldn't forget that each byte on low storage devices count, right? For now, it is this or logd. Not good, but still useful.

@muddyfeet used two services, which were responsible for logging and for some devices it can be heavy. The primary reason, why he used it was that LuCI does not display system log as it was not found and logd was able to show it there. I will check the status of this patch https://gitlab.labs.nic.cz/turris/turris-build/blob/hbk/patches/packages/to-upstream/0014-syslog-ng-rotate-default-log-with-logrotate.patch which should allow rotate /var/log/messages to prevent fulfilling RAM. But hopefully, you are not using any flash storage for logging. It can go pretty bad.

[neheb]

@BKPepe will you be reverting the change? It seems luci was fixed.

[BKPepe]

I need to take a look. Hopefully, I will do it this week. The better solution would be implement alternatives.

[Catfriend1]

Meanwhilst, I've fixed my scripts to auto detect via "opkg list | grep -q "syslog-ng"" and reading the openwrt version to use the correct line matching regex and /sbin/... location.