rsyslog assumes UTC if TZ env var is not set and misses using the actual system's localtime

[JPvRiel]

When RFC3164 messages are processed (e.g. via remote inputs), they lack timezone information (unfortunate, but this is the legacy standard). Nonetheless, the most common/safest assumption is then that localtime was applied and that the source shares the same timezone (not always true, but the more likely case).

When relaying RFC3164 messages to other systems as RFC5424 (or using %timestamp:::date-rfc3339% in a template) _AND_ with TZ not set, rsyslog will assume it's operating in a UTC timezone (even if alternate OS and libc system calls could be used to find out the actual timezone).

Desired behaviour (when timezone info is not present in source timestamp)

rsyslog finds the system localtime and assumes the source system is in the same timezone.

And if TZ env var is not found, rsyslog should either log and/or output an error message stating that it requires this to be set in order to process messages without timezone info. Alternatively, rsyslog can fall back to using system calls such as tzname in libc which should be provided by most POSIX systems.

Actual behaviour

rsyslog ignores the system's localtime (if TZ environment variable is not set) and coverts and processes the timestamp using UTC.

Impact

If a system running syslog does not have TZ set, messages without timezone info that are relayed by rsyslog in RFC5424 format will assume UTC instead of localtime and causing any system-to-system syslog relay within any timezone other to have an incorrect timestamp transform applied.

Cause

RSyslog assumes TZ will always be set, but this might not be the case. E.g. Major distros like ubuntu don't set TZ by default.

See:

• How setting the TZ environment variable avoids thousands of system calls
• How do I find the current system timezone?

[rgerhards]

mmmhhh... Honest question: rsyslog does not touch TZ. So the user has unset it. Should we really overrule that (un)setting? @davidelang I'd also appreciate your comment

[rgerhards]

Major distros like ubuntu don't set TZ by default.

ummm... isn't that a bug in the distro?

[JPvRiel]

ummm... isn't that a bug in the distro?

If it is, it's a bug that appear both in Ubuntu 16.04.x and Red Hat 7.x as far as I've seen. And during install, I did go through timezone setup. I'm not sure if or how TZ is considered mandatory? What is mandatory for a POSIX OS is to supply those timezone system calls (via libc)?

As per How setting the TZ environment variable avoids thousands of system calls

simply set the TZ environment variable to :/etc/localtime (or some other timezone file of your choice) for a process. This will cause glibc to avoid making extra (and unnecessary) system calls.

So using TZ can help glibc be more efficient, but using the system calls in glibc (e.g. when the rsyslog daemon starts) up is possibly safer than expecting the TZ env var to be set?

Also, this is somewhat related to https://github.com/rsyslog/rsyslog/issues/424. But I do understand always checking / using the system calls for timezone every message would be inefficient, hence perhaps just at startup, but then again, not sure how daylight savings time could cause pain with that, so perhaps a periodic check at midnight every day also makes sense?

(*disclaimer, I haven't checked how rsyslog looks for localtime, but other logging utilities log logstash do manage to infer my systems localtime without needing TZ to be set)

[rgerhards]

you are probably right, but you see me pretty surprised ;-)

[rgerhards]

sorry, need to shuffle to 8.33 -- will try to do it there as one of the first things.

[rgerhards]

seems to be a problem in containers as well

[rgerhards]

I now went through all the options. It looks like the best way to do it is actually set TZ=/etc/localtime on rsyslog startup if TZ is not set and that file exists. I have verified that it exists on all major Linux versions. It won't work on Solaris etc (I think), but in that case I can set TZ=UTC and emit a warning messages. Better than nothing...

@JPvRiel thanks again for bringing this up and the good description and links. I'll now go ahead and implement the proposal.

[JPvRiel]

TZ=/etc/localtime

Thanks, good fix. The way I worked around it (my use-case was running rsyslog in docker) was to ensure that env variable was set.

[JPvRiel]

Two updates related to this for posterity sake:

1. When I logged this issue, the UTC problem was because I was running rsyslog in a container, and it's actually Docker that seems to force UTC in containers, even if the host OS has a proper timezone set. Checking for TZ handles this issue.
2. Still not sure about How setting the TZ environment variable avoids thousands of system calls and have not benchmarked it myself, but it would be interesting to test out in conjunction with TimeRequery as a tunable to handle the expense of time system calls for imudp.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.