Collectd: [tail plugin - does not preserve the last parsed line between collectd restarts]

[ser]

When you restart collectd, tail plugin parses the file from the beginning, which makes total mess in stats.

It should preserve the last analysed line during collectd shutdown.

[notpeter]

My assumption was that the tail plugin would follow a given file emitting data as new log entries were written. I can't imagine any scenario where the current behavior is desirable. I don't believe collectd has an existing framework for persisting plugin state (e.g. file offset) between invocations.

So I'm proposing the default tail plugin behavior be changed to follow the principle of least astonishment. Specifically, only handle lines in a log file which are written while the plugin is running, e.g. from the TAIL of the opened file. Note: seek to EOF only happens on initial plugin invocation; when a log rotation has occurred (inode change, offset beyond EOF) the entire contents of the file should still be matched to catch up.

Yes, when collectd is restarted some entries might be missed, but I'm totally ok with my collectd counters being incorrect for time(s) while collectd wasn't running. Currently anytime collectd is restarted (config changed, pkg upgraded, system rebooted) the tail plugin greedily consumes the entire file and emit a crazy town value which makes tail plugin metrics impossible to graph sanely.

Thoughts?

More info @ Plugin:Tail docs

[rpv-tomsk]

This issue is not confirmed. Mentioned "total mess in stats" may be caused by incorrect ds-types in used datasets or in DSType option, or types mismatch with RRD file (if used).

Tail plugin has logic to do EOF seek on file first open since 2008, and it works. My simple tests confirm this.
Code Example: https://github.com/collectd/collectd/blob/b978177200519893f6ff9c5e459a3d9d6a72b887/src/utils_tail.c#L78

  /* Seek to the end if we re-open the same file again or the file opened
   * is the first at all or the first after an error */
  if ((obj->stat.st_ino == 0) || (obj->stat.st_ino == stat_buf.st_ino))
     seek_end = 1;

If you have any thoughts please open new issue, I will take care.

[ser]

I do not understand that piece of code, how a restarted collectd instance knows it re-opens the file and which line number, where that information is stored between restarts?

[rpv-tomsk]

Hi! Thanks for your attention to this issue ;-)

When collectd starts, it creates obj structure and fills it with zero bytes:

cu_tail_t *cu_tail_create (const char *file)
{
    cu_tail_t *obj;

    obj = (cu_tail_t *) malloc (sizeof (cu_tail_t));
    if (obj == NULL)
        return (NULL);
        memset (obj, '\0', sizeof (cu_tail_t));

So, the check in cu_tail_reopen() will be true and seek_end will be set.

  /* Seek to the end if we re-open the same file again or the file opened
   * is the first at all or the first after an error */
  if ((obj->stat.st_ino == 0) || (obj->stat.st_ino == stat_buf.st_ino))
     seek_end = 1;

Last processed position is not saved between restarts, but Collectd should read file from EOF.
And behaviour should match to described by @notpeter in his comment.