Fluentd: The in_tail plugin with wildcard path misses the first entries in a newly-created file

[jasonzio]

in_tail misses data in new files with wildcard path

The in_tail plugin, when setup to watch a wildcard path, will “miss” the first entries in a newly-created file in the watched path.

Version information

• fluentd 0.12.24
• Ubuntu 16.04

fluentd config

<source>
  @type tail
  path /logz/*
  tag multilogs
  pos_file /pos/multilogs
  format none
  path_key tailed_path
</source>

<match multilogs>
  @type file
  path /out/multi.json
  format json
  include_time_key true
  flush_at_shutdown true
  flush_interval 30s
</match>

Repro steps

1. Start fluentd with /logz as an empty directory.
2. echo “This is the first record” >> /logz/test
3. sleep 3
4. echo “This is the second record” >> /logz/test
5. Look at /out/multi.json - there will be one chunk, and it will only contain the second record.

Suspected cause

The code that watches the path directory for new entries sees the new file and starts the tail task. By the time the tail task gets around to opening the file and advancing to the end, the first entry has already been written, and the “seek to end” operation misses it.

Typical scenario

All output from docker containers is directed to a single folder. As containers are started, new files spring into existence, capturing the stdout and stderr from the containers. A naïve use of fluentd would use in_tail with path set to the docker output folder; that will miss the first lines emitted by a new container. One wouldn’t want to simply specify “read_from_head true” in the config, though; in that case, a restart of fluentd would double-pump all the logged output for every container that had already been started and whose logs hadn’t been scrubbed.

Suggested Fix

There should be a way to say “I want read_from_head behavior only for files discovered after fluentd has started; if the file exists during the initial enumeration of the files matching the path, seek to the end before reading.” Ideally, read_from_head would change from a binary to a ternary: “false” means always seek to the end, “true” means always start from the head, and “new” would seek to the end for files that existed when path was first scanned and start from head for newly-discovered files that match path.

[repeatedly]

“true” means always start from the head, and “new” would seek to the end for files that existed when path was first scanned and start from head for newly-discovered files that match path.

It seems current behaviour with pos_file.
Could you show me reproducible step with pos_file and read_from_head true combo?

[lavoiedn]

We're having this issue too.

My understanding is that, with read_from_head true, all existing log files that are discovered will be sent from the start, which can cause issues with large existing files being sent to, say, a Splunk indexer.

Currently, we'd have to start fluentd once with read_from_head false to create the pos_files and then once they are properly created, change the config to read_from_head true to get complete logs from newly created log files?

This is where being able to specify a setting to have existing log files on fluentd startup be read from the last line, but files created afterwards follow read_from_head true.

Feel free to correct me if I'm misunderstanding how read_from_head true works

[jasonzio]

@lavoiedn is exactly correct: read_from_head true _all the time_ is not the desired behavior. I want to start from current end-of-file for all the files that exist _when the plug-in starts_; after that point in time, all new files should be treated as if read_from_head were true.

[marcin-je]

We are experiencing same issue - first few log lines of containers are not processed by in_tail

[robbinscp]

I'm still seeing the same thing in our deployment of fluentd with a wildcard path and forwarding to EFK. fluentd shows it is reading the file, but we don't get data until the file gets written to again (similar to what @jasonzio has observed). Our application (Magento) is only writing to files in a particular directory once.

[kecin17]

I'm having the same issue. First few lines not processed when log rotating. Any solution regarding this?

[fcecagno]

Any intention to implement this?
IMHO it's an important request since for just created files it's required to use read_from_head true otherwise you'd miss content, but then you'll be processing old logs you might not be interested in.
If there were an easy way to create pos_file pointing to the file tail for existing logs, it would be helpful in this case, while it isn't properly handled by the plugin.

[fcecagno]

I wrote a small snippet to initialize pos_file for existing files we needed to use read_from_head true and were not interested in old content: https://gist.github.com/fcecagno/c46494b520151cf7fee21421560efbb8