Beats: Filebeat should do log rotation

[bsikander]

Log rotation is a very common scenario with log files. Filebeat should have the option to do log rotation.

Initial discussion: https://discuss.elastic.co/t/log-rotation-and-filebeat/140285

[ruflin]

To prevent any confusion: Filebeat support rotated/renamed files. Identification of files is not based on the filename but inode + device id. Let's continue the discussion on the discuss topic.

[kvch]

As @ruflin pointed out Filebeat indeed supports reading from rotated files.

However, by this request I mean that Filebeat should rotate the files it is reading. @bsikander Does it reflect what you requested in the second point of your list? Do you mind changing the description to add retails regarding Filebeat doing the rotation?

[jmkgreen]

You're talking about the logs Filebeat is reading, and not the logs Filebeat produces itself which is another way of reading the current title.

[tdmalone]

The title could possibly be updated, but the way I understand this is... if filebeat could support log rotation for files it reads, we wouldn't need to worry about configuring logrotate as well. Once filebeat has shipped off the logs and a configurable time has passed, it could remove the old logs from the system. That'd be awesome!

[tomgrossman]

@ruflin looking at the initial discussion by @bsikander and this issue correspondence, I'm a little confused. In the discussion itself another one of your team member says:

We cannot guarantee... Unfortunately, Filebeat does not support log rotation yet.

and here you say:

To prevent any confusion: Filebeat support rotated/renamed files

So, i'm still confused :)
How should I configure my filebeat to support log rotation by linux logrotate on my nginx access/error logs?
If I will add in the YML all of the possible log file names (0,1,2...), will it handle it once the files are being rotated?

[ruflin]

Let me try to clear up the confusion here:

• Filebeat supports following rotated files. Meaning if logrotate renames a file to .1 Filebeat is able to understand that log was rotated and continues reading.
• Filebeat does not support rotating / renaming log files of an other service. Meaning Filebeat can't be used as a replacement for logrotate or other similar tools.

I think what you are looking for falls under the first use case and is supported. Instead of defining 1,2,3 just use a * pattern.

[tomgrossman]

@ruflin that does clears it up, thanks!

[waldoweng]

I just got an idea. Could filebeat just hard link the log files it's watching and remove the hard link after filebeat done reading it ? This should solve the data losing case where log files were rotate and filebeat restart shortly after.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[frittentheke]

Still a good idea.

[freddie2020]

Hello,

I have the same need.

Is this going to be something for future releases of filebeat agent?

Thanks, freddie2020

[LoicAG]

Also interested in seeing this happen

[gk-fschubert]

+1