Fluentd: placeholder in Buffer Path / file symlink_path is not work ?

[agrozyme]

• fluentd version: 1.2.4
• Environment information: Docker 18.03.1-ce, Alpine 3.8
• Your configuration:

<source>
  @type forward
  @label @mainstream
  port 24224
</source>

<filter **>
  @type stdout
</filter>

<label @mainstream>
  <match **>
    @type file
    append  true
    path /var/log/fluent/${tag}.%Y%m%d
    symlink_path /var/log/fluent/${tag}.log
    <buffer tag,time>
      timekey 1d
      flush_mode interval
      flush_interval 1s
    </buffer>    
  </match>
</label>

@include /usr/local/etc/fluent/*.conf

• Fact

drwxr-xr-x. 2 core core 4096 Aug 18 13:58 '${tag}.%Y%m%d'
lrwxrwxrwx. 1 core core   74 Aug 18 13:58 '${tag}.log' -> '/var/log/fluent/${tag}.%Y%m%d/buffer.b573b6133a79e860a56be70dd326ae51f.log'
drwxr-xr-x. 3 core core 4096 Aug 18 13:58  .
drwxr-xr-x. 6 core core 4096 Aug 18 07:40  ..
-rw-r--r--. 1 core core 1245 Aug 18 13:58  local_httpd.20180818.log

• Wish

drwxr-xr-x. 2 core core 4096 Aug 18 13:58 local_httpd.20180818
lrwxrwxrwx. 1 core core   74 Aug 18 13:58 local_httpd.log -> '/var/log/fluent/local_httpd.20180818/buffer.b573b6133a79e860a56be70dd326ae51f.log'
drwxr-xr-x. 3 core core 4096 Aug 18 13:58  .
drwxr-xr-x. 6 core core 4096 Aug 18 07:40  ..
-rw-r--r--. 1 core core 1245 Aug 18 13:58  local_httpd.20180818.log

[repeatedly]

This is similar case with https://docs.fluentd.org/v1.0/articles/out_file#i-can-see-files-but-placeholders-are-not-replaced,-why?
symlink_path refers temporal buffer files so placeholders are not replaced.

[athurg]

This is similar case with https://docs.fluentd.org/v1.0/articles/out_file#i-can-see-files-but-placeholders-are-not-replaced,-why?
symlink_path refers temporal buffer files so placeholders are not replaced.

That page just explained how/what it worked as, not about why or why not.

The feature @agrozyme request looks very useful, especially while we use one match directive to match more than one tag, and we need to split each tagged log (and it's immediately buffer) to independence file.

Currently, we can use placeholder on match's pach params, so when the buffer flushed, we can got log files split by tag.

But before the buffer flushed, there are many content in it. If we need to check those lines by tail -f, we can't find any way to pair the tag with the buffer file's random name.

So could you please support some way ,to make tail -f check the lines by tag or other chunk keys?

By the way , if we set the buffer's flush mode as immediate, to force the lines in buffer as less as we can. There are some other bugs. While many logs enters in a short time. Many temporary buffer file/memory are created, and never be removed.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days