Fluent-bit: @include files using parser_files or wildcarding for parser_files

[thrift24]

I am trying to use a configuration file that I have pulled in with an @include to set an additional parser file with parsers_file, however it doesn't appear that this can be done as the parsers_file directive must be defined in the [SERVICE] section, however there can only be one [SERVICE] section and attempting to define it again in the include leads to the parsers_file directive not being read.

I have also tried wildcarding the parsers_file to be able to include multiple parsers files whose name I may not know when I create the main configuration. The @include usage is severely limited if new parsers cannot be included as well.

Imagine the situation that I want one main distribution of the td-agent-bit which includes a prebuilt conf file, but where I may have other packages that can be installed and bring their own included configuration (for example apache and nginx) to have their logs handled by fluent-bit. I can easily wrap up the /etc/td-agent-bit/td-agent-bit.conf and I can @include include/*.conf to allow the apache/nginx packages to provide their own include/apache.conf and include/nginx.conf, however they each want to provide and call their own parsers from their conf file..... I can't seem to figure out a way to do that. Is this possible or should there potentially be a way to append to parsers_file from within each included configuration? Or even a way to wildcard parsers_file so the packages can bring their own and have them loaded in that manner?

[edsiper]

At the moment you can add many Parsers_File entries in the service section,
no wildcard support for now, but make sense to implement a parsers_dir.

On Fri, Oct 12, 2018, 01:18 thrift24 [e-mail] wrote:

I am trying to use a configuration file that I have pulled in with an
@include https://github.com/include to set an additional parser file
with parser_files, however it doesn't appear that this can be done as the
parser_files directive must be defined in the [SERVICE] section, however
there can only be one [SERVICE] section and attempting to define it again
in the include leads to the parsers_file directive not being read.

I have also tried wildcarding the parser_files to be able to include
multiple parsers files whose name I may not know when I create the main
configuration. The @include https://github.com/include usage is
severely limited if new parsers cannot be included as well.

Imagine the situation that I want one main distribution of the
td-agent-bit which includes a prebuilt conf file, but where I may have
other packages that can be installed and bring their own included
configuration (for example apache and nginx) to have their logs handled by
fluent-bit. I can easily wrap up the /etc/td-agent-bit/td-agent-bit.conf
and I can @include https://github.com/include include/*.conf to allow
the apache/nginx packages to provide their own include/apache.conf and
include/nginx.conf, however they each want to provide and call their own
parsers from their conf file..... I can't seem to figure out a way to do
that. Is this possible or should there potentially be a way to append
parser_files from within each included configuration? Or even a way to
wildcard parser_files so the packages can bring their own and have them
loaded in that manner?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluent-bit/issues/846, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWkNikwzbbD9hblNq6WIKOgBDioPVrHks5ukCYTgaJpZM4XYzmS
.

[thrift24]

Yes, the issue with that is you have to know the names of all the
parsers_files before hand with that limitation. In this manner the
@include statements can be used for configuration organization, but doesn't
aid at all in configuration packaging. It would be very nice if each new
package could just drop their config and parser somewhere and be ready to
go without having to modify preexisting configurations. I don't believe
this is possible currently.

On Fri, Oct 12, 2018, 1:40 AM Eduardo Silva [e-mail]
wrote:

At the moment you can add many Parsers_File entries in the service section,
no wildcard support for now, but make sense to implement a parsers_dir.

On Fri, Oct 12, 2018, 01:18 thrift24 [e-mail] wrote:

I am trying to use a configuration file that I have pulled in with an
@include https://github.com/include to set an additional parser file
with parser_files, however it doesn't appear that this can be done as the
parser_files directive must be defined in the [SERVICE] section, however
there can only be one [SERVICE] section and attempting to define it again
in the include leads to the parsers_file directive not being read.

I have also tried wildcarding the parser_files to be able to include
multiple parsers files whose name I may not know when I create the main
configuration. The @include https://github.com/include usage is
severely limited if new parsers cannot be included as well.

Imagine the situation that I want one main distribution of the
td-agent-bit which includes a prebuilt conf file, but where I may have
other packages that can be installed and bring their own included
configuration (for example apache and nginx) to have their logs handled
by
fluent-bit. I can easily wrap up the /etc/td-agent-bit/td-agent-bit.conf
and I can @include https://github.com/include include/*.conf to allow
the apache/nginx packages to provide their own include/apache.conf and
include/nginx.conf, however they each want to provide and call their own
parsers from their conf file..... I can't seem to figure out a way to do
that. Is this possible or should there potentially be a way to append
parser_files from within each included configuration? Or even a way to
wildcard parser_files so the packages can bring their own and have them
loaded in that manner?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluent-bit/issues/846, or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAWkNikwzbbD9hblNq6WIKOgBDioPVrHks5ukCYTgaJpZM4XYzmS

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluent-bit/issues/846#issuecomment-429210314,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AXyjNNllY4WWug8GcyXgxaZF5uMnnJIrks5ukCtjgaJpZM4XYzmS
.

[donbowman]

I'd like to see this done the way that (most) linux things do: make the config read all the files in a conf.d directory.

e.g. we could have /etc/fluent-bit/parsers/conf.d
each package 'foo' could drop a 'foo.conf' in there, and it would be loaded.

this would allow the regex to be e.g. individual rather than merged, etc.

[thrift24]

So it appears that there is already an easier way to do this and we can likely skip the whole Parsers_dir idea. The documentation doesn't make this as crystal clear as it could be, so perhaps we should consider updating that, but the situation is that:

the parsers configuration itself supports @include, which I did not realize.

This basically allows the following to happen:

  td-agent-bit.conf: can include all the seperate configurations as, for example,  conf.d/*.conf
                              can load a single master parsers file, for example, parsers.conf
  parsers.conf: can include all the seperate parsers as, for example, conf.parsers.d/*.conf

So this all works just fine without any patches or anything like that. My confusion comes from the documentation not stating that the parsers configuration can support that.

So here's the main configuration documentation that I have been using:
https://fluentbit.io/documentation/current/configuration/file.html
You can see that at the bottom of the file it has an Include section for this main configuration.

Here is the parsers configuration documentation that I have been using:
https://fluentbit.io/documentation/current/parser/
You can see that nowhere in the file does it mention that any type of include is possible, and as these files contain different configuration items (despite having a similar layout), it is not immediately obvious that an include can be done from here. It also seems like no one commenting on this issue was aware that it could do this. My testing has shown that this works just fine.