Beats: [filebeat] How do I write a processor plugin by myself ?

[wenma]

filebeat verson: 6.2

[andrewkroh]

At a high level a Processor must implement the Processor interface and a constructor function must be registered by calling RegisterPlugin so that your Processor can be instantiated from config by the Beat.

You can look at the add_host_metadata processor as an example. In order for the init() function to be invoked that register the plugin the package needs to be imported and Beats does this from here. This is an example of a processor that is statically compiled into the Beat.

You can do this without entirely forking Beats. You just need to duplicate the Filebeat main.go and add in your import (_ "github.com/me/beats-processor-xyz"). Then compile this and you'll have a customized filebeat.exe containing your processor.

Another option (for Linux) is to compile the processor into a shared object (.so) and register it with Beats by using --plugin ./myprocessors.so. IMO this is more complicated because there are strict requirements that must be met in order to use plugins in Go. You must target an exact version of Filebeat by compiling against the elastic/beats source from that release and you must compile using the same exact Go version. An example of this is my beats-processor-fingerprint plugin.

[wenma]

@andrewkroh awesome, I'll try it, thanks

[wenma]

@andrewkroh when I write a plugin, I face this problem:

# >  filebeat  -e  --plugin  ./myplugin.so
Exiting: plugin.Open: plugin was built with a different version of package github.com/elastic/beats/vendor/github.com/pkg/errors

the package github.com/elastic/beats/xxx/xxx, I use the source code of v6.2.3, and my filebeat is also version 6.2.3, what's wrong with my step ?

[andrewkroh]

Can you try copying your processor into your checkout of beats, then build from there. This will allow your code to use the vendored dep instead of a different one and hopefully resolve the issue.

cp -rp my_processor github.com/elastic/beats/libbeat/processors/
cd github.com/elastic/beats/libbeat/processors/my_processor
go build -buildmode=plugin