Beats: Filebeat sets type automatically

[mut3]

I recently began migrating the shippers to my ELK stack from logstash forwarder to filebeat. Unfortunately the log lines I get from filebeat blow past all my filters without getting parsed in any useful way.

This is because my filters check the type field of the events and filebeat sets a type field of "log" automatically at origin.

From the Beats input plugin in Logstash

Types are used mainly for filter activation.
...
If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.

So should I stop using type to do filter activation?
Is there already a way to properly set the type in filebeat?

If not, there should be a way in filebeat config to set the type or to not add the field on in the first place. I would rather set this at the Logstash level depending on the input port

[mut3]

I realized that this new repo would be a better place for this issue than the old one https://github.com/elastic/filebeat/issues/310

[mut3]

Aaaaaaand, I was using

fields:
  type: <type-name>

but not the correct

document_type: <type-name>

This was just a case of me failing to RTFM.

I might go open an issue for the beats input docs to note that setting a type field there is basically useless since beats will almost always send a type.

[ruflin]

@mut3 Thanks for also posting the conclusion in case someone else stumbles over the issue.

[falzm]

Hi

I have an issue similar to this one, however in my case it's not related to using document_type: I have filebeat reading a file containing JSON-formatted logs which already contain a type field, and I need to keep this field unaltered through Logstash and then Elasticsearch indexing. Is there a way to instruct filebeat NOT to override type type field of processed events?

[ruflin]

@falzm Currently there isn't. As you seem to have Logstash in place, you should try to do such transformations on the Logstash side. I assume that is also where your JSON read?

[falzm]

@ruflin the type field rewriting occurs before Logstash has any chance to transform anything. The JSON log lines are read by filebeats and would be shipped to multiple Logstash instances using the loadbalance feature.

[urso]

filebeat is not parsing your json, but forwarding only lines as is. I think this can be done in logstash:

1. no codec in beats input plugin
2. use mutate filter to rename or remove type field
3. use json filter to parse the json from message

[falzm]

@urso thank you for this suggestion, I'll try it out.

[falzm]

@urso works as suggested, thanks again. For the record if anybody faces the same challenge, here is a working Logstash configuration:

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "beat" {
    mutate {
      remove_field => ["type", "beat", "input_type", "offset", "source", "fields"]
    }

    json {
      source => "message"
    }

    mutate {
      remove_field => "message"
    }
  }
}