Logstash: Add ability to set all fieldnames to lowercase

[jsvd]

MIGRATED FROM: https://logstash.jira.com/browse/LOGSTASH-732

Text:

Since there's no way to make searching on field names case insensitive, a filter/feature to set all fieldnames to lowercase would be helpful. For example, if you have two different log sources and one uses "hostname" as a field name and the other uses "Hostname" as a field. By setting all fieldnames to lower case, it would allow joining of log source searches.

[hanej]

+1

[yukiman76]

+1

[dforste]

It is dirty but it works:

filter {
  ruby { code => "
event_hash = event.to_hash
new_event = {}
event_hash.keys.each do |key|
  new_event[key.downcase] = event[key]
end
event.instance_variable_set(:@data, new_event)
" }
}

[willemdh]

+1

[kepten]

+1

[missnebun]

+1

[chandlermelton]

+1

[benben]

+1

[mut3]

+1

[ksmets]

+1

[mut3]

Since this has gone a year without any change, lets talk about how this filter would function.

It would need to have a behavior set ( and probably options to change this behavior) for the case of one event having two fieldnames that now conflict when both set to all lowercase.

For example:

"Hostname" => "localhost",
"hostName" => 127.0.0.1

should the lowercase_fieldname filter. . .

merge?

"hostname" => ["localhost", 127.0.0.1]

overwrite?

"hostname" => 127.0.0.1

drop?

"hostname" => "localhost"

or some other more intelligent operation?

[ksmets]

When duplicates occur in the kv filter, an array is created so merging the event would seem like the standard operation, not?

[samuraiii]

dforste
It is dirty but it works:

filter {
  ruby { code => "
event_hash = event.to_hash
new_event = {}
event_hash.keys.each do |key|
  new_event[key.downcase] = event[key]
end
event.instance_variable_set(:@data, new_event)
" }
}

Works nicely but it is not recursive:
Message "Ip=1.2.3.4 error=REFUSED" with kv once set to insert into root of event and "ruby hack":

{
       "message" => "Ip=1.2.3.4 error=REFUSED",
      "@version" => "1",
    "@timestamp" => "2016-05-09T08:31:13.692Z",
          "host" => "kibana",
           "ip" => "1.2.3.4",
         "error" => "REFUSED"
}

And with kv set to insert into test subdocument of event and "ruby hack"

{
       "message" => "Ip=1.2.3.4 error=REFUSED",
      "@version" => "1",
    "@timestamp" => "2016-05-09T08:30:31.518Z",
          "host" => "kibana",
          "test" => {
          "Ip" => "1.2.3.4",
        "error" => "REFUSED"
    }
}

Any ideas to make it recoursive?

[joemiller]

I ran into strange behavior with the ruby filter recommended by @dforste. In particular, it seemed like some of my other filters were not working (in particular, mutate/add_field and mutate/remove_field filters were not actually modifying events). I switched to using this instead and things seem to be working better for our config now:

code => "event.to_hash.keys.each do |k|
                 next unless k =~ /[A-Z]/
                 event[k.downcase] = event[k]
                 event.remove(k)
               end"

[jeremyforan]

+1

[chicco27]

+1

[ceeeekay]

+1

[marke72]

+1

[marke72]

In 5.0 they changed how to access and set the keys and values of the event. I was originally using @dforste's method but now I'm using a modified version of @joemiller's method.

ruby {
  code => "
    event.to_hash.keys.each do |k|
      event.set(k.downcase, event.remove(k))
    end
  "
}

[ph]

This feel to me that a plugin should do that, I don't think the core should provide Indifferent access to to keys.

[v01t]

@marke72's method works great for 5.x, but I would be happy to find some way to use such method recursively or for specific child key like foo.bar.baz to downcase not only root (parent) objects but also nested.