Beats: Automatically lowercase the index name when custom fields is used to build it

[daohodac]

As uppercase is invalid in elasticsearch indices, Beats should either provide a lowercase processor or always ensure index names are lowercased when using custom fields

Use case (filebeat):

A document:

{"HDG": 125.0, "fwot": "FOO", "date": "2018-01-31 23:00:03"}

A beat configuration:

.......
- type: log
  paths:
    - /var/log/myfeed/*.log
  json.keys_under_root: true
......
output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "http"
  username: "elastic"
  password: "changeme"
  index: "custom-%{[fwot]}-%{+yyyy.MM}-v1"
.....

[ph]

I would prefer explicit lowercase of the field vs magic. @daohodac are you getting bulk insertion error when uppercase are present?

[daohodac]

@ph well, not an error, but an error log,. Nothing inserted, just fails.
And beat does not have a dead letter, so it only appears on the logs

[kvch]

Hits to get started:
A new processor needs to be added to lowercase field values. Example processor: https://github.com/elastic/beats/blob/master/libbeat/processors/actions/truncate_fields.go

[urso]

I wonder if the processor is the right place. Maybe we should not tamper with event contents itself, but ensure that the strings we extract when creating the index name are lower-case.

[Joginder22]

+1 facing same issue with ES output when index name is in uppercase. I'ts better to have lowercase keyword in the output to make every index name in lowercase letters only.