Beats: Metricbeat index date timezone

[Exocomp]

Per https://discuss.elastic.co/t/metricbeat-index-date-timezone/78296.

How do you control the timezone metricbeat uses for the date part of the index name:

For ex:

metricbeat-2017.03.12

Is there a way I can tell metricbeat to use a particular timezone when generating the date part above. I'd like it to use UTC (which is what Filebeat and winlogbeat uses) but is using local.

The reason why this causes a problem is that filebeat and winlogbeat indices rollover to the next day (since it uses UTC) but metricbeat doesn't (since it uses local timezone) so they are not in sync and using 3rd party tools where you set the timezone for the index doesn't work because they are not all the same.

Thanks,
E

[Elufimov]

I have the same problem and I think this is a bug.

[ruflin]

I'm surprised that filebeat and metricbeat do not use the same indices as the same logic is used: https://github.com/elastic/beats/blob/master/libbeat/outputs/elasticsearch/client.go#L370. As the timestamp is used for the index name, somewhere in metricbeat the timestamp must be set differently. Which metricsets are you using? Can you share your config?

[Exocomp]

@ruflin

I found out the difference, where it is working with filebeat and winlogbeat I'm sending it to through logstash where I'm using the date filter and setting the timezone, I'm sending metricbeat to elasticsearch directly.

[ruflin]

Thanks for the update. Does that mean all implementations do not report UTC?

[Exocomp]

@ruflin

It is just for the date part of the index, as you say they all share same logic so if you send the other beats to elasticsearch directly I bet those index dates would be local also. But I haven't tested the others going to elasticsearch directly just metricbeat.

[urso]

Checking the implementation, the '@timestamp' value is reported in UTC (timestamp is normalized before serializing), but when building the index or using the time format-string, the timestamp is still local time. This is true for all beats.

My proposal would be to always normalize the timestamp to UTC (so timestamps/events are comparable between different timezones) and report the locale in the event beat namespace. If required, one can use local and @timestamp to create the local timestamp in logstash or ES ingest node.

[ruflin]

This processor is the first step to get the timezone into the event: https://github.com/elastic/beats/pull/3902

[andig]

Not sure this is related but looking at https://discuss.elastic.co/t/metricbeat-timestamp-mapping-as-string-type-should-be-date/112521 I've noticed that metricbeat timestamp is not even a date but a string according to the index mapping- might this be part of the issue?

[andig]

Here's a comparison of the mapping definitions:

{
"metricbeat-6.1.1-2017.12.19": {
    "aliases": {},
    "mappings": {
    "doc": {
        "_meta": {
        "version": "6.1.1"
        },
        "dynamic_templates": ...
        "date_detection": false,
        "properties": {
        "@timestamp": {
            "type": "keyword",
            "ignore_above": 1024
        },


{
"filebeat-6.1.1-2017.12.19": {
    "aliases": {},
    "mappings": {
    "doc": {
        "_meta": {
        "version": "6.1.1"
        },
        "dynamic_templates": ...
        "date_detection": false,
        "properties": {
        "@timestamp": {
            "type": "date"
        },

[ruflin]

@andig I think the "wrong" template is cause by not loading a template before starting the beat? I assume that is related to the your own arm build?

[andig]

I assume that is related to the your own arm build?

Yes. Very sorry if this idea was offtopic- can no longer reproduce with the official release.

[fjgal]

@ruflin I faced the same issue as describe here.

It appears that the function libbeat/elasticsearch/client.go#getIndex is only converting the timestamp to UTC for index extension when the "index" metadata is available in the event. Otherwise it uses the default mechanism to format the index string (index.Select(event)) which doesn't convert the Timestamp to UTC.

I submitted a PR (#6485) that intends to solve the issue. I had to solve the issue in formatevents.go so it may affect several places and a couple of unit tests broke. So I am not sure sure whether this is an acceptable solution.

I modifed existing unit tests in libbeat/elasticsearch/client.go to expose and reproduce the issue.

We already have this fix deployed in our environment, which uses our own beat, and we have not observed any undesired side effects.

I'd like to hear your feedback on the PR.

[ruflin]

@fjgal Thanks, will continue the discussion on the PR.

[dariusj18]

This has been sitting around for a while, is this issue ever going to be resolved?

My two cents, My server is TZ-5, I use Grafana to visualize my data, when I am viewing any chart using an index from Metricbeat from 8 to midnight I get no data, because Grafana specificly queries the index with the UTC date. But all the data is still going into the prior day's index.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.