Beats: Filebeat 'system' module assumes UTC in ingest pipeline

[inputvalidation]

My initial question on ES discuss:

I'm using filebeat to import syslog messages. When looking at the ES document it appears filebeat incorrectly assumes UTC:

ES document:

"@timestamp": "2017-04-01T15:26:51.000Z"
Syslog message

"timestamp": "Apr 1 15:26:51"
OS:

Sat Apr 1 15:26:51 CEST 2017

Should filebeat not store the @timestamp as 13:26:51?

Response from @tsg:

Thanks for the feedback, I think a variable for the timezone makes sense in this module. Could you open a Github ticket for it, please? It's not trivial to implement because currently we insert the pipelines "as is" without expanding variables into them, but I think that's something we can add.

For confirmed bugs, please report:

• Version: 5.3.0
• Operating System: Debian 8 (using ES repo)
• Steps to Reproduce: add syslog messages in localtime using filebeat, messages are stored in UTC while the time is still localtime and not converted to UTC

[dbluxo]

This issue should be labeled as a bug. My analysis:

When using Filebeat 5.3.0 the module "system" is writing the false timestamp to Elasticsearch.
As of Kibana:

Actually this message was written at 2017-04-06T15:20:07.149Z
System timezone: Europe/Berlin

For example the module "apache2" ist working correctly, as of Kibana:

This message was written at 2017-04-06T15:11:27.521Z
System timezone: Europe/Berlin

Version: 5.3.0
Operating System: Amazon Linux (Cent OS 6)

[ruflin]

I think https://github.com/elastic/beats/pull/3902 will be a first step to solve the problem.

[dbluxo]

My workaround, in file: /usr/share/filebeat/module/system/syslog/ingest/pipeline.json

Change:

  "date": {
    "field": "system.auth.timestamp",
    "target_field": "@timestamp",
    "formats": [
      "MMM  d HH:mm:ss",
      "MMM dd HH:mm:ss"
    ],
    "ignore_failure": true
  }

To:

  "date": {
    "field": "system.auth.timestamp",
    "target_field": "@timestamp",
    "formats": [
      "MMM  d HH:mm:ss",
      "MMM dd HH:mm:ss"
    ],
    "timezone" : "Europe/Amsterdam",
    "ignore_failure": true
  }

Delete the "old" pipeline:

curl -XDELETE "http://localhost:9200/_ingest/pipeline/filebeat-5.3.0-system-syslog-pipeline"

Make sure that the "new" pipeline was loaded:

curl -XGET "http://localhost:9200/_ingest/pipeline/filebeat-5.3.0-system-syslog-pipeline"

[andrewkroh]

I think #3902 will be a first step to solve the problem.

We need a way to tell the date ingest processor about our beat.timezone field so that it can use the value of this field (if present) when processing the date.

@talevy Is this something you would consider adding to the processor? If so I can open an enhancement ticket on ES side.

[tsg]

I was also looking at the same. The Logstash date filter does support dynamic fields for the timezone, so it would make sense to allow it in the Ingest Node.

[talevy]

@tsg the modification to allow templating of timezone and locale parameters has been merged and will be available in 6.1