Kibana: [SIEM] Exceptions from a custom date time mapping

[FrankHassanabad]

Describe the bug:

In the SIEM application if you have a particular type of custom date time mapping shown below you will end up getting errors in the UI within SIEM.

Steps to reproduce:

Create this loose date time mapping like so for a new index and then reindex some data into it:

PUT /frank-test-delme
PUT /frank-test-delme/_mapping
{
    "date_detection": true,
    "numeric_detection": false,
    "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
    ],
    "dynamic": "true",
    "properties": {
        "@timestamp": {
            "type": "date",
            "format": "strict_date_optional_time"
        }
    }
}

POST _reindex
{
  "conflicts": "proceed", 
  "source": {
    "index": "filebeat-8.0.0-2020.01.01-000008"
  },
  "dest": {
    "index": "frank-test-delme"
  }
}

Then set in advanced to use this new index like so:

Load the SIEM application and you see this error:

In Kibana logs you see:

Error: [parse_exception] failed to parse date field [1582852870977] with format [strict_date_optional_time]: [failed to parse date field [1582852870977] with format [strict_date_optional_time]]
    at respond (/Users/frankhassanabad/projects/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/frankhassanabad/projects/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/frankhassanabad/projects/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/frankhassanabad/projects/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

However, loading discovery and using discovery I do not see any of these errors. Only when I use the SIEM UI

You can use the dev tools to replicate this same error with this query:

GET /frank-test-delme/_search
{
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "must": [],
              "filter": [
                {
                  "bool": {
                    "filter": [
                      {
                        "match_all": {}
                      },
                      {
                        "bool": {
                          "filter": [
                            {
                              "bool": {
                                "should": [
                                  {
                                    "range": {
                                      "@timestamp": {
                                        "gte": 1582918232745
                                      }
                                    }
                                  }
                                ],
                                "minimum_should_match": 1
                              }
                            },
                            {
                              "bool": {
                                "should": [
                                  {
                                    "range": {
                                      "@timestamp": {
                                        "lte": 1583004632745
                                      }
                                    }
                                  }
                                ],
                                "minimum_should_match": 1
                              }
                            }
                          ]
                        }
                      }
                    ]
                  }
                }
              ],
              "should": [],
              "must_not": []
            }
          },
          {
            "match_all": {}
          }
        ]
      }
    },
    "sort": [
      {
        "@timestamp": "desc"
      },
      {
        "_doc": "desc"
      }
    ]
}

Changing the above query to utilize "format": "epoch_millis" when doing range queries seems to fix the problem and running this query below shows that there is no longer an error:

GET /frank-test-delme/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "bool": {
                  "filter": [
                    {
                      "match_all": {}
                    },
                    {
                      "bool": {
                        "filter": [
                          {
                            "bool": {
                              "should": [
                                {
                                  "range": {
                                    "@timestamp": {
                                      "gte": 1582918232745,
                                      "format": "epoch_millis"
                                    }
                                  }
                                }
                              ],
                              "minimum_should_match": 1
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "range": {
                                    "@timestamp": {
                                      "lte": 1583004632745,
                                      "format": "epoch_millis"
                                    }
                                  }
                                }
                              ],
                              "minimum_should_match": 1
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "match_all": {}
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": "desc"
    },
    {
      "_doc": "desc"
    }
  ]
}

Kibana version:
7.6.0

Elasticsearch version:
7.6.0

Workaround:
Re-work your date time mappings to use Elastic defaults for date time mappings.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[sim0nx]

I have the same issue. Any news on this one ? :-)

[spong]

Thanks for +1'ing and letting us know @sim0nx -- this helps us with prioritization! No specific date for when we'll have a fix, but we're tracking this internally as well, so stay tuned 🙂

[jeethsam]

I face the same issue

[FrankHassanabad]

"Reviewed by Frank Hassanabad on 7/29/2020", This looks complete and will be shippable at this point. Going to close this and associated tickets. Any new bugs/features should be new tickets.

Other notes is that new features, new developers, existing developers with endpoints and data might from time to time forget to add the prescribed fixes as it is counter-intuitive to the natural defaults of date time stamps of features of Elastic Search. If you encounter a new issue, please open that new issue against the new feature/API, or existing one as a regression rather than re-opening this one and we will fix it in those areas and keep a watchful 👁️ to hopefully not introduce this again.