Elasticsearch: EQL: index patterns not recognized for data streams

[rw-access]

Index names for data streams are recognized by EQL, both with the hidden backed indices as well as the full name. But index _patterns_ are not.

Create the data stream

PUT /_index_template/my-data-stream-template
{
  "index_patterns": [ "my-data-stream*" ],
  "data_stream": { }
}

Add a document

POST /my-data-stream/_doc/?refresh=wait_for
{
  "@timestamp": "2020-12-06T11:04:05.000Z",
  "process": {
    "name": "cmd.exe"
  },
  "event": {
    "category": "process"
  }
}

```javascript
POST /my-data-stream/_rollover

Search the data stream by the full name
```javascript
GET /my-data-stream/_eql/search
{
  "query": "process where process.name == \"cmd.exe\""
}

// returns this result
{
  "is_partial" : false,
  "is_running" : false,
  "took" : 1,
  "timed_out" : false,
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "events" : [
      {
        "_index" : ".ds-my-data-stream-000001",
        "_type" : "_doc",
        "_id" : "L3ouxHMBxE7N-EVkNADM",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-12-06T11:04:05.000Z",
          "process" : {
            "name" : "cmd.exe"
          },
          "event" : {
            "category" : "process"
          }
        },
        "fields" : {
          "@timestamp" : [
            "1607252645000"
          ]
        }
      }
    ]
  }
}

Search the index pattern

GET /my-data-stream*/_eql/search
{
  "query": "process where process.name == \"cmd.exe\""
}

// get this result
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [my-data-stream*]",
        "index_uuid" : "_na_",
        "index" : "my-data-stream*"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [my-data-stream*]",
    "index_uuid" : "_na_",
    "index" : "my-data-stream*"
  },
  "status" : 404
}

But if you use _search it works

GET /my-data-stream*/_search
{
  "query": {
    "match_all": {}
  },
  "size": 1
}

// get this result
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-my-data-stream-000001",
        "_type" : "_doc",
        "_id" : "L3ouxHMBxE7N-EVkNADM",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-12-06T11:04:05.000Z",
          "process" : {
            "name" : "cmd.exe"
          },
          "event" : {
            "category" : "process"
          }
        }
      }
    ]
  }
}

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[astefan]

@rw-access I've just tested this on master and I am not seeing the issue. With or without * wildcard the request returns the same output. Can you, please, double check now to see if the issue is still there?

[costin]

Let's add datastreams to our integration testing to remove any doubt.

[rw-access]

We sure this is working? This is what I get on 7.9 (not sure if this is fixed in master, but not 7.9. that could explain the inconsistencies):

// returns 1 result
GET logs-endpoint.alerts-default/_search
{
  "query": {
    "match_all": {}
  },
  "size": 1
}

// returns 1 result
GET logs-endpoint.alerts-default/_eql/search
{
  "query": """
    any where true
  """
  ,
  "size": 1
}

// returns 1 result
GET logs-endpoint.*/_search
{
  "query": {
    "match_all": {}
  },
  "size": 1
}

```javascript
// “no such index [logs-endpoint.]”
GET logs-endpoint./_eql/search
{
"query": """
any where true
"""
,
"size": 1
}

[rw-access]

Backing datastream indices are also not matched

// 1 result returned
GET .ds-logs-elastic.*/_search
  "query": """
    any where true
  """
  ,
  "size": 1
}

// no results, no errors
GET .ds-logs-elastic.*/_eql/search
{
  "query": """
    any where true
  """
  ,
  "size": 1
}

[astefan]

Update: after a lot of tests local and remote on Cloud, the same build on both Cloud and local seem to work differently, as in locally the wildcard on data streams works, on Cloud they do not.

[rw-access]

Thank you @martijnvg!