Elasticsearch: Watcher inconsistencies with hidden indices

[joegallo]

Given this setup:

PUT ordinary-index-2020.11.17/_doc/1
{
  "foo":"bar"
}

PUT hidden-index-2020.11.17/_doc/1
{
  "foo":"bar"
}

PUT hidden-index-2020.11.17/_settings
{
   "index.hidden": true
}

PUT _watcher/watch/some-watch
{
  "trigger" : {
    "schedule" : {
      "yearly" : { "in" : "january" }
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "ordinary-index*" ],
        "body" : {
          "query" : {
            "match_all" : {}
          }
        }
      }
    }
  },
  "actions" : {
    "log_error" : {
      "logging" : {
        "text" : "ctx.payload.hits.total was {{ctx.payload.hits.total}}"
      }
    }
  }
}

POST _watcher/watch/some-watch/_execute
{
  "record_execution" : true
}

You will end up with these indices on a fresh 7.x cluster:

GET _cat/indices?s=index&v&expand_wildcards=all
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-ilm-history-5-000001       EhTUkJclTzugXWOV9H9Iiw   1   0          4            0     16.8kb         16.8kb
green  open   .watcher-history-13-2020.11.17 fjsVgpL3Rl-Regpd65Y_1A   1   0          2            0     13.1kb         13.1kb
green  open   .watches                       rzufqdImQKadvUQxJUm7cg   1   0          1            0      7.9kb          7.9kb
yellow open   hidden-index-2020.11.17        qNiLM9g1QsyPja9-sMnVkQ   1   1          1            0      3.7kb          3.7kb
yellow open   ordinary-index-2020.11.17      qHRVJZSFT_WRsMDE5JYkZQ   1   1          1            0      3.7kb          3.7kb

Note, in particular, that .watcher-history-13-2020.11.17 is hidden:

GET .watcher-history-13-2020.11.17/_settings
{
  ".watcher-history-13-2020.11.17" : {
  [...]
        "hidden" : "true",
  [...]
  }
}

Then the following watcher execution template results in sometimes surprising results:

POST _watcher/watch/_execute
{
  "watch" : {
    "trigger" : {
      "schedule" : {
        "yearly" : { "in" : "january" }
      }
    },
    "input" : {
      "search" : {
        "request" : {
          # see snippets below
          "body" : {
            "query" : {
              "match_all" : {}
            }
          }
        }
      }
    },
    "actions" : {
      "log_error" : {
        "logging" : {
          "text" : "ctx.payload.hits.total was {{ctx.payload.hits.total}}"
        }
      }
    }
  }
}

Search request snippets and results:

"indices" : [ "ordinary-index-*" ],
-->
 "logged_text" : "ctx.payload.hits.total was 1" # unsurprising, it's not hidden

"indices" : [ "hidden-index-*" ],
-->
 "logged_text" : "ctx.payload.hits.total was 0" # unsurprising, it's hidden

"indices" : [ "hidden-index-*" ],
"indices_options" : {
  "expand_wildcards" : "all"
},
-->
 "logged_text" : "ctx.payload.hits.total was 0" # surprising, 'all' didn't work (potential bug A)

"indices" : [ ".watcher-history-*" ],
-->
"logged_text" : "ctx.payload.hits.total was 1" # surprising, since it *is* hidden (potential bug B)

Potential bug A

It looks like WatcherSearchTemplateRequest needs to learn about hidden rather than just open and closed.

Potential bug B

It's not clear to me why this is happening. 😬

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[joegallo]

In the same space as potential bug B, and indeed perhaps as an extension of the same bug, is the following behavior:

GET ordinary-index-*/_count
--> 1 # no surprise here

GET hidden-index-*/_count
--> 0 # no surprise here, it is hidden after all

GET hidden-index-*/_count?expand_wildcards=all
--> 1 # no surprise here

GET .watcher-history-*/_count
--> 1 # wait, I thought .watcher-history-13-2020.11.17 was hidden?

[jaymode]

Potential bug A

You're correct, this is a bug where watcher needs to know about hidden :(.

Potential bug B

This is expected and the naming of the pattern indicates this is an attempt to access a hidden index. The hidden paradigm is kind of like *nix shells and dot files where they are hidden unless you specify a pattern that begins with a . like .e* or .ss*. Does that make sense?

[joegallo]

Does that make sense?

Yes, I think so. Essentially GET .watcher-history-*/_count is implicitly ?expand_wildcards=all (or, well, maybe hidden) because of the leading dot at the beginning of the wildcard, right?

[jaymode]

Essentially GET .watcher-history-*/_count is implicitly ?expand_wildcards=all (or, well, maybe hidden) because of the leading dot at the beginning of the wildcard, right?

It is essentially adding hidden as an expand wildcard option (not exactly how it is implemented but how it functions)