Kibana: Index Pattern wizard does not work on empty indices

[loekvangool]

Kibana version: 6.1.0

Description of the problem including expected versus actual behavior:
The new Index Patterns wizard does not allow me to continue unless I have data in my indices. I'd like to import some dashboards/viz/saved searches into Kibana before putting actual data there but to import successfully, the index pattern has to exist. In 5.x I could define a pattern in Kibana before putting any data into the indices.

Steps to reproduce:

1. Start with a fresh ES/KB install
2. Create an index (Put Mapping)
3. Try to create an index pattern from that new index. It will state "couldn't find any Elasticsearch data" and block.
   
   
   
   1. If I click "Include system indices", I can manually enter a pattern. It states "The index pattern you've entered doesn't match any indices"
   
   

[grieke]

I have the same issue with version 6.1.0. I went back to version 6.0.1, and everything works like it did before.

[chrisronline]

@skearns64 Thoughts on this? Should we support this? We can probably add some kind of option that is like "Yes, I know this index does not exist and I still want to make the index pattern"

[loekvangool]

Actually, it does exist but it's empty @chrisronline

[skearns64]

Yea, in this case (the index exists, but is empty), I believe should allow the index pattern to be created.

I believe this issue is caused by the way we search for index names today [1] - we do one search and aggregate on the _index response. If there are no documents in the matching index, it doesn't show up in the response, because there is nothing to aggregate on.

I wonder if we could either make a second request to retrieve the mappings for the specific index, or try an approach that doesn't require the search + aggregate (though I recall this approach having security and possibly performance advantages). Would something like a GET /<provided index pattern>/_alias or GET /<provided index pattern>/_mappings give us enough information?

[1] https://github.com/elastic/kibana/blob/6fb7d5238629b9dce63ba1b6f66547ef1da6ab54/src/core_plugins/kibana/public/management/sections/indices/create_index_pattern_wizard/create_index_pattern_wizard.js#L92

[chrisronline]

Definitely worth looking into. I can take a pass later this week

[ycombinator]

If you are looking for just an existence check on a single index, HEAD <index name> would work. You might also consider GET <index name>/_stats/docs to get a minimal amount of info back, including the number of docs in the index.

[tavagyan-bb]

+1 waiting for the fixing of the issue asap as according to my project structure I need to have all my indexes imported in Kibana even empty.
Please note that I did some experiments and could import empty index to Kibana by removing "disabled" property for "Next step" button from network console. :)

[loekvangool]

+1 waiting for the fixing of the issue asap as according to my project structure I need to have all my indexes imported in Kibana even empty.

@tavagyan-bb If you're in a hurry, note that it's perfectly possible to create index patterns on empty indices in Kibana by adding documents to .kibana, and soon by using the Kibana API. I think you can work around this issue so it will not impact your project.

[naisanza]

I was wondering why it wasn't picking up my index like it did back when I last used ELK, in 4.0 days.

The index does exist, it just doesn't have anything in it.

[spicoflorin]

This problem occurs also in Kibana version 6.5.4. Is this problem fixed? Can you please mention in which version?

[Hronom]

So what the status of this, how I can disable this warning/error? I'm on 6.4.2 version now

[elasticmachine]

Pinging @elastic/kibana-app-arch

[mattkime]

addressed by https://github.com/elastic/kibana/pull/70271