Kibana: Unable to delete patterns if the indices don't exist

[markwalkom]

Kibana version: 5.0.1

Elasticsearch version: 5.0.1

Server OS version: Ubuntu 14.04

Browser version: Chrome

Browser OS version: OS X 10.12.1

Original install method (e.g. download page, yum, from source, etc.): apt

Description of the problem including expected versus actual behavior:
If an index pattern exists in the KB settings, but not in ES, you cannot delete that pattern.

Are we able to allow the user to go to the pattern definition, display a warning that the index doesn't exist, and at least offer the ability to delete said pattern at least?

Steps to reproduce:

1. Delete an index that has an existing index pattern in Kibana (via console/curl)
2. Try to remove that index pattern in Kibana Index Management page

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[danielmotaleite]

i also had this, had to recreate the index on ES so i could remove it :)

[ycombinator]

This might be related to https://github.com/elastic/kibana/issues/8373.

[burnstreet]

I am also having this issue, with 5.2

[branchnetconsulting]

I am also having this problem on 5.3

[chrisronline]

Hi @branchnetconsulting. I'm unable to reproduce the error in 5.3, but I'm also not really sure _what_ the error actually is. Can you provide some more information on steps to reproduce and also what the error actually looks like?

[danielmotaleite]

@chrisronline It is simple:

-Create a index on ES
-Create a index pattern in kibana for that index
-Remove the index in ES
-Try to remove the pattern index in kibana.

As the index does not exist anymore, you can not select it to remove the pattern, forcing the admin to create the index again, remove pattern and then remove the index for good.

[chrisronline]

Thanks for the added information @danielmotaleite.

I'm still having some issues so I just want to verify something while also getting a little more information.

What version of ES and Kibana are you seeing this?
Can you attach a screenshot of the management page when you are unable to select the index pattern?

Thanks

[branchnetconsulting]

I saw this issue with ElasticStack 5.3.0. In Kibana, go to
Management->Index Patterns. If any of the index patterns listed there have
no existing indexes in Elasticsearch, then the page will not respond all to
you left clicking on those patterns, making it impossible to highlight the
pattern you want to delete. The only solution I am aware of is to create a
matching index, at which point Kibana will let you delete the index
pattern, and then you can delete the dummy index.

Kevin

On Tue, May 16, 2017 at 1:35 PM, Chris Roberson [e-mail]
wrote:

Thanks for the added information @danielmotaleite
https://github.com/danielmotaleite.

I'm still having some issues so I just want to verify something while also
getting a little more information.

What version of ES and Kibana are you seeing this?
Can you attach a screenshot of the management page when you are unable to
select the index pattern?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/9224#issuecomment-301857134,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMj33vFN8VfUPWjV-LkMAWpENt3NNpumks5r6d5kgaJpZM4K8w61
.

[chrisronline]

Thanks for the clarification @branchnetconsulting

I'm attempting to follow those same steps in 5.3.0 but after I manually delete the index from ES (through a curl command) and refresh Kibana, I can still see, select, and delete the index pattern. It'd helpful if I can see maybe a video or at least some screenshots of the error scenario so it might help me track down the root cause. Unfortunately, I can't find any code within Kibana that would result in the index pattern showing up in the list, but not be clickable.

[branchnetconsulting]

Chris, there may be some lag time involved. I recall that after creating a
dummy index, it didn't actually let me delete the matching Kibana index
pattern until a few minutes later, and possibly after restarting Kibana
and/or Chrome.

Kevin

On Tue, May 16, 2017 at 2:52 PM, Chris Roberson [e-mail]
wrote:

Thanks for the clarification @branchnetconsulting
https://github.com/branchnetconsulting

I'm attempting to follow those same steps in 5.3.0 but after I manually
delete the index from ES (through a curl command) and refresh Kibana, I can
still see, select, and delete the index pattern. It'd helpful if I can see
maybe a video or at least some screenshots of the error scenario so it
might help me track down the root cause. Unfortunately, I can't find any
code within Kibana that would result in the index pattern showing up in the
list, but not be clickable.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/9224#issuecomment-301879367,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMj33nyU5l4CScJWXs9ylhUhlhJtq8v7ks5r6fBegaJpZM4K8w61
.

[chrisronline]

@branchnetconsulting Thanks Kevin.

Just so I fully understand, you're saying that you eventually could delete the Kibana index pattern, but it just took some time before Kibana let you do it?

[branchnetconsulting]

Yes, I used curl to create a dummy index that matched the index pattern
that previously had no matching indexes, and after a few minutes Kibana and
a restart of Kibana and/or Chrome, it let me delete the Kibana index
pattern.

On Tue, May 16, 2017 at 3:04 PM, Chris Roberson [e-mail]
wrote:

@branchnetconsulting https://github.com/branchnetconsulting Thanks
Kevin.

Just so I fully understand, you're saying that you eventually could delete
the Kibana index pattern, but it just took some time before Kibana let you
do it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/9224#issuecomment-301882671,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMj33r4xY4K0GuhlhyU2AUekJT33puTBks5r6fMpgaJpZM4K8w61
.

[chrisronline]

Thanks for the clarification @branchnetconsulting.

Does this match the behavior you're seeing too @danielmotaleite?

[markwalkom]

I just tried this on 5.4.0 and I couldn't replicate.

[danielmotaleite]

I'm attempting to follow those same steps in 5.3.0 but after I manually delete the index from ES
(through a curl command) and refresh Kibana, I can still see, select, and delete the index pattern

You probably need to restart kibana and/or maybe the browser to be able to be unable to select the pattern, if you do not do that, you are still using some cached info that allow you to select it.

Just so I fully understand, you're saying that you eventually could delete the Kibana index pattern, but it just took some time before Kibana let you do it?

Yes, I used curl to create a dummy index that matched the index pattern that previously had no matching indexes,

No! you both are talking about different steps, one after the index removal, the other after the index re-creation! :)

After creating the index and the kibana pattern, if you remove the index, you probably need to restart kibana, the browser or both. After that you are unable to select the removed index pattern in kibana. As you can not select it, you can not remove it.

Then, as workaround, you have to create the index (or something that match that index pattern) and restart kibana/browser or again both, you can then select the pattern in kibana and finally remove it

[chrisronline]

Here is a video I took trying to reproduce the problem on 5.3.0.

https://www.youtube.com/watch?v=d1M4CNYw-BQ&feature=youtu.be

My steps:

1. Verify ES index exists and matching Kibana index pattern.
2. Delete the ES index through a curl command, and verify it doesn't exist anymore.
3. Restart Kibana and open a new incognito Chrome window.
4. Load Kibana.
5. Verify index pattern matches nothing on Discover section (no results found).
6. Load all index patterns in the Management section.
7. Verify the index pattern shows up in the list.
8. Click and load the index pattern.
9. Successfully delete the index pattern.

I've also tried these same steps with another existing index pattern and I'm still able to properly select and delete the orphaned index pattern.

Is there another step involved that reproduces the error?

[danielmotaleite]

i'm on kibana 5.3.2 and creating the index, adding the pattern, removing the index, restart kibana and browser, i can still select the pattern!! :)

But i know that is isn't fixed, as i still have one index pattern that i can not delete as the related index does not exist anymore.

So there is something else missing to trigger this issue. Try to restart the elasticsearh cluster (i can't do that on mine, as it is live) , maybe there is some index cache on the ES that can change the test? or maybe the index or pattern must be created with a older (< v5.x) elasticsearch or kibana? Or maybe the pattern must be a wildcard (logstash-*), as all my patterns have wildcards

i will wait for the nighly rotation and maintenance to see if that changes anything in my cluster.

at least i have one broken pattern, so i can do some tests there. In the browser when i press the broken pattern i get this in the browser console:

Error: No matching indices found: [index_not_found_exception] no such index, with { resource.type="index_or_alias" & 
resource.id="logstash-0016*-filebeat-nginx-error-live" & index_uuid="_na_" & index="logstash-0016*-filebeat-nginx-error-live" }
KbnError@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:63:21936
errors.IndexPatternMissingIndices@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:63:25640
handleMissingIndexPattern@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:27:14338
processQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23621
scheduleProcessQueue/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23888
$eval@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4607
$digest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:2343
$apply@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:5026
done@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:25016
completeRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:28702
createHttpBackend/</xhr.onload@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:29634
EventHandlerNonNull*createHttpBackend/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:29409
sendReq@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:26505
serverRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:23333
processQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23621
scheduleProcessQueue/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23888
$eval@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4607
$digest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:2343
$evalAsync/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4751
completeOutstandingRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:4607
Browser/self.defer/timeoutId<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:7706
setTimeout handler*Browser/self.defer@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:7650
$evalAsync@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4706
$QProvider/this.$get</<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:22786
scheduleProcessQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23868
then@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:24896
$http@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:23803
AngularConnector.prototype.request@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:26920
CustomAngularConnector/this.request<@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:21735
wrapper@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:2:5945
sendReqWithConnection@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:1148
_.applyArgs@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:29624
bound@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:9:20721
Item.prototype.run@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25863
drainQueue@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25018
setTimeout handler*runTimeout@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:24003
process.nextTick@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25809
_.nextTick@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:30113
ConnectionPool.prototype.select@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:13649
Transport.prototype.request@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:4589
exec@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:24024
action@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:20756
Mapper/self.getFieldsForIndexPattern/<@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:27:13241
processQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23621
scheduleProcessQueue/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23888
$eval@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4607
$digest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:2343
$apply@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:5026
done@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:25016
completeRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:28702
createHttpBackend/</xhr.onload@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:29634
EventHandlerNonNull*createHttpBackend/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:29409
sendReq@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:26505
serverRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:23333
processQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23621
scheduleProcessQueue/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23888
$eval@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4607
$digest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:2343
$evalAsync/<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4751
completeOutstandingRequest@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:4607
Browser/self.defer/timeoutId<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:7706
setTimeout handler*Browser/self.defer@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:36:7650
$evalAsync@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:39:4706
$QProvider/this.$get</<@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:22786
scheduleProcessQueue@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:23868
then@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:38:24896
$http@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:37:23803
AngularConnector.prototype.request@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:26920
CustomAngularConnector/this.request<@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:21735
wrapper@https://kibana-internal.qwerty.com/bundles/commons.bundle.js?v=14849:2:5945
sendReqWithConnection@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:1148
_.applyArgs@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:29624
bound@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:9:20721
Item.prototype.run@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25863
drainQueue@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25018
setTimeout handler*runTimeout@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:24003
process.nextTick@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:25809
_.nextTick@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:1:30113
ConnectionPool.prototype.select@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:13649
Transport.prototype.request@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:4589
exec@https://kibana-internal.qwerty.com/bundles/kibana.bundle.js?v=14849:13:24024
  commons.bundle.js:38:11454

[chrisronline]

@danielmotaleite That stack trace is _VERY_ helpful! I'm seeing some area of the code that might be causing it. I'll update when I know more

[ycombinator]

But i know that is isn't fixed, as i still have one index pattern that i can not delete as the related index does not exist anymore.

Hey @danielmotaleite, for this index pattern that is un-deletable, would you mind making the following request against your Elasticsearch cluster and posting its results here?

GET .kibana/index-pattern/<your-index-pattern-name>?pretty

Thanks!

[danielmotaleite]

Sure:

$ curl -s  -XGET 192.168.xx.xx:9200/.kibana/index-pattern/logstash-0016*-filebeat-nginx-error-live?pretty
{
  "_index" : ".kibana",
  "_type" : "index-pattern",
  "_id" : "logstash-0016*-filebeat-nginx-error-live",
  "_version" : 3,
  "found" : true,
  "_source" : {
    "title" : "logstash-0016*-filebeat-nginx-error-live",
    "timeFieldName" : "@timestamp",
    "fields" : "[{\"name\":\"input_type.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"source\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"fields.venture\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"fields.venture.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"host\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"beat.hostname.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"geoip.longitude\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"tags.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"offset\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"input_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"fields.role\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"beat.name\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"geoip.latitude\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false},{\"name\":\"fields.environment.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false},{\"name\":\"error\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"fields.type.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"geoip.ip\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"error.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"fields.service\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"geoip.location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"@version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"severity\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"beat.name.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"host.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"fields.type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"type.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"fields.environment\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"fields.service.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"message\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"severity.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"beat.hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false},{\"name\":\"@timestamp\",\"type\":\"date\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"fields.role.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"source.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true},{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false}]"
  }
}

[chrisronline]

@danielmotaleite Thanks for providing that.

Quick question, do you recall when this index pattern was created? I have a hunch it might have been created before ES 5.0.

[branchnetconsulting]

All the index patterns that I could not delete were definitely created pre
ES 5.0.

Kevin

On Thu, May 18, 2017 at 9:42 AM, Chris Roberson [e-mail]
wrote:

@danielmotaleite https://github.com/danielmotaleite Thanks for
providing that.

Quick question, do you recall when this index pattern was created? I have
a hunch it might have been created before ES 5.0.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/9224#issuecomment-302407218,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMj33lSesJA-5chcZ6Klb2gRbUqwZG6Xks5r7ErZgaJpZM4K8w61
.

[danielmotaleite]

the same thing here, those patterns are old, more than one year old... and not even used for a long time

That was a debug pattern for a weird bug that i had and that i forgot until i upgraded to ELK 5.0

[chrisronline]

Okay thanks so much for all the information @danielmotaleite and @branchnetconsulting. Please keep an eye on the PR and I'll update this ticket once the fix is ready to use.

https://github.com/elastic/kibana/pull/11885

[chrisronline]

As a followup, I just wanted to explain what we think happened.

The issue is the affected index patterns do not contain searchable or aggregatable fields which can happen from a couple reasons, per @Bargs

1. Index pattern without searchable/aggregatable is created directly in ES via an external process. This usually happens when migrating from 2.x to 5.x. If the user doesn't migrate the data indices the pattern points to, field stats will fail and Kibana will never be able to fill in the searchable/aggregatable info.

1. This is a newer one. If you have a geo field and you upgrade from 2.x to 5.3 your field stats request will fail due to a bug in ES. Kibana will never be able to retrieve searchable/aggregatable info.

Basically anything that causes field_stats to fail after upgrading from 2.x to 5.x will put the index pattern in a bad state. Things should hopefully be better once we switch to the field_caps API.

Hopefully this can help identify potentially stale index patterns that need their fields refreshed in your cluster(s) but the fix will now allow you to properly delete orphaned index patterns.

[danielmotaleite]

Thanks for the update and the fix!

by the way, what happens when one press "refresh fields list" on one of those patterns? as there is no index, i assume it would simply fail and report one error and not open a new kind of bug :)

[chrisronline]

@danielmotaleite That's right. The behavior will mostly match when you're seeing now, but it won't break the page. It will just report the same error in the notification section at the top.

[soulesjms]

Might be a bit late to the conversation but this may help those who need to manually delete index patterns.
To get the doc id of index pattern:
curl -XGET ":9200/.kibana/index-pattern/"
then:
curl -XDELETE ":9200/.kibana/index-pattern/"
then restart kibana, and index-pattern should be gone.

[soulesjms]

curl -XGET "host:9200/.kibana/index-pattern/index-pattern"

[zazi]

for kibana/es 6.3.x (and above) it should be
curl -XGET ":9200/.kibana/doc/_search?q=index-pattern.title:[INDEX-PATTERN-NAME]"
to get the doc id of the index pattern ([INDEX-PATTERN-NAME] should be replaced with the real index pattern name)
then
curl -XDELETE ":9200/.kibana/doc/[INDEX-PATTERN-ID]/"
to delete index pattern ([INDEX-PATTERN-ID] should be replaced with the real index pattern id)

a restart of kibana is not really required (but a browser refresh maybe)