Kibana: Getting "all shards failed: [unsupported_operation_exception] null" after upgrade to 5.6.0
UPDATE: We released a script that will automate the workaround mentioned in this issue. You can find instructions here: https://discuss.elastic.co/t/upgrade-issue-with-elastic-stack-5-6-0-workaround-option-until-fix-is-available/100595
Kibana version: 5.6.0
Elasticsearch version: 5.6.0
Steps to reproduce:
- Use
.kibanaindex that was created prior to5.x - Upgrade to
5.6.0 - Try to log in
Provide logs and/or server output (if relevant):
In kibana
Note: While it says Elasticsearch plugin was red, the cluster itself was green at this point.
In Elasticsearch:
[2017-09-12T12:19:47,549][DEBUG][o.e.a.s.TransportSearchAction] [IQdK6vv] [.kibana-2.4.1][0], node[IQdK6vvyR4mAJbSEysf7JA], [P], s[STARTED], a[id=D4ser0mIT4KXg30kb1w7rw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128, source={
"from" : 0,
"size" : 1000,
"query" : {
"bool" : {
"must" : [
{
"match_all" : {
"boost" : 1.0
}
}
],
"filter" : [
{
"bool" : {
"should" : [
{
"term" : {
"_type" : {
"value" : "config",
"boost" : 1.0
}
}
},
{
"term" : {
"type" : {
"value" : "config",
"boost" : 1.0
}
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"version" : true,
"sort" : [
{
"buildNum" : {
"order" : "desc",
"unmapped_type" : "keyword"
}
},
{
"config.buildNum" : {
"order" : "desc",
"unmapped_type" : "keyword"
}
}
]
}}]
org.elasticsearch.transport.RemoteTransportException: [IQdK6vv][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209) ~[?:1.8.0_131]
at org.elasticsearch.index.mapper.KeywordFieldMapper$TypeParser.parse(KeywordFieldMapper.java:152) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.unmappedFieldType(MapperService.java:727) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:260) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:156) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:634) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.0.jar:5.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-09-12T12:19:47,554][DEBUG][o.e.a.s.TransportSearchAction] [IQdK6vv] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: null
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:90) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:46) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase$1.onFailure(InitialSearchPhase.java:169) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1171) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:655) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.0.jar:5.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209) ~[?:1.8.0_131]
at org.elasticsearch.index.mapper.KeywordFieldMapper$TypeParser.parse(KeywordFieldMapper.java:152) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.unmappedFieldType(MapperService.java:727) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:260) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:156) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:634) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.0.jar:5.6.0]
... 3 more
[2017-09-12T12:19:47,559][WARN ][r.suppressed ] path: /.kibana/_search, params: {size=1000, index=.kibana, from=0}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:130) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:90) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:46) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.InitialSearchPhase$1.onFailure(InitialSearchPhase.java:169) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1171) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:655) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.0.jar:5.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: org.elasticsearch.ElasticsearchException$1
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:170) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:111) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:139) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.impl.MutableLogEvent.getThrownProxy(MutableLogEvent.java:336) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:333) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:232) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:217) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:57) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) [log4j-core-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217) [log4j-api-2.9.0.jar:2.9.0]
at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:102) [elasticsearch-5.6.0.jar:5.6.0]
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2116) [log4j-api-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2100) [log4j-api-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1989) [log4j-api-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1851) [log4j-api-2.9.0.jar:2.9.0]
at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2589) [log4j-api-2.9.0.jar:2.9.0]
at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:133) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:220) [elasticsearch-5.6.0.jar:5.6.0]
... 16 more
Caused by: java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209) ~[?:1.8.0_131]
at org.elasticsearch.index.mapper.KeywordFieldMapper$TypeParser.parse(KeywordFieldMapper.java:152) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.unmappedFieldType(MapperService.java:727) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:260) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:156) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:634) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.0.jar:5.6.0]
... 3 more
Running the query manually results in the same stack trace and this response:
{
"error": {
"root_cause": [
{
"type": "unsupported_operation_exception",
"reason": null
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": ".kibana-2.4.1",
"node": "IQdK6vvyR4mAJbSEysf7JA",
"reason": {
"type": "unsupported_operation_exception",
"reason": null
}
}
]
},
"status": 500
}
The problem seems to be that this index was created on Elasticsearch 2.x (I actually already using this since ES 1.x and reindexed once in 2.4.1 before upgrading to 5.0) and it's still using the string type, rather than the keyword / text. However, the same index worked fine on all 5.x version until I upgraded to 5.6.0 today.
The mapping looks like this:
{
".kibana-2.4.1": {
"mappings": {
"search": {
"properties": {
"columns": {
"type": "string"
},
"description": {
"type": "string"
},
"hits": {
"type": "integer"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "string"
}
}
},
"sort": {
"type": "string"
},
"title": {
"type": "string"
},
"version": {
"type": "integer"
}
}
},
"graph-workspace": {
"properties": {
"description": {
"type": "string"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "string"
}
}
},
"numLinks": {
"type": "integer"
},
"numVertices": {
"type": "integer"
},
"title": {
"type": "string"
},
"version": {
"type": "integer"
},
"wsState": {
"type": "string"
}
}
},
"visualization": {
"properties": {
"description": {
"type": "string"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "string"
}
}
},
"savedSearchId": {
"type": "string"
},
"title": {
"type": "string"
},
"uiStateJSON": {
"type": "string"
},
"version": {
"type": "integer"
},
"visState": {
"type": "string"
}
}
},
"timelion-sheet": {
"properties": {
"description": {
"type": "string"
},
"hits": {
"type": "integer"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "string"
}
}
},
"timelion_chart_height": {
"type": "integer"
},
"timelion_columns": {
"type": "integer"
},
"timelion_interval": {
"type": "string"
},
"timelion_other_interval": {
"type": "string"
},
"timelion_rows": {
"type": "integer"
},
"timelion_sheet": {
"type": "string"
},
"title": {
"type": "string"
},
"version": {
"type": "integer"
}
}
},
"dashboard": {
"properties": {
"description": {
"type": "string"
},
"hits": {
"type": "integer"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "string"
}
}
},
"optionsJSON": {
"type": "string"
},
"panelsJSON": {
"type": "string"
},
"refreshInterval": {
"properties": {
"display": {
"type": "string",
"fields": {
"keyword": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
},
"fielddata": false
},
"pause": {
"type": "boolean"
},
"section": {
"type": "long"
},
"value": {
"type": "long"
}
}
},
"timeFrom": {
"type": "string"
},
"timeRestore": {
"type": "boolean"
},
"timeTo": {
"type": "string"
},
"title": {
"type": "string"
},
"uiStateJSON": {
"type": "string"
},
"version": {
"type": "integer"
}
}
},
"config": {
"properties": {
"buildNum": {
"type": "long"
},
"defaultIndex": {
"type": "string"
},
"discover:aggs:terms:size": {
"type": "long"
},
"discover:sampleSize": {
"type": "string"
},
"visualization:colorMapping": {
"type": "string"
},
"xPackMonitoring:allowReport": {
"type": "boolean"
},
"xPackMonitoring:showBanner": {
"type": "boolean"
}
}
},
"index-pattern": {
"properties": {
"customFormats": {
"type": "string"
},
"fieldFormatMap": {
"type": "string"
},
"fields": {
"type": "string"
},
"intervalName": {
"type": "string"
},
"notExpandable": {
"type": "boolean"
},
"timeFieldName": {
"type": "string"
},
"title": {
"type": "string"
}
}
},
"_default_": {
"dynamic": "strict"
},
"url": {
"dynamic": "strict",
"properties": {
"accessCount": {
"type": "long"
},
"accessDate": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"createDate": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"url": {
"type": "string",
"fields": {
"keyword": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 2048
}
},
"fielddata": false
}
}
},
"server": {
"dynamic": "strict",
"properties": {
"uuid": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
Not yet sure which part of the mapping / query is responsible for failing. I suspect it to be related to a string field that should be keyword now.
I feel like a lot of users will hit this, as many still have an .kibana index that was created on ES 2.x.
Workaround
As a workaround I created a new empty kibana index and extracted the mapping, then I created a new kibana index with that mapping and re indexed my old kibana index into the new one.
Create new kibana index with 5.x compatible mapping
curl -XPUT -s -uelastic:changeme 'https://localhost:9200/.kibana-5.6.0' -d'
{
"mappings": {
"_default_": {
"dynamic": "strict"
},
"config": {
"properties": {
"buildNum": {
"type": "keyword"
}
},
"dynamic": "true"
},
"timelion-sheet": {
"properties": {
"version": {
"type": "integer"
},
"title": {
"type": "text"
},
"timelion_sheet": {
"type": "text"
},
"description": {
"type": "text"
},
"hits": {
"type": "integer"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "text"
}
}
},
"timelion_chart_height": {
"type": "integer"
},
"timelion_columns": {
"type": "integer"
},
"timelion_interval": {
"type": "keyword"
},
"timelion_other_interval": {
"type": "keyword"
},
"timelion_rows": {
"type": "integer"
}
},
"dynamic": "strict"
},
"visualization": {
"properties": {
"visState": {
"type": "text"
},
"version": {
"type": "integer"
},
"uiStateJSON": {
"type": "text"
},
"title": {
"type": "text"
},
"savedSearchId": {
"type": "keyword"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "text"
}
}
},
"description": {
"type": "text"
}
},
"dynamic": "strict"
},
"index-pattern": {
"properties": {
"title": {
"type": "text"
},
"timeFieldName": {
"type": "keyword"
},
"sourceFilters": {
"type": "text"
},
"notExpandable": {
"type": "boolean"
},
"intervalName": {
"type": "keyword"
},
"fields": {
"type": "text"
},
"fieldFormatMap": {
"type": "text"
}
},
"dynamic": "strict"
},
"dashboard": {
"properties": {
"version": {
"type": "integer"
},
"uiStateJSON": {
"type": "text"
},
"title": {
"type": "text"
},
"timeTo": {
"type": "keyword"
},
"description": {
"type": "text"
},
"hits": {
"type": "integer"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "text"
}
}
},
"optionsJSON": {
"type": "text"
},
"panelsJSON": {
"type": "text"
},
"refreshInterval": {
"properties": {
"value": {
"type": "integer"
},
"section": {
"type": "integer"
},
"pause": {
"type": "boolean"
},
"display": {
"type": "keyword"
}
}
},
"timeFrom": {
"type": "keyword"
},
"timeRestore": {
"type": "boolean"
}
},
"dynamic": "strict"
},
"url": {
"properties": {
"url": {
"fields": {
"keyword": {
"ignore_above": 2048,
"type": "keyword"
}
},
"type": "text"
},
"createDate": {
"type": "date"
},
"accessDate": {
"type": "date"
},
"accessCount": {
"type": "long"
}
},
"dynamic": "strict"
},
"server": {
"properties": {
"uuid": {
"type": "keyword"
}
},
"dynamic": "strict"
},
"search": {
"properties": {
"version": {
"type": "integer"
},
"title": {
"type": "text"
},
"sort": {
"type": "keyword"
},
"kibanaSavedObjectMeta": {
"properties": {
"searchSourceJSON": {
"type": "text"
}
}
},
"hits": {
"type": "integer"
},
"description": {
"type": "text"
},
"columns": {
"type": "keyword"
}
},
"dynamic": "strict"
}
}
}'
Reindex from old kibana index into the newly created one
curl -XPOST -s -uelastic:changeme 'https://localhost:9200/_reindex' -d'
{
"source": {
"index": ".kibana"
},
"dest": {
"index": ".kibana-5.6.0"
}
}'
edit kibana.yml to use the new index
Just change kibana.index: ".kibana to kibana.index: ".kibana-5.6.0" and restart kibana.
Note: Rather than re configuring kibana.index: one could also DELETE the old .kibana index and create an alias with that name pointing to the new index name.
Ping me if you need the broken kibana index for testing.
bugpool
jakommo
👍6
All 10 comments
Running into the exact same problem. Thanks for the workaround @jakommo 馃憤
rodgermoore
on 12 Sep 2017
You are welcome @rodgermoore . It's actually an Elasticsearch issue causing this: https://github.com/elastic/elasticsearch/issues/26162
I leave this one open for now, so users hitting the issue can find the workaround.
jakommo
on 12 Sep 2017
Got the same issue. Thanks for the workaround @jakommo !
sagolubev
on 14 Sep 2017
We have released a script that will automate the workaround above:
https://discuss.elastic.co/t/upgrade-issue-with-elastic-stack-5-6-0-workaround-option-until-fix-is-available/100595
jakommo
on 15 Sep 2017
👍1
script is working for me without problems, thanks
derEremit
on 15 Sep 2017
script is not working for me in the beginning...only getting "Before" and it quits
-> have to edit every single Curl with "--noproxy '*'"
Now it worked, thanks
ronronron1
on 18 Sep 2017
@ronronron1 can you create a post in https://discuss.elastic.co/c/kibana and let us know which command you used, if you have x-pack installed and what the full output of the script is?
jakommo
on 18 Sep 2017
FYI: 5.6.1 is out, where this should be fixed.
tomsommer
on 19 Sep 2017
Is there a patch for 6.0.0 yet? I am seeing this issue on a bare-metal kubeadm 5-node setup and had it working for a hot second after applying the patch, but then failed.
mtaylor769
on 2 Dec 2017
@mtaylor769 this is fixed in 5.6.1. If you are seeing a similar error it's something different.
Can you post in https://discuss.elastic.co/c/kibana with a few more details on how your error looks like?
jakommo
on 4 Dec 2017
👍3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Vineeth-Mohan
路
149Comments
ctindel
路
81Comments
cff3
路
83Comments
panda87
路
206Comments
bquartier
路
79Comments
Most helpful comment
@mtaylor769 this is fixed in 5.6.1. If you are seeing a similar error it's something different.
Can you post in https://discuss.elastic.co/c/kibana with a few more details on how your error looks like?