Kibana: Kibana needs to handle "Fielddata access on the _id field is disallowed"

Created on 9 Dec 2019  路  9Comments  路  Source: elastic/kibana

Kibana version: 8.0.0

Elasticsearch version: 8.0.0

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from snapshot build on 8th dec

Describe the bug: Elasticsearch has a breaking change https://github.com/elastic/elasticsearch/pull/49166 and Kibana needs to handle it.

Steps to reproduce:

  1. Download latest snapshot for 8.0.0 ES/Kibana
  2. Start ES/Kibana
  3. Logs about the breaking change appear all over both ES/Kibana logs

Kibana logs:

 log   [11:11:26.794] [warning][plugins][usageCollection] { Error: [illegal_argument_exception] Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
    at respond (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/bhavyarajumandya/Desktop/test_master/kibana-8.0.0-SNAPSHOT-darwin-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message:
   '[illegal_argument_exception] Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled',
  path: '/.kibana_task_manager/_search',
  query: { ignore_unavailable: true },
  body:
   { error:
      { root_cause: [Array],
        type: 'search_phase_execution_exception',
        reason: 'all shards failed',
        phase: 'query',
        grouped: true,
        failed_shards: [Array],
        caused_by: [Object] },
     status: 400 },
  statusCode: 400,
  response:
   '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".kibana_task_manager_1","node":"PJ3jQRajSkW-IzEAP3-5LA","reason":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"}}},"status":400}',
  toString: [Function],
  toJSON: [Function] }

ES logs:

[2019-12-09T11:15:07,181][DEBUG][o.e.a.s.TransportSearchAction] [bhavyarajumandya] [.kibana_task_manager_1][0], node[PJ3jQRajSkW-IzEAP3-5LA], [P], s[STARTED], a[id=7l-bdd6cQ2uDl-bv06SzdQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.kibana_task_manager], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"bool":{"must":[{"term":{"type":{"value":"task","boost":1.0}}},{"bool":{"filter":[{"term":{"_id":{"value":"task:oss_telemetry-vis_telemetry","boost":1.0}}}],"boost":1.0}}],"boost":1.0}},"sort":[{"task.runAt":{"order":"asc"}},{"_id":{"order":"desc"}}]}}]
org.elasticsearch.transport.RemoteTransportException: [bhavyarajumandya][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
    at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:197) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]
[2019-12-09T11:15:07,182][DEBUG][o.e.a.s.TransportSearchAction] [bhavyarajumandya] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
    at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:304) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:242) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:423) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1059) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1168) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1142) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.lang.IllegalArgumentException: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled
    at org.elasticsearch.index.mapper.IdFieldMapper$IdFieldType$1.build(IdFieldMapper.java:172) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:134) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:197) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:358) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    ... 9 more

Any additional context:
https://github.com/elastic/elasticsearch/pull/49166

Task Manager Alerting Services bug
Source
picture bhavyarm
❤1

All 9 comments

Pinging @elastic/kibana-stack-services (Team:Stack Services)

elasticmachine picture elasticmachine on 9 Dec 2019

Please note ES change is a breaking change for 7.6.0

Edit - breaking change is for 8.0.0 - cc @LeeDr @timroes confirmed it in my 7.6.0 logs.

bhavyarm picture bhavyarm on 9 Dec 2019
👍1

This is related to https://github.com/elastic/elasticsearch/pull/49166 and https://github.com/elastic/elasticsearch/issues/43599

tylersmalley picture tylersmalley on 9 Jan 2020

We're currently pinned to an older version of Elasticsearch 8.0.0 both locally and on CI. To unpin this, we will need to remove our usage of sorting on _id which is done here. Is sorting on _id required, or would this be something easy to remove?

tylersmalley picture tylersmalley on 9 Jan 2020

There's also potentially this:

https://github.com/elastic/kibana/blob/dfac5d894ec4a2076ab7e0475a6151a7ca021a42/x-pack/legacy/plugins/task_manager/server/queries/mark_available_tasks_as_claimed.ts#L98-L121

pmuellr picture pmuellr on 9 Jan 2020

Quick peek through the code, I think it would be save to change the function doing this to remove the sortById bits. Clearly the thinking was we'd get exact pagination results, and there doesn't seem to be any other field that could be used instead. OTOH, I don't think where this gets used, it's going to matter, right now. Following the references, it's up to a fetch() API on the task_manager service, and I doubt anyone making any calls that depend on stable pagination.

https://github.com/elastic/kibana/blob/dfac5d894ec4a2076ab7e0475a6151a7ca021a42/x-pack/legacy/plugins/task_manager/server/task_store.ts#L430-L442

pmuellr picture pmuellr on 9 Jan 2020

Running FTS with KBN_ES_SNAPSHOT_USE_UNVERIFIED=1 (which uses the most recent snapshot), shows an error in the mark_available_tasks_as_claimed.ts file, noted above. The one in task_store.ts is also problematic, needs to be fixed.

pmuellr picture pmuellr on 9 Jan 2020

The only usage I can see is the telemetry usage collector, which filters for a specific _id value when it does a fetch (so it doesn't use pagination). Seems safe to remove pagination support, we can re-introduce later if/when needed.

peterschretlen picture peterschretlen on 9 Jan 2020

I'm running FTS/FTR from master (with Gidi's fix merged), setting the following when running FTS so it uses the latest unverified ES snapshot, which is what broke with the old code (when sorting with _id):

# FTS
KBN_ES_SNAPSHOT_USE_UNVERIFIED=1 cd-x-pack.sh node scripts/functional_tests_server.js \
>     --config=test/alerting_api_integration/security_and_spaces/config.ts```

# FTR
cd-x-pack.sh node ../scripts/functional_test_runner.js --grep "Alerting" \
    --config=test/alerting_api_integration/security_and_spaces/config.ts

passed!

pmuellr picture pmuellr on 16 Jan 2020
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

timmolter picture timmolter  路  3Comments
mark54g picture mark54g  路  3Comments
socialmineruser1 picture socialmineruser1  路  3Comments
timroes picture timroes  路  3Comments
stacey-gammon picture stacey-gammon  路  3Comments