Kibana: [UPGRADE] can't save objects with K6.7 and Es7.0

[LeeDr]

Kibana version: 6.7.0-SNAPSHOT BUILD 19875 COMMIT db38e66d

Elasticsearch version: 7.0.0-SNAPSHOT build_date 2019-02-05T13:49:29.097757Z

Server OS version: CentOS 7

Browser version: Chrome

Browser OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): rpm default distribution

Describe the bug: While in a mixed version upgrade scenario (Elasticsearch is upgraded from 6.7 to 7.0 but Kibana is still at 6.7) I tried to add one of the sample data sets.
But I think I might have tried it before Kibana realized the Elasticsearch node was at 7.0.0 and turned the elasticsearch plugin status red (see https://github.com/elastic/kibana/issues/30287)

Steps to reproduce:

1. Have Kibana 6.7.0 on the Add Sample Data page
2. Upgrade Elasticsearch from 6.7.0 to 7.0.0
3. click Add on one of the sample test sets (I used kibana_sample_data_flights)
4. check elasticsearch.log for error like this;
   Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [doc : {dynamic=strict, properties={ml-telemetry={prop... (full log below)

Expected behavior: Data gets added

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[2019-02-06T18:19:19,478][INFO ][o.e.c.m.MetaDataCreateIndexService] [localhost.localdomain] [kibana_sample_data_flights] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[2019-02-06T18:19:24,364][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [localhost.localdomain] executed at 2019-02-06T18:19:24.331Z
[2019-02-06T18:19:25,639][DEBUG][o.e.a.a.i.t.p.TransportPutIndexTemplateAction] [localhost.localdomain] failed to put template [kibana_index_template:.kibana]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {dynamic=strict, properties={ml-telemetry={properties={file_data_visualizer={properties={index_creation_count={type=long}}}}}, server={properties={uuid={type=keyword}}}, visualization={properties={savedSearchId={type=keyword}, description={type=text}, uiStateJSON={type=text}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, visState={type=text}}}, apm-telemetry={properties={has_any_services={type=boolean}, services_per_agent={properties={python={null_value=0, type=long}, java={null_value=0, type=long}, go={null_value=0, type=long}, nodejs={null_value=0, type=long}, js-base={null_value=0, type=long}, ruby={null_value=0, type=long}}}}}, graph-workspace={properties={numVertices={type=integer}, description={type=text}, numLinks={type=integer}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, wsState={type=text}}}, kql-telemetry={properties={optInCount={type=long}, optOutCount={type=long}}}, type={type=keyword}, space={properties={color={type=keyword}, _reserved={type=boolean}, initials={type=keyword}, name={type=text, fields={keyword={ignore_above=2048, type=keyword}}}, description={type=text}}}, infrastructure-ui-source={properties={logAlias={type=keyword}, metricAlias={type=keyword}, name={type=text}, description={type=text}, fields={properties={container={type=keyword}, pod={type=keyword}, host={type=keyword}, tiebreaker={type=keyword}, timestamp={type=keyword}}}}}, url={properties={accessCount={type=long}, accessDate={type=date}, url={type=text, fields={keyword={ignore_above=2048, type=keyword}}}, createDate={type=date}}}, migrationVersion={dynamic=true, type=object}, upgrade-assistant-reindex-operation={dynamic=true, properties={indexName={type=keyword}, status={type=integer}}}, index-pattern={properties={notExpandable={type=boolean}, fieldFormatMap={type=text}, sourceFilters={type=text}, typeMeta={type=keyword}, timeFieldName={type=keyword}, intervalName={type=keyword}, fields={type=text}, title={type=text}, type={type=keyword}}}, search={properties={hits={type=integer}, columns={type=keyword}, description={type=text}, sort={type=keyword}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}}}, updated_at={type=date}, canvas-workpad={dynamic=false, properties={@created={type=date}, @timestamp={type=date}, name={type=text, fields={keyword={type=keyword}}}, id={index=false, type=text}}}, namespace={type=keyword}, telemetry={properties={enabled={type=boolean}}}, timelion-sheet={properties={hits={type=integer}, timelion_sheet={type=text}, timelion_interval={type=keyword}, timelion_columns={type=integer}, timelion_other_interval={type=keyword}, timelion_rows={type=integer}, description={type=text}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, timelion_chart_height={type=integer}}}, config={dynamic=true, properties={buildNum={type=keyword}}}, map={properties={bounds={type=geo_shape}, mapStateJSON={type=text}, description={type=text}, layerListJSON={type=text}, uiStateJSON={type=text}, title={type=text}, version={type=integer}}}, upgrade-assistant-telemetry={properties={ui_open={properties={overview={null_value=0, type=long}, cluster={null_value=0, type=long}, indices={null_value=0, type=long}}}, features={properties={deprecation_logging={properties={enabled={null_value=true, type=boolean}}}}}, ui_reindex={properties={stop={null_value=0, type=long}, start={null_value=0, type=long}, close={null_value=0, type=long}, open={null_value=0, type=long}}}}}, dashboard={properties={hits={type=integer}, timeFrom={type=keyword}, timeTo={type=keyword}, refreshInterval={properties={display={type=keyword}, section={type=integer}, value={type=integer}, pause={type=boolean}}}, description={type=text}, uiStateJSON={type=text}, timeRestore={type=boolean}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, optionsJSON={type=text}, panelsJSON={type=text}}}}}]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:393) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:322) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.validateAndAddTemplate(MetaDataIndexTemplateService.java:253) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.access$300(MetaDataIndexTemplateService.java:65) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService$2.execute(MetaDataIndexTemplateService.java:176) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Root mapping definition has unsupported parameters:  [doc : {dynamic=strict, properties={ml-telemetry={properties={file_data_visualizer={properties={index_creation_count={type=long}}}}}, server={properties={uuid={type=keyword}}}, visualization={properties={savedSearchId={type=keyword}, description={type=text}, uiStateJSON={type=text}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, visState={type=text}}}, apm-telemetry={properties={has_any_services={type=boolean}, services_per_agent={properties={python={null_value=0, type=long}, java={null_value=0, type=long}, go={null_value=0, type=long}, nodejs={null_value=0, type=long}, js-base={null_value=0, type=long}, ruby={null_value=0, type=long}}}}}, graph-workspace={properties={numVertices={type=integer}, description={type=text}, numLinks={type=integer}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, wsState={type=text}}}, kql-telemetry={properties={optInCount={type=long}, optOutCount={type=long}}}, type={type=keyword}, space={properties={color={type=keyword}, _reserved={type=boolean}, initials={type=keyword}, name={type=text, fields={keyword={ignore_above=2048, type=keyword}}}, description={type=text}}}, infrastructure-ui-source={properties={logAlias={type=keyword}, metricAlias={type=keyword}, name={type=text}, description={type=text}, fields={properties={container={type=keyword}, pod={type=keyword}, host={type=keyword}, tiebreaker={type=keyword}, timestamp={type=keyword}}}}}, url={properties={accessCount={type=long}, accessDate={type=date}, url={type=text, fields={keyword={ignore_above=2048, type=keyword}}}, createDate={type=date}}}, migrationVersion={dynamic=true, type=object}, upgrade-assistant-reindex-operation={dynamic=true, properties={indexName={type=keyword}, status={type=integer}}}, index-pattern={properties={notExpandable={type=boolean}, fieldFormatMap={type=text}, sourceFilters={type=text}, typeMeta={type=keyword}, timeFieldName={type=keyword}, intervalName={type=keyword}, fields={type=text}, title={type=text}, type={type=keyword}}}, search={properties={hits={type=integer}, columns={type=keyword}, description={type=text}, sort={type=keyword}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}}}, updated_at={type=date}, canvas-workpad={dynamic=false, properties={@created={type=date}, @timestamp={type=date}, name={type=text, fields={keyword={type=keyword}}}, id={index=false, type=text}}}, namespace={type=keyword}, telemetry={properties={enabled={type=boolean}}}, timelion-sheet={properties={hits={type=integer}, timelion_sheet={type=text}, timelion_interval={type=keyword}, timelion_columns={type=integer}, timelion_other_interval={type=keyword}, timelion_rows={type=integer}, description={type=text}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, timelion_chart_height={type=integer}}}, config={dynamic=true, properties={buildNum={type=keyword}}}, map={properties={bounds={type=geo_shape}, mapStateJSON={type=text}, description={type=text}, layerListJSON={type=text}, uiStateJSON={type=text}, title={type=text}, version={type=integer}}}, upgrade-assistant-telemetry={properties={ui_open={properties={overview={null_value=0, type=long}, cluster={null_value=0, type=long}, indices={null_value=0, type=long}}}, features={properties={deprecation_logging={properties={enabled={null_value=true, type=boolean}}}}}, ui_reindex={properties={stop={null_value=0, type=long}, start={null_value=0, type=long}, close={null_value=0, type=long}, open={null_value=0, type=long}}}}}, dashboard={properties={hits={type=integer}, timeFrom={type=keyword}, timeTo={type=keyword}, refreshInterval={properties={display={type=keyword}, section={type=integer}, value={type=integer}, pause={type=boolean}}}, description={type=text}, uiStateJSON={type=text}, timeRestore={type=boolean}, title={type=text}, version={type=integer}, kibanaSavedObjectMeta={properties={searchSourceJSON={type=text}}}, optionsJSON={type=text}, panelsJSON={type=text}}}}}]
        at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:152) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:140) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:92) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:391) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        ... 17 more

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-operations

[elasticmachine]

Pinging @elastic/kibana-app

[tylersmalley]

Tagging the app team who own this - it sounds like the _doc changes need to be backported to 6.7.

[LeeDr]

This looks like a pretty widespread issue thoughout Kibana. I can't make any setting changes I've tried so far. I can't select yes or no on the Telemetry banner.

Telemetry Error
Unable to save telemetry preference.
Check that Kibana and Elasticsearch are still running, then try again.

I can't create an index pattern for .kibana. I get Service Unavailable in the browser console.
I can select Include system indices
I enter .kibana and see there is the index
I can click next and I do see any date fields and can select I don't want to use a timefield
but when I click finish the page just spins with Creating index pattern...

[tylersmalley]

I believe this needs to be backported: https://github.com/elastic/kibana/pull/29184

Looking into it.

[chrisdavies]

Thanks, Tyler. Let me know if you need a hand, since that was my PR, and my fault it didn't get backported.

[LeeDr]

FYI, on snapshots from today (Kibana on 6.7.0 and Elasticsearch on 7.0.0);

Also, I can't even save a search;

Same error in the log as above;
failed to put template [kibana_index_template:.kibana]

[chrisdavies]

FYI, @LeeDr we merged https://github.com/elastic/kibana/pull/30636

So, that should unblock you. We still have a bit more to do before CI is green for the 6.7 to 7.0 scenario: https://github.com/elastic/kibana/pull/30647

But we're waiting on some infrastructure outages to clear up before we know the status of that.

[tylersmalley]

One more thing has been discovered, which requires https://github.com/elastic/kibana/pull/30036, which is a backport of the dependency on doc versions. Versions are currently being silently ignored which could create contention as we utilize it for optimistic locking and prevention of overrides.

[spalger]

This should be fixed now that https://github.com/elastic/kibana/pull/30036 is merged, can you verify @LeeDr?

[danielmitterdorfer]

@LeeDr as this is a release blocker, can we please prioritize this accordingly?

[tylersmalley]

@LeeDr this should be resolved in BC3.

[tylersmalley]

Lee has confirmed this issue is now resolved.

[LeeDr]

Confirmed using .deb packages on Ubuntu 14. Upgraded from 5.6.15 to 6.7.0 BC3 and then to 7.0.0.

I was able to save a new saved search and add and view a set of sample data.