Kibana: Importing Saved Objects using JSON doesn't resolve filter index references

[bhavyarm]

Kibana version: 7.3.0 BC6/ 7.2.1 release

Elasticsearch version: 7.3.0 BC6/ 7.2.1 release

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: If user checks include related objects - then exporting saved objects fails in Kibana.

Please note - my saved objects are from 6.8.x

Steps to reproduce:

1. Export saved objects from 6.8.x and import them into 7.3.0 ( I used logstash data from load data into Kibana tutorial - https://www.elastic.co/guide/en/kibana/6.8/tutorial-load-dataset.html
2. Go to saved objects UI and click on export objects
3. Make sure include related objects option is checked
4. Exporting fails with "Unable to generate export"
5. If you uncheck that option - exporting works fine

Screenshots (if relevant):

Errors in browser console (if relevant):

commons.bundle.js:3 POST http://localhost:5601/api/saved_objects/_export 400 (Bad Request)
_callee6$ @ commons.bundle.js:3
tryCatch @ vendors.bundle.dll.js:541
invoke @ vendors.bundle.dll.js:541
prototype.<computed> @ vendors.bundle.dll.js:541
asyncGeneratorStep @ commons.bundle.js:3
_next @ commons.bundle.js:3
(anonymous) @ commons.bundle.js:3
(anonymous) @ commons.bundle.js:3
fetcher @ commons.bundle.js:3
Promise.then (async)
fetch @ commons.bundle.js:3
(anonymous) @ commons.bundle.js:3
Promise.then (async)
kfetch @ commons.bundle.js:3
kfetch @ commons.bundle.js:3
_callee$ @ kibana.bundle.js:3
tryCatch @ vendors.bundle.dll.js:541
invoke @ vendors.bundle.dll.js:541
prototype.<computed> @ vendors.bundle.dll.js:541
asyncGeneratorStep @ kibana.bundle.js:3
_next @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
fetchExportByType @ kibana.bundle.js:3
_callee6$ @ kibana.bundle.js:3
tryCatch @ vendors.bundle.dll.js:541
invoke @ vendors.bundle.dll.js:541
prototype.<computed> @ vendors.bundle.dll.js:541
asyncGeneratorStep @ kibana.bundle.js:3
_next @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
ca @ vendors.bundle.dll.js:224
ja @ vendors.bundle.dll.js:224
ka @ vendors.bundle.dll.js:224
wa @ vendors.bundle.dll.js:224
Aa @ vendors.bundle.dll.js:224
ya @ vendors.bundle.dll.js:224
Da @ vendors.bundle.dll.js:224
Ad @ vendors.bundle.dll.js:224
Gi @ vendors.bundle.dll.js:224
Kb @ vendors.bundle.dll.js:224
Dd @ vendors.bundle.dll.js:224
(anonymous) @ vendors.bundle.dll.js:224
exports.unstable_runWithPriority @ vendors.bundle.dll.js:175
Ii @ vendors.bundle.dll.js:224
Cd @ vendors.bundle.dll.js:224
Show 15 more frames
commons.bundle.js:3 Detected an unhandled Promise rejection.
Error: Bad Request
commons.bundle.js:3 Uncaught (in promise) Error: Bad Request
    at commons.bundle.js:3
KFetchError @ commons.bundle.js:3
(anonymous) @ commons.bundle.js:3
Promise.then (async)
asyncGeneratorStep @ kibana.bundle.js:3
_next @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
(anonymous) @ kibana.bundle.js:3
ca @ vendors.bundle.dll.js:224
ja @ vendors.bundle.dll.js:224
ka @ vendors.bundle.dll.js:224
wa @ vendors.bundle.dll.js:224
Aa @ vendors.bundle.dll.js:224
ya @ vendors.bundle.dll.js:224
Da @ vendors.bundle.dll.js:224
Ad @ vendors.bundle.dll.js:224
Gi @ vendors.bundle.dll.js:224
Kb @ vendors.bundle.dll.js:224
Dd @ vendors.bundle.dll.js:224
(anonymous) @ vendors.bundle.dll.js:224
exports.unstable_runWithPriority @ vendors.bundle.dll.js:175
Ii @ vendors.bundle.dll.js:224
Cd @ vendors.bundle.dll.js:224

[elasticmachine]

Pinging @elastic/kibana-platform

[bhavyarm]

cc @rudolf

[bhavyarm]

@marius-dr told me - this works for him but he doesn't have any saved objects from previous versions. So I am attaching the saved objects export file without the option - include selected objects checked. Please refer to bug https://github.com/elastic/kibana/issues/41240 for a possible related bug
export_without_include_selected_objects.ndjson.txt

[marius-dr]

If I import the objects from Bhavya's export file, then the bug happens for me too.

[Ron-70]

Seems similar to:
https://discuss.elastic.co/t/unable-to-generate-export-kibana-7-2-0/188692

[rudolf]

@bhavyarm Importing your 7.3.0 export file doesn't produce any errors for me and I can successfully export these objects again with the include related objects option. But I think the best way to reproduce this would be if you could share the export from 6.8.0. Then I can try to import that file into 7.3.0

@marius-dr @bhavyarm's example file ends in .ndjson.txt and will produce an error when importing. You have to rename the file so that it ends in .ndjson first.

[bhavyarm]

@rudolf use this please:
bubbles.json.txt

I have to rename the .json to txt otherwise github doesn't let me upload.

@rudolf this is from 6.8.2
saved_objects_681.json.txt

[bhavyarm]

@LeeDr export worked fine in 7.0.0. But it doesn't have include related objects part. Looking at 7.1.0 now

[rudolf]

I can confirm that importing saved_objects_681.json.txt into 7.3.0 and then exporting all Saved Objects causes the "Unable to generate export" error.

The reason for this is that the migrated saved objects reference index-patterns that doesn't exist (even though the user selects a new index pattern to map to).

[rudolf]

Looking at the code, the problem seems to be because we validate references before migrating the document to be imported. This means missing indexes are picked up for existing references, but if the document has a migration which adds new references then the import API never picks up that these references don't exist. We therefore let the import succeed, but actually the data that was imported isn't valid. Once we try to export these objects the export fails because it's unable to find the references.

Edit: seems like we "migrate" searchSourceJSON on the client side when importing JSON

[rudolf]

When importing from JSON files (deprecated) we use client-side logic to find missing indexes. Since this is a bug in a deprecated API I'm wondering if we want to fix it. The workaround for this bug would be to export all saved objects in 7.3 without including related objects and then re-importing this export. When re-importing, the missing filter indexes will be resolved and once complete, exports including related objects should be possible.

@mikecote I'd appreciate your thoughts?

[mikecote]

I see what @rudolf means. After importing the 6.8.x file, I can see most index patterns get resolved properly but index patterns from saved search filters get imported without resolved index patterns.

It seems it's always been designed like that and the new export feature exposes the bug when trying to include index patterns that do not exist.

I think there's 2 problems here:
1) Export fails when related objects are requested and missing
2) Legacy import doesn't resolve index patterns for filters (ex: kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index)

I think the first problem will always be around since users can explicitly delete saved objects within the management section. Once they do this, including dependencies will always fail. We could display a more user friendly message when that happens or skip the missing objects on export (maybe with a warning message). Then, if the user wishes to go forward importing that file with missing objects, it will replicate what they have in the environment they exported from.

[rudolf]

Warning about missing references but still trying to export as much as possible seems to me like the most desirable behaviour.

Since the API returns the export NDJSON we can't "inline" an error message. One way to work around that would be to have a ignoreMissingReferences option in the API. The UI could then try to perform an export and if that fails because of missing references, display a warning dialog and retry with ignoreMissingReferences = true.

[mikecote]

That makes sense to me. Would of been nice to only request an export once but it makes sense since we can't add any extra information when a file is returned.

[bhavyarm]

@rudolf we found the problem with few saved objects in metricbeats dashboard too. In the same release with 7.3.1 saved objects. cc @LeeDr

Reproduction steps:

1. import metricbeats -
   ./metricbeat setup -E "setup.dashboards.url=
   Index setup finished.
   Loading dashboards (Kibana must be running and reachable)
   Loaded dashboards

2. Try to export saved objects with "include related objects"

3. You get Unable to generate export error.

4. Dig in deep into 255 visualizations to figure out the problem - these are the culprit visualizations
   - Keyspace selector [Redis] ECS
   - Lists length [Redis] ECS
   - Keys by type [Redis] ECS
   - Average string key size [Redis] ECS
   - Average keys TTL [Redis] ECS

Please note saved searches are not causing the problem - if you select just saved searches - you can export them with include related objects option ON

1. if you look at the relationship in the UI - you can see 500 internal server error:
   

API request:

http://localhost:5601/api/kibana/management/saved_objects/relationships/visualization/517a5fd0-0514-11e9-9c60-d582a238e2c5-ecs?savedObjectTypes=config&savedObjectTypes=map&savedObjectTypes=canvas-workpad&savedObjectTypes=canvas-element&savedObjectTypes=index-pattern&savedObjectTypes=visualization&savedObjectTypes=search&savedObjectTypes=dashboard&savedObjectTypes=url

Kibana logs:

Debug: internal, implementation, error 
    TypeError: Cannot read property 'title' of undefined
    at getTitle (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/index.js:164:35)
    at SavedObjectsManagement.getTitle (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/core/server/saved_objects/management/management.js:66:16)
    at injectMetaAttributes (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/inject_meta_attributes.js:32:46)
    at concat.referencedObjects.saved_objects.map.obj (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/relationships.js:58:112)
    at Array.map (<anonymous>)
    at findRelationships (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/relationships.js:58:52)
    at process._tickCallback (internal/process/next_tick.js:68:7)
 error  [22:42:36.621]  TypeError: Cannot read property 'title' of undefined
    at getTitle (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/index.js:164:35)
    at SavedObjectsManagement.getTitle (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/core/server/saved_objects/management/management.js:66:16)
    at injectMetaAttributes (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/inject_meta_attributes.js:32:46)
    at concat.referencedObjects.saved_objects.map.obj (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/relationships.js:58:112)
    at Array.map (<anonymous>)
    at findRelationships (/Users/bhavyarajumandya/Desktop/BC3_7.3.1/kibana-7.3.1-darwin-x86_64/src/legacy/core_plugins/kibana/server/lib/management/saved_objects/relationships.js:58:52)
    at process._tickCallback (internal/process/next_tick.js:68:7)

export_beats.ndjson.txt

[rudolf]

Thanks @bhavyarm

Although this is related and results in the same export problem, the underlying problem lies within the metricbeat dashboards themselves. These dashboards contain references to indexes that don't exist. Because the metricbeat dashboards are imported using the API and not the import UI, we don't provide users with a chance to resolve missing references.

[rudolf]

@bhavyarm I've created a bug in the beats repo: https://github.com/elastic/beats/issues/13300