Kibana: Importing beats dashboards creates a duplicate index pattern if one already exists.

[bhavyarm]

Kibana version: 5.6.0 latest snapshot

Elasticsearch version: 5.6.0 latest snapshot

Server OS version: darwin_X86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): snapshot build

Description of the problem including expected versus actual behavior: I manually created metricbeat index pattern, checked the data on discover and then loaded beats dashboards. This created a duplicate metricbeat-* indexpattern in Kibana.

Steps to reproduce:

1. Start ES/Kibana/Metricbeat
2. Create metricbeat indexpattern manually
3. Load dashboards using ./script/import_dashboards - Kibana creates another metricbeat index pattern
   

[bhavyarm]

Trying to get this going in 5.4.0 to see whats the behaviour

[bhavyarm]

In 5.4.0: (you have to set ulimit -n 2048 to import dashboards) - I did this:

1. Started ES/Kibana/metricbeat
2. Created indexpattern - metricbeat manually
3. Changed the popularity of the field: system.process.name to 15 and changed the field format of system.process.cpu.start_time to string
4. Created 2 saved objects - 1 visualization and 1 dashboard
5. Imported dashboards using ./scripts/import_dashboards
6. Kibana didn't create a duplicate index - pattern. There is only one metricbeat index pattern in management
7. But it overrode the field settins for system.process.name and system.process.cpu.start_time to 0 and date
8. My saved objects - 1 viz and 1 dashboard are fine

[tylersmalley]

We could handle this on the dashboard import API, and when an index pattern doesn't exist we can see if something else with the same pattern exists. The one caveat is you can't search for it if it contains a dash. We might have to bring down all of them and search the array.

[ChrisWeissNike]

Not sure if it falls under this bug, but I'm seeing the same pattern duplication with 6.3 writing to /api/saved_objects/index-pattern/. If I run the POST multiple times, I'll get multiple copies of the same index.

Example CURL:

## Insert Saved Objects curl -X "POST" "http://localhost:5601/api/saved_objects/index-pattern/" \ -H 'Content-type: application/json' \ -H 'kbn-xsrf: true' \ -d $'{ "attributes": { "fields": "[{\\"name\\":\\"jenkins_timestamp\\",\\"type\\":\\"string\\",\\"count\\":0,\\"scripted\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"readFromDocValues\\":true}]", "timeFieldName": "@timestamp", "title": "jenkins-master-*", "fieldFormatMap": "{\\"@timestamp\\":{\\"id\\":\\"date\\"}}" } }'

Will create a new copy of jenkins-master-* each run. Maybe my content is malformed?

[epixa]

@elastic/kibana-platform

[BobBlank12]

Has there been a workaround identified for when this happens? How do we know which index pattern to delete? This is 7.4.0

[arkel-s]

I have an ugly workaround for that, that I used in a space with many duplicated dashboards:

1. To avoid issues, do a backup: On Kibana/Saved objects, export all you objects as Json
   
2. Look for the duplicated patterns ID, by looking a the URL when clicking on them, in Kibana/Index patterns
   
3. Note somewhere the correct index Index ID you want to keep, and the ones you want to delete:
   
4. Delete all duplicated index:
   
5. Again, export all your objects as Json, like step 1.
   This time, the duplicated index won't be part of the Json.
6. Do a search and replace in your Json file, to keep only one index
   
7. Import your modified Json
   

Worked for me on Kibana 7.6.2.
If you have a better solution, I'be happy to know :-)
Cheers