Kibana: Change index on visualization/dashboard

Created on 23 Apr 2015  ·  170Comments  ·  Source: elastic/kibana

Update on April 4th 2018
We really appreciate all of your feedback on this mega issue so far, but we need your help as we break this down into more concrete items to work on. Even if you've already +1'd this specific issue, it would be extremely helpful if you could +1 and describe your use case in the more granular issues that are provided below. We won't ignore all the feedback folks have provided so far in this issue, but more focused feedback on the individual feature proposals would be invaluable.

Here are the three sub issues that I believe make up this mega issue:

Changing index patterns on a visualization: #17542
Dashboard level index patterns: #16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: #16831
Nested dashboards: https://github.com/elastic/kibana/issues/16919

Dashboard level index patterns seems to match this initial request best and there is a currently available workaround for it, which is mentioned in https://github.com/elastic/kibana/issues/16917.

If those three issues do not cover your particular case, please let us know!


Original Issue Request:

I'm in a situation where I have the exact same dashboard and it's visualizations, but want it to use different aliases/indexes.

The idea would be to segregate different user's data/views, but each have the same view/dashboard (just his/her index/alias differs), especially as I read Shield does authorization/access control via aliases/indexes and that is what I need/want to use for the user authorization/access control then.

Dashboard Visualizations enhancement

Most helpful comment

Is it possible with the current architecture? The index appears bound to the visualization, not the dashboard in K4. This would be hugely useful for me. Like most people (I imagine) I have each of my environments split into separate indexes; maintaining the dashboards becomes difficult...

In Kibana 3 I have hacked in a select list box which allows users to switch indexes for each dashboard, meaning only 1 dashboard needs to be maintained for all environments. Kibana 4 however binds the index to the visualisation not the dashboard. It might be helpful if the dashbaord could (optionally) override the index on its visualisations? I can see how it is useful to have a dashboard that is index agnostic in some use cases however.

All 170 comments

You can do this by copying the visualization documents in elasticsearch and changing the parameter for index. I don't see us supporting a UI on for this sort of functionality any time soon.

:( It then becomes an quickly intractable O(M*N) issue (N=the number of indexes, and M the number of visualizations.

Is it possible with the current architecture? The index appears bound to the visualization, not the dashboard in K4. This would be hugely useful for me. Like most people (I imagine) I have each of my environments split into separate indexes; maintaining the dashboards becomes difficult...

In Kibana 3 I have hacked in a select list box which allows users to switch indexes for each dashboard, meaning only 1 dashboard needs to be maintained for all environments. Kibana 4 however binds the index to the visualisation not the dashboard. It might be helpful if the dashbaord could (optionally) override the index on its visualisations? I can see how it is useful to have a dashboard that is index agnostic in some use cases however.

+1

Copying the visualization documents in elasticsearch is not really an option. That generates a lot of duplicated information and updating would be a nightmare.

+1 on this. The "index pattern" concept naturally maps onto a "source" of data and the dashboard currently make it impossible to do this.

At the very least, the search bar at the top of the dashboard page should allow filtering by a "more specific" index pattern but it doesn't seem to accept searches with _index:­...-* patterns (nor specific values for that matter).

Our current workaround is to have a separate document type for each source because it's the only way to quickly filter by source while staying the same dashboard, but this creates Yet Another Configuration Problem (tm) because we need to create new type mappings for each environment...

We face a situation where we will have users from different departments doing different roles accessing the same data via their own dashboards.
e.g. developers vs front line support staff.

How can we restrict the dashboard modifications - so that the set of dashboards created by developers for eg. cannot be modifed by support staff?

If shield is tied solely to the index, it wouldnt work as the underlying data for both dashboards is the same.

Thank you.

+1 for this.
At this moment I am using Ansible and templates to at least automate a bit of the copy and substite process but this should not be _THE_ way to do it. Redundancy is not cool from a maintenance perspective...

+1

+1 for this.
It would be very usefull to have this feature when you have one index patter for every environment (dev, homolog, prod) but dashboards and visualizations are the same.

+1
Having index tied to visualization, embedded in dashboards, is a complete nightmare. It makes impossible to use aliases for recent data and long term logstash data, without rewriting completely the dashboards.

Please have a look at how dashboard interface and design works in Grafana 2, that totally nailed it form my point of view. For example, graphs are not separate entity but created inside a dashboard.

+1 Right now it is a complete nightmare when you have complex platforms.

+1 Please add this support fast...

+1

+1 It's a huge nightmare for us to have to create and then upkeep 3 different dashboard across 3 different environment. The dashboards should be the same for us in each environment but without this feature we have to manually create visualizations and dashboards 3 times over.

+1

+1

+1

+1
Duplicate visualizations is not the way.
Why did you change Kibana 3 way to handle dashboards!?!?

+1

+1

+1

+1

+1

+1 this would be really important for us

+1

In Bitergia we have created a tool to create automatic dashboards from a template dashboard. One of the main tasks is modifying the index.

Here is the open source tool in case it is useful to anybody:

https://github.com/acs/GrimoireELK/blob/master/utils/e2k.py

+1

+1

@rashidkpc : I tried your suggestion, and I'm having difficulty with creating the proper entries. Then I get a Kibana warning:

"Proceed with caution

Modifying objects is for advanced users only. Object properties are not validated and invalid objects could cause errors, data loss, or worse. Unless someone with intimate knowledge of the code told you to be in here, you probably shouldn't be."

Can you provide an example of copying such a dashboard? It looks like users could very well mess up their kibana instance mucking with this.

+1

+1

+5

+1

+1

+1

+1

+1

Hi @rashidkpc
Still feel the same about this issue a year later with the responses given above?

Perhaps an "official" response as to how to handle the environments mentioned above?

+1

+1

+1

+1

+1 please.

+1

+1

+1

+1

+1

+1

+1

+1

+1

👍

+1

+1

+1

+1

+1

+1

+1

OMG. This would be so useful!!!

+1 - Kibana really needs to support reusable "assets."

+1

+1

+1111!!!1one

+1

+1

+1 this would be great!

+1 for this.

+1

+1

+1

+1

+1

+1

+1 ! Could be very useful

+1

+1

+1

+1

+1
I thought this was already possible :(

+1

This is def. a must have. Please make it happen.

+1

+1

+1

Will a requested feature exist after 2 years? Who knows! +1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+100000000000

+1

+1

+1

+1

+1

+1

+1

+1

So does anyone over at Elastic want to actually implement this ?

+1

+1

+1

+1

+1 for this, are there any plans to implement it?

FYI all, you can pretty easily manually edit the JSON representation of each visualization in the Saved Objects tab: /app/kibana#/management/kibana/objects?_g=()&_a=(tab:visualizations). Haven't tried it, but doing an export, find-replace (or pipe to jq), and reimport would probably work for most bulk reassignment workflows.

+1 absolutely , its a highly needed feature ; It will reduce lot of manual dashboard creation efforts and their maintenance work.

+1

+1

can use export and import implemented

image

8e39432f-4136-4061-8c5d-945fc81ce1e6

+1

i do not understand why on discovery we can change indexes, but on visualization we are forces to start over.

I'm just building some compex visualizations with nginx index and wanted to check if using the varnish index i could get better results... but as we are not allowed to change indexes, i have to open a new tab, change the index and setup again the visualization fields.

Notice that this is not useful to copy already implemented visualizations between different environments, but also useful to compare index visualizations and help creating the best index to use for a visualization. It is also very helpful for maintenance, if one field name changes, we can change one visualization and save for several environments. The alternative is to export all visualizations, edit and import again... it works, but it very hacky

Please, add this.

+1

+1

I implemented this feature by injecting a JS script into the Kibana page. See https://github.com/gofreddo/kibana-copy-objects/ .

+1

+1 This would be a massive improvement for us. Currently we are stuck with either having all our data in one set of daily indices so that we can share dashboards, or having to manually copy and keep up-to-date dashboards just in order to alter the index they work on.

+1

+1

+1

+1

Best work around is to save search in the discover tab of Kibana and build your visualizations/dashboards off of the saved search instead of the index pattern. Now whenever you need to switch to a different index, you could go back to the saved search and change the index from the drop down and save it. This will change the index in the visualization/dashboard.

save_search

viz

@svadapalli how is that helpful, when a search is saved with an index pattern anyway?

You can replace the existing search by mapping it to a new index. If you happen to have any dependent visualizations, switching to the index happens automatically.

₊1

Guys this is an open source piece of software, and honestly with the level of URI passed config for each visualization / dashboard, I feel like there's an easy way to just override all the instances of a referenced index pattern.

This use case is common in the world of permissions and security enforcement, where you might have tons of buckets of JSON with the same schemas, but different categories of those buckets, and different data to show users with the same dashboards but different indices. Its super sexy when you say it

+1

+1

+1

+1

+1

+1 ........... But the 111+ people * two years waiting......There should be some threshold in open source software where after a certain number of man-years (person-years?) a request should pin to the top of issues list and the person who has graciously given of his/her time to be the area owner should enter some detailed pointers (code, strategy) for solving the problem, then one of the rest of us will feel empowered to go make it happen.

I'm sure there are many more people who would dive in if they could get air lifted to the 10,000 level.

+1

+1

1+. After the new upgrade of ELK to 6.X, where types aren't being supported anymore, we will have to make a whole migration of our dashboards, as they were all based the same index, only differing on their type.

+1

+1

+1
Can we change the index of the dashboard in grafana ??

anyone kibana can close this issue if no plan for this feature

Looking at .kibana, we may need to update the value index in searchsourcejson of visualizations and searches.

"searchSourceJSON": """{"index":"024642a0-faf5-11e7-bacc-890ff6c3f975","filter":[],"query":{"query":"","language":"lucene"}}"

Regardless, if we can get index as a separate attribute, and provide API to change the attribute would help all.

Would also like this +1

I made it possible https://github.com/ArtemUstynov/kibana_dashboard_manager . also if you need to copy dashboard for new index just text me

I have a few questions to everyone that is eagerly looking forward to this feature being implemented, as I start to explore different approaches to solving these problems.

First let me try to understand the main problems that have been discussed here:

Problem 1: There is no way to easily exchange one index pattern for another in a given set of existing dashboards and visualizations.

Problem 2: You don't want to maintain a slew of duplicate visualizations and dashboards where the only thing that differs is the index pattern. This situation occurs because viewers of your dashboards have access to data that resides in different index patterns.

Does this sound accurate?

For now I'd like to focus only on Problem 2.

To give a hypothetical scenario, a company creates Kibana dashboards for their clients, client A and client B. client A's data is in client-a-* index pattern and client B's data is in client-b-* index pattern. The company now has two duplicate sets of visualizations and dashboards, where everything is equal but the index pattern used to create the visualizations.

Is that similar to the situation many of you are running into? If so, I'm curious whether anyone has tried to work around that by creating a single set of visualizations that point to index pattern client-*. The two clients data would naturally be filtered because of their permissions, so they would only see their own data.

If the issue is that the dashboard creator wants to be able to easily see the data that each client sees, could you add a client field to filter by?

screen shot 2018-03-01 at 10 47 49 am

As a side note, there is an _index meta field but you can't search using a wildcard on it, which means if your pattern is client-a-date you wouldn't be able to see across dates.

If you go this route, there is only one set of visualizations, and one dashboard to maintain. Data is filtered naturally for viewers based on their permissions, but admins who have access to all the index patterns can view everything, or segment the data. Clients will see the client field but will only see their own name as an option to select in it (so they wouldn't see a reference to a at all).

Understanding how that workaround falls short as a solution will help me figure out what concerns need to be addressed as I think about implementing this feature.

well what you describe is not the problem for me. Say i index N elements to
elastic and name it "approach 1" then i index data that had same Jsons
fields, but that data was extracted in a different (hopefully better) and
name it "approach 2" way so now i want set of mine visualizations to be
applied to a different index. Think of dashboard as of a pointer and index
as a data pointer is pointing to, so i want pointer (dashboard with all
mine visualizations ) to be applied to a new index.

To be honest with you i made a python script to work around that problem.
you can check it out i posted it as a solution. Also i added feature to
"clone" dashboard applying it to new index, so you will have two
dashboards, but they will be referencing different indexes .

On Thu, Mar 1, 2018 at 5:11 PM, Stacey Gammon notifications@github.com
wrote:

I have a few questions to everyone that is eagerly looking forward to this
feature being implemented, as I start to explore different approaches to
solving these problems.

First let me try to understand the main problems that have been discussed
here:

Problem 1: There is no way to easily exchange one index pattern for
another in a given set of existing dashboards and visualizations.

Problem 2: You don't want to maintain a slew of duplicate visualizations
and dashboards where the only thing that differs is the index pattern. This
situation occurs because viewers of your dashboards have access to data
that resides in different index patterns.

Does this sound accurate?

For now I'd like to focus only on Problem 2.

To give a hypothetical scenario, a company creates Kibana dashboards for
their clients, client A and client B. client A's data is in client-a-*
index pattern and client B's data is in client-b-* index pattern. The
company now has two duplicate sets of visualizations and dashboards, where
everything is equal but the index pattern used to create the visualizations.

Is that similar to the situation many of you are running into? If so, I'm
curious whether anyone has tried to work around that by creating a single
set of visualizations that point to index pattern client-*. The two
clients data would naturally be filtered because of their permissions, so
they would only see their own data.

If the issue is that the dashboard creator wants to be able to easily see
the data that each client sees, could you add a client field to filter by?

[image: screen shot 2018-03-01 at 10 47 49 am]
https://user-images.githubusercontent.com/16563603/36854121-2f297614-1d3e-11e8-8036-c9f69aa99ea0.png

As a side note, there is an _index meta field but you can't search using
a wildcard on it, which means if your pattern is client-a-date you
wouldn't be able to see across dates.

If you go this route, there is only one set of visualizations, and one
dashboard to maintain. Data is filtered naturally for viewers based on
their permissions, but admins who have access to all the index patterns can
view everything, or segment the data. Clients will see the client field but
will only see their own name as an option to select in it (so they wouldn't
see a reference to a at all).

Understanding how that workaround falls short as a solution will help me
figure out what concerns need to be addressed as I think about implementing
this feature.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/3668#issuecomment-369642283,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ae1KBra2IKRCwsvTuL6wLY0UKVBlwDXxks5taB2_gaJpZM4EHRML
.

Interesting, thanks for the response @ArtemUstynov. I'm not able to visit the link you supplied above, I get a 404 error.

How often do you create new indexes with different approaches? Once you have those two approaches, do you keep both around, or do you pick one and discard the other?

If your second index shared a prefix with the index created by your first approach, and you added a field called 'approach' to both indexes, would my solution above then work for you? You would be able to have a single set of visualizations using the shared prefix as the index pattern, and could switch approaches by filtering.

If we allowed users to filter with a wildcard on the automatically included _index field, then you wouldn't even need to manually add an extra field.

I could definitely see the usefulness of a bulk index change for a selected group of saved objects for one off situations. For instance, you created all your visualizations pointing to approach-a-* and now you want to point them to approach-*.

I'm just not sure what the difference is between changing indexes, and filtering on indexes (assuming the index mappings are the same, which they'd have to be for an index swap to seamlessly work). Both are a way to say "I only want to view this set of data, exclude everything else". Users already know how to filter their data on a dashboard, but changing the index would be a new concept with a new UI. I want to make sure, before we introduce a new UI/UX, that it doesn't over complicate or confuse, and that there isn't something already being used we could take advantage of to solve the problem instead.

@Stacey-Gammon here you go https://github.com/ArtemUstynov/kibana_dashboard_manager/blob/master/kibana_manager.py
`

import os
import string
import requests
import json

nativeURL = "http://localhost:5601/es_admin/.kibana/_mget"
HEADERS = {'Content-Type': 'application/json', 'kbn-xsrf': 'true', 'Host': 'localhost:5601',
'Connection': 'keep-alive', 'Accept': 'application/json'}

visURL = "http://localhost:5601/api/saved_objects/visualization?per_page=2000"

vis_list = requests.get(visURL, headers=HEADERS).json()['saved_objects']
oldName = input("Old index name: ")
newName = input("New index name: ")
for vis in vis_list:
    payload = "{\"docs\":[{\"_id\":\"" + vis['id'] + "\" ,\"_type\": \"visualization\"}]}" `'`

    VIS = json.dumps(requests.post(nativeURL, json=json.loads(payload), headers=HEADERS).json())
    VIS = json.dumps(json.loads(VIS)['docs'][0]['_source']).replace(oldName, newName)

    POSTURL = "http://localhost:5601/es_admin/.kibana/visualization/" + vis['id']
    print("ERRORS: " + str(requests.post(POSTURL, json=json.loads(VIS), headers=HEADERS).raise_for_status()))`

Well i am sorry if i didn't explain myself properly and we ended up talking about two different things. This script pretty much does what i wanted. also i made one for "cloning" dashboard. If there a way to do this natively, then i am sorry for waisting your time.

Hi @ArtemUstynov !

I have the problem:
nativeURL = "http://localhost:5601/es_admin/.kibana/_mget"
{"statusCode":404,"error":"Not Found"}'

in line: VIS = json.dumps(requests.post(nativeURL, json=json.loads(payload), headers=HEADERS).json())

Can you help me?

thx!!!

i access kibana via ssh, so your url might be different, you can find out
what is it by going to your browser and monitor network (right click ->
inspect element -> network). then go to kibana management and see and
download something , that should give you your "native url" somewhere in
the response (it will not be called "native url") but it will be easy to
spot it, just change mine url to yours.

On Tue, Mar 13, 2018 at 9:17 PM, juancar1979 notifications@github.com
wrote:

Hi @ArtemUstynov https://github.com/artemustynov !

I have the problem:
nativeURL = "http://localhost:5601/es_admin/.kibana/_mget"
{"statusCode":404,"error":"Not Found"}'

in line: VIS = json.dumps(requests.post(nativeURL,
json=json.loads(payload), headers=HEADERS).json())

Can you help me?

thx!!!


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/3668#issuecomment-372803715,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ae1KBmkOB6q1cFJaBnm_s6NXcdv92W86ks5teClUgaJpZM4EHRML
.

Thanks @ArtemUstynov, but i can't find this URL anywhere. Do you know how I can look for it? Thank you very much!
:)

@juancar1979 as i said above -> open your kibana -> go to management -> saved objects -> right click in browser -> inspect element -> network -> (if there is something clear it(crossed circle)) -> now in kibana click export everything -> some stuff will appear in your networking menu -> under column "name" select any field (first for example) -> there will be "Request URL" -> play with it and you will figure out your url. maybe download something else it was't straight forward for me either, but i tried to make it as general of an approach as i could. Also try to print out intermediate values from script, because for some reason console urls not always match with ui. Sorry, but i don't think i can help you more than this, if you use some unusual configuration for elastic server and so on, i would be able to debug it for you. Ask person who set up the servers he might help. Good luck (also you can just export everything and change index names in text editor and import it back, but this is not as cool )

thanks @ArtemUstynov !!!! I try and say u!!! 👍

@juancar1979 if you use any sort of proxy, that might be a reason why it fails. just add flag not to use proxies in script (it is easily googleble i don't remember exactly )

+1. Why the hell is this very basic request 3 years old and receiving no attention?

@ReanimationXP - this is on our radar. Part of the issue is that there are many requests lumped into this single issue. I've split them out to help us better prioritize the work:

Changing index patterns on a visualization: https://github.com/elastic/kibana/issues/17542
Dashboard level index patterns: https://github.com/elastic/kibana/issues/16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: https://github.com/elastic/kibana/issues/16831

After thinking about it, I believe the best way forward is to close this mega issue to hopefully steer the community towards giving us more specific feedback on the sub issues. If anyone strongly disagrees with this path forward, I'm certainly open to reconsidering, so feel free to comment!

To reiterate what I just added to the top of the issue:

We really appreciate all of your feedback on this mega issue so far, but we need your help as we break this down into more concrete items to work on. Even if you've already +1'd this specific issue, it would be extremely helpful if you could +1 and describe your use case in the more granular issues that are provided below. We won't ignore all the feedback folks have provided so far in this issue, but more focused feedback on the individual feature proposals would be invaluable.

Here are the three sub issues that I believe make up this mega issue:

Changing index patterns on a visualization: #17542
Dashboard level index patterns: #16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: #16831

Dashboard level index patterns seems to match this initial request best and there is a currently available workaround for it, which is mentioned in #16917.

If those three issues do not cover your particular case, please let us know!

The only thing not addressed is I could see someone wanting to do the reverse as well - monitor the same field from multiple sites (assuming sites == indexes) in one dashboard, varying some other portion of the search.. i.e. "Show me humidity levels at all sites when the temperature is > 50." "Okay, now > 51." Splunk uses optional dashboard-level "global" user variables and controls which are injected into each visualization's search string (including index) to solve both this and the index issue, as I described in #17542. I don't think current filtering could handle such a request across multiple indexes, but I could be wrong. That said, I think simply changing visuals by site (index) will obviously be far more common. A simple 'global' dropdown for changing the index, or a way of bulk-changing the index across multiple visualizations would both be better, faster to implement intermediate solutions than what currently exists, I suspect.

Very interesting @ReanimationXP. I think our filters can handle this, at least using the workaround mentioned in https://github.com/elastic/kibana/issues/16917.

Here is my attempt - I have two indexes animal-cats and animal-dogs, and I have three visualizations - one to show just the cat sounds, one for the dog sounds, and one for all. I can add a filter and it will filter the visualizations even though they have data from different indexes:

screen shot 2018-04-11 at 2 51 01 pm
screen shot 2018-04-11 at 2 51 19 pm

Is that about right?

The downside to this workaround, that I see it, is that you might not want to have to create a visualization for each "site/index pattern", in which case I could see dashboard panel level index patterns be a better solution (although then we run into some of the same issues with dashboard level index patterns).

Does sound like it's worth investigating using some sort of search/replace for variables like you mention splunk does. It would be tough to incorporate into our current infrastructure though, we'd need to change a lot of stuff around to make it user friendly, IMO.

I suffer the same issue.
I am using some trick to solve that.

  1. save your visualization.( check the "Save as a new visualization")
  2. go Management. => saved objects => Visualizations
    and save you want the visualizations.
  3. and delete the visualizations.
  4. open the json file. and change "kibanaSavedObjectMeta - searchSourceJSON - index uuid" to any uuid
    i change "5dad88d0-475b-11e8-9a8b-51472dd99c91" to "5dad88d0-475b-11e8-9a8b-51472dd99c92"
  5. go Management. => saved objects => Visualizations
    and import that json file.
  6. you can choose a new index.

good luck.

+1

+1

+1

+1

+1

@heris25 , could you please explain why in step 3 you delete the visualization? Also, in step 4, could you please clarify where you found the index uuid, I don't see anything named uuid. I appreciate if you can clarify which field I need to modify in kibanaSavedObjectMeta.searchSourceJSON.

{
  "query": {
    "query": "",
    "language": "kuery"
  },
  "filter": [
    {
      "$state": {
        "store": "appState"
      },
      "meta": {
        "alias": null,
        "disabled": false,
        "key": "cloudwatch_logs.log_group",
        "negate": false,
        "params": {
          "query": "/aws/lambda/b2_record_processor"
        },
        "type": "phrase",
        "value": "/aws/lambda/b2_record_processor",
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
      },
      "query": {
        "match": {
          "cloudwatch_logs.log_group": {
            "query": "/aws/lambda/b2_record_processor",
            "type": "phrase"
          }
        }
      }
    },
    {
      "meta": {
        "alias": null,
        "negate": false,
        "type": "phrase",
        "key": "message",
        "value": "ERROR - RECPROC - PROCESS",
        "params": {
          "query": "ERROR - RECPROC - PROCESS"
        },
        "disabled": false,
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index"
      },
      "query": {
        "match": {
          "message": {
            "query": "ERROR - RECPROC - PROCESS",
            "type": "phrase"
          }
        }
      },
      "$state": {
        "store": "appState"
      }
    }
  ],
  "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
}
Was this page helpful?
0 / 5 - 0 ratings