Kibana: Visualization and dashboard state, URLs, and sharing

[cjcenizal]

Meta issue

Child issues are listed below: https://github.com/elastic/kibana/issues/7899#issuecomment-257060271

Background on the problem

This topic is rooted in an IE bug caused by a hard limit on the length of characters IE accepts into its URL bar. We've been able to verify that JavaScript can read from a URL that's over this limit if a user clicks a link (but not if the URL is copy/pasted).

This topic overlaps with additional feedback we've gotten about the difficulty with sharing dashboards/visualizations and updating shared dashboards/visualizations. The problem with sharing these documents is that sharing a URL defaults to sharing a "fork" of the document, instead of sharing a link to the original document. This creates a problem when the document is changed, and other viewers of the document don't see the changes -- e.g. a bookmarked dashboard will be stale.

References

• Issue #3947: Smaller state representation to fix IE errors due URL length
• Issue #4022: Option to load saved object state instead of current object state when sharing
• Issue #1553: Url shortening (has received 10 +1s since being closed)
• PR #5497: Url shortener
  

  Current system

Visualization state is stored in both the URL and on the server. Users share visualizations by copy/pasting URLs to each other. They access visualizations by bookmarking them. There is some demand for being able to create, edit, and delete visualizations programmatically.

New visualizations

https://localhost:5601/myz/app/kibana#/visualize/create?type=pie&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'2',params:(field:bytes,ranges:!((from:0,to:100),(from:100,to:600))),schema:segment,type:range)),listeners:(),params:(addLegend:!t,addTooltip:!t,isDonut:!f,shareYAxis:!t),title:'New%20Visualization',type:pie))&indexPattern=logstash-*

Data stored in URL:

• Type
• Index pattern
• Refresh interval
• Time period
• Visualization state

Saved visualizations

https://localhost:5601/myz/app/kibana#/visualize/edit/Pie-chart?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'2',params:(field:bytes,ranges:!((from:0,to:100),(from:100,to:600))),schema:segment,type:range)),listeners:(),params:(addLegend:!t,addTooltip:!t,isDonut:!f,shareYAxis:!t),title:'Pie%20chart',type:pie))

Data stored in URL:

• Name
• Refresh interval
• Time period
• Visualization state

Shared visualizations

You can share a visualization with someone by sending them a link with just the visualization name in it.

Clicking the link will load the app, which appends the interval and visualization state – the same as the saved visualization.

Short URLs

Short URLs are hashes that reference visualizations persisted on the server. This is a very similar user-facing interface and persistence mechanism as the "Save visualization" feature.

https://localhost:5601/myz/app/kibana#/visualize/edit/Pie-chart-2 (saved)
https://localhost:5601/myz/goto/7fc85c9bb03917da6dd608c9a98bec3a (short)

Browser history

Making changes to the visualization pushes a new state to the browser history. Clicking the back and forward buttons allow you to navigation the different states the visualization has gone through as you've edited it. This is essentially an "undo/redo" feature.

Proposed system

The current system uses the URL as the source of truth for the data visualization, and stores the visualization state in memory as a secondary step.

This creates several problems: long URLs become unwieldy to share, super-long URLs cause IE bugs, and URLs that are copy/pasted default to "forking" (where each user has his/her own visualization state), instead of the generally-preferred "sharing" (where all users have the same visualization state).

We can solve these problems by inverting the roles of memory and URL. The single source of truth for the data visualization will live within memory, and the URL will be populated with the visualization state as a secondary, optional step (when forking).

In general, the URL search string will only be used for storing data for functionality such as refresh interval, time period, search, filtering, pagination, etc (ways of adjusting perspective).

As an additional and supplemental improvement, we can also use the visualization ID to reference it in the URL, instead of using its name. This will enable name-editing functionality in the future.

Users can create, edit, and delete visualizations programmatically via ES API calls. They can also programmatically create "forked visualizations" (see below).

_Note: the goals of this proposed system can be largely met by the Intermediate system, below._

https://localhost:5601/myz/app/kibana#/visualize/edit/7fc85c9bb03917da6dd608c9a98bec3a?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))

Data stored in URL:

• ID
• Refresh interval
• Time period

New visualizations

New visualizations will not have an ID associated with them until they're saved. For this reason, we'll need to surface information to the user that the visualization needs to be saved before it can be shared (e.g. a notification, banner, or warning message). This information could be surfaced when the user clicks the play button to re-render the visualization.

_Note: the visualization can still be forked without being saved._

https://localhost:5601/myz/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))

Data stored in URL:

• Refresh interval
• Time period

Saved visualizations

https://localhost:5601/myz/app/kibana#/visualize/edit/7fc85c9bb03917da6dd608c9a98bec3a?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))

Data stored in URL:

• ID
• Refresh interval
• Time period

Shared visualizations

This has the same URL format as a saved visualization. Visualizations can be shared by copy/pasting this URL to users.

As a bonus, we can remove the URL shortener, since the URL will already be short. We should leave the server-side routing functionality in place, to preserve backwards-compatibility. The short URL will direct to the old URL format, which will be handled gracefully by the client (see "Migrations", below).

In the future, we can add a polling function to detect when the visualization has changed. This change can either be automatically applied to the visualization (useful for public dashboards) if there are no local unsaved changes, or surface a "conflict" message to the user if there are local unsaved changes.

Forked visualizations

If you want to share the state of your visualization without sharing the original, you can fork it. This essentially duplicates our current URL-sharing functionality (adding index pattern and visualization state to the URL).

The user can click a "Fork" button, to generate a copy/pasteable link in a text field:

https://localhost:5601/myz/app/kibana#/visualize/fork/?type=pie&indexPattern=logstash-*&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'2',params:(field:bytes,ranges:!((from:0,to:100),(from:100,to:600))),schema:segment,type:range)),listeners:(),params:(addLegend:!t,addTooltip:!t,isDonut:!f,shareYAxis:!t),title:'New%20Visualization',type:pie))

When a user clicks this link, the app will load up, parse the URL, store visualization state in memory, and replace the URL with a format that follows the "new visualization" pattern:

https://localhost:5601/myz/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5y,mode:quick,to:now))

At this point the user can share the visualization by saving or forking it.

This supports the use-case of one of our customers:

"The reason we need the human-readable/editable URLs is because we create webpages and emails that contain links to Kibana dashboards with certain filters applied. While it was not very easy to figure out what to change in the URL for the filter value, we were able to figure it and it is now a key part of our offering to users. Without human-readable/editable URLs, I don't think we could use Kibana. We are paying gold level subscription customers btw." - @eamonngryan

_Note: This feature will work for IE users as long as they click a forked URL link. If they copy/paste a URL into the browser and the URL exceeds IE's length limit, then the visualization data will be un-parseable. In this scenario, we'll need to surface an error message to the user informing them of the problem and how to solve it (by clicking a link instead of copy/pasting)._

Browser history

Other document-centric web apps use the browser history for storing route changes (i.e. navigation changes), and surface undo/redo functionality separately (e.g. with buttons or keyboard shortcuts). We should leverage user expectations by conforming to these patterns. We can add this functionality in the future.

Migration

It's important that the new system still handle the current system's URL format gracefully. There are two migration scenarios.

Saved visualizations

A saved visualization's URL contains either a) the name and visualization state or b) just the name. In the event a user opens a saved visualization, here's what happens:

1. The client sends an API request to fetch document (using the visualization's name to look it up). The API returns the visualization state and ID.
2. The client stores the visualization state in memory.
3. The client reformats the URL to conform to the "Saved visualization" format, above (adding the ID, refresh interval, and time period, and removing the name and visualization state).

This way, the old URL will still load the most recently saved state of the visualization.

Non-saved visualizations

A user can currently create a visualization and bookmark it without saving it. If a user opens this bookmark, here's what happens:

1. The client stores the visualization state in memory.
2. The client reformats the URL to conform to the "New visualization" format, above (containing just the refresh interval and time period).

Intermediate system

We can work our way towards the Proposed system in steps. As an intermediate step, we can build the Proposed system, but keep using visualization names instead of switching to IDs. We can also preserve existing "undo/redo" functionality using history.pushState.

[rashidkpc]

Via @w33ble

Some food for thought: one of the considerations we had early on, and why all that state lives in the URL in the first place, was that a user can share their current URL and another user can end up with the same state.
Kibana 3 did this well, Kibana 4 is a bit lacking as you still need the existing saved objects and such, so if the user getting the URL wasn't on the exact same Kibana instance, they couldn't use the URL anyway. By moving all of the state into the user's session, this gets even more skewed, since the user needs to take another step to share what they see with someone else; either saving their work, or using the share/fork control.
The bigger bonus we get with state in the URL is the use of the back and forward buttons to undo and redo changes, or navigate through your "timeline" of using the app. I see this mentioned, but as kind of an afterthought, when in reality it's probably an important consideration and something that probably can't go away, even as part of an intermediate step. Pushing all of the state into localStorage or some other mechanism means we can't use the browser's built in state management anymore, and would have to build our own into Kibana. This may also require adding our own custom undo/redo controls.
I'd like to see more details here about how we're going to address undo/redo with all the state removed from the URL.

Via @tbragin

I actually don't know how many users rely heavily on Back/Forward buttons in the browser. CJ and I were talking about that after the meeting and I can't remember the last time I used it in Kibana, as the navigation typically takes care of it for me. Not saying we have to do away with supporting Back/Forward – if we can figure out a way to make it work, that's great, but the current behavior of bookmarking bothers me much more than whether Back/Forward works. At a minimum, I'd like to see the default "share URL" behavior provide an option to just share the URL without any saved state, so when the dashboards are updated on the sever, users do not end up with bookmarked URLs containing stale state.

[rashidkpc]

Don't break the backwards/forwards buttons, you will regret it, heavily. I broke it in Kibana 3 and it was a total nightmare. Make it a top priority, don't let the change happen without making sure it works. Implementing state management using the built in browser functionality makes it a lot easier for plugin authors as well as our own developers.
Seriously, don't try to justify ignoring that problem, you will regret it. The browser's backwards/forwards buttons are far more popular than IE.

[cjcenizal]

Undo/redo via history

@rashidkpc @w33ble Regarding maintaining the current "undo/redo via history" functionality. I think we can retain the existing functionality by pushing the dashboard/visualization state into the browser history. So whenever a user makes a change that currently updates the search string, we will instead do something like this:

history.pushState({
  visualizationState: {
    // State goes here
  }
});

And then in an onpopstate handler or some Angular-friendly equivalent, we can re-render the UI based on this state.

This will also put us in a good position for someday migrating to an undo/redo feature that is mediated via the app UI, because then we'll just need to push the state into an internal array instead of using the History API. (This is essentially how Redux gets you undo/redo for "free").

I don't think we need localStorage for this solution. Firefox allows you to pass state objects of up to 640kb to pushState, Chrome allows 10mb and IE11 allows 1mb (SO answer).

Linking to sessionStorage

If we need to be able to store larger history snapshots (e.g. if we allow embedding images in a dashboard, as @rashidkpc mentions below), we can use sessionStorage, since we want the stored undo/redo history to expire when the tab is closed. We can do this by saving state in sessionStorage, referenced by a unique key per snapshot, and then pass this key to pushState.

[rashidkpc]

@cjcenizal 640K ought to be enough for anybody ;-)

[rashidkpc]

@cjcenizal consider wiring it to localStorage if it isn't any more work. In the case that we denormalize dashboards at some point and allow for resource uploading (say, images) it could be possible to bump into the 640k limit fairly quickly. Certainly we could introduce work arounds, but would probably be better if we didn't have to.

[w33ble]

localStorage's 5MB limit (seems to be low end, unless we count old Android at 2MB) is more appealing. Plus, both localStorage and sessionStorage are evented, so updating history via pushState on changes should be pretty easy.

[Bargs]

Can we use a different term than "Fork" in the UI? I don't think any non-devs know what "Fork" means.

[cjcenizal]

@Bargs "Clone"?

[Bargs]

Clone, Copy, Duplicate, one of those might work. OSX uses "Duplicate" in finder's right click menu. Google Drive uses "Make a Copy"

[Bargs]

Also, when pushing/popping history entries would you also be changing the URL? I don't feel great about adding history entries without changing the URL. My intuition as a user is that when I click back/forward, I'm navigating between locations that I could share with someone via copy paste.

[cjcenizal]

@Bargs I agree with you. Maybe we should just include the "undo/redo" UI controls as part of this task. I think that might be the only way of fulfilling our goals:

1. Removing application state from the URL
2. Preserving undo/redo functionality
3. Providing the user with a clear mental model of how the browser history is being used

My only concern is that does increase the amount of time it will take to complete this feature, and it also means we can't make these improvements incrementally.

[Bargs]

If other folks agree that's the right solution, I think it's better to rip off the bandaid now rather than break backwards compatibility in multiple releases. Removing state from the URL will be one backwards compatibility break (because users can no longer copy/paste from url bar), then moving undo/redo from the browser nav buttons to in-app UI will be another break.

[cjcenizal]

Application state should be sticky

@w33ble pointed out to me that if a user is viewing a cloned visualization, navigates to another app, and then comes back to the Visualization app, then they would expect the app to default to the state in which they left it (i.e. displaying the cloned visualization). This just means that we shouldn't clear a loaded cloned visualization until the user has explicitly done so via the Visualization UI (e.g. by creating a new visualization or opening an existing one).

[cjcenizal]

State "dirty" flags

Let's look ahead for a moment. We have plugins such as Reporting which need to know whether a state is "dirty" or not, i.e. has it been edited but not yet saved.

Currently, this information is provided _implicitly_ via a view controller's consumption of the stateMonitorFactory (https://github.com/elastic/kibana/pull/8148). This implicit dependency is problematic because if you look at the code in this PR it looks like zombie code, because nothing in the Kibana codebase depends upon it.

Ideally, Reporting (and other plugins) should _explicitly_ depend on this information, by retrieving it from an Angular service or Redux store.

As we reassess the way we store state, we should also think about how to surface "dirty" state in a globally-accessible manner, so we can make these implicit dependencies explicit.

[cjcenizal]

Test use cases

After doing some testing with @LeeDr, he suggested that we record some test use cases to ensure our long-term solutions works for all of them:

1. Can a user bookmark a URL? E.g. bookmarking hashed URLs doesn't work.
2. Do old bookmarked URLs still work?
3. Can a user copy-paste a URL into the URL bar? E.g. copy-pasting a long URL in IE11 doesn't work.
4. Can a user copy-paste a URL to a coworker in an email? E.g. copy-pasting a hashed URL doesn't work.

[cjcenizal]

Next steps

• [ ] Display indicator in the UI about whether a document is saved, unsaved, or in the process of saving. https://github.com/elastic/kibana/issues/8886
• [ ] Implement memory-based undo/redo. https://github.com/elastic/kibana/issues/8887
• [ ] Identify which parts of vis state are appropriate for storing in the URL (e.g. refreshInterval and timePeriod). https://github.com/elastic/kibana/issues/8888

[trevan]

Is the browser back/forward button still going to function?
And if you reload the browser, can you still use the back/forward button (either the browser one or the undo/redo controls in the app UI)?

[cjcenizal]

The browser back/forward buttons will be used to help the user support their navigational history (e.g. changing apps, changing sections within an app).

Reloading the browser will probably wipe the undo/redo history (this is the behavior in Google Docs), but of course the browser history back/forward buttons will be unaffected.

We could theoretically retain the undo/redo history of a document by storing it in localStorage, so that a reload wouldn't wipe it, but I doubt this scenario will occur often enough for it to be useful.

[elasticmachine]

Pinging @elastic/kibana-app