Kibana: Discover is overwriting scripted fields when adding the field as a column

[alexfrancoeur]

Steps to reproduce:

1. Open Kibana in two tabs
2. In tab 1, create a scripted field with say just the value 1
   
3. In tab 1, navigate to discover and add that scripted field as a column
   
4. In tab 2, modify and update the scripted field to change the value to 2
   
5. In tab 1, remove the scripted field and add it as a column again
6. In tab 2, refresh the scripted field and notice that the value and now been rewritten from 2 to 1
   

It seems that discover is actually writing to the .kibana index when adding scripted fields to discover. This may be related to the popularity of a field. Either way, I imagine it is unexpected behavior. Can we get some eyes on this and introduce a quick fix if possible?

@Bargs @lukasolson @AlonaNadler @epixa

[Bargs]

I wouldn't really call this a Discover bug. It's a symptom of a more fundamental problem with how we manage saved objects in Kibana. The index pattern API shouldn't do updates based on stale data. A method like popularizeField should always get the most recent version of the doc and increment the popularity value in an atomic manner using Elasticsearch's versioning features. We don't do this at all today, so any update operation to any saved object from any app in Kibana will suffer from the same concurrency issue.

I think fixing popularizeField would be the quickest, easiest solution for this specific concurrency issue in 6.x. It's probably more medium fruit than low fruit. This bug is also as old as scripted fields themselves and it's the first time I've seen it reported, so that should be a hint about the severity.

[alexfrancoeur]

While I agree there is work to be done managing saved objects and conflicts between those, I don't necessarily agree that adding a column to Discover should write to the .kibana index or overwrite a scripted field. Unless of course you are explicitly saving a saved search. If we were to some day introduce auto-save and versioning within Discover, I could see the argument for writing to the .kibana index automatically. However, that's not the experience today so I find it concerning that we are writing to and modifying saved objects simply by interacting with the UI. That being said, this is also the first time I've heard of this bug so I imagine it is not a common issue. It's simply a concerning one for me.

[Bargs]

.kibana is Kibana's own private index. There's nothing inherently wrong with an app writing data to its own private data store as a result of any user interaction, whether the user is aware of it or not. Adding a column in Discover absolutely should not overwrite a scripted field, but that's a bug in the popularizeField method, not a fundamental issue with writing to .kibana.

That said, there are UX arguments to be made for storing the popularity metric outside of .kibana. If we think it makes sense for each user to have their own list of "popular fields" (I do) then we could store this metric in localstorage instead. I'd consider that a breaking change though so I'd only want to do it in 7.0.

For 6.x I think we either need to decide not to fix it, or fix it by fixing the popularizeField method.

[epixa]

+1 to the sentiment about user-specific popular fields, but I don't necessarily think we should rely on localstorage for that. Hanging that off of xpack users would be better since that is based on true user identities, whereas localstorage is shared amongst all users on shared machines.

I think whether or not we fix popularizeField in the meantime really comes down to the effort required.

[Bargs]

User has worked around this by turning off popularity. Since this hasn't been a highly reported issue we'll sit on it for now, down the road a better solution would be to solve the root issue with a better UI for quickly accessing any field.

[andrask]

I just found the Advanced Settings -> fields:popularLimit option. I set it to 0 that saved my brain cells from exploding. :+1:
Changing the background color based on popularity would be perfect for me. Reordering the fields is not expected or desired and just causes confusion (at least for me).

[chrisronline]

Closed by #18937