Kibana: Panel that shows the latest value of a field

[elvarb]

In some cases you want to have a dashboard that shows the latest value of a field. For example you have concurrent connections and want the Ops team to see the current status at a glance. Same if you have calls waiting from the phone system. Would work well with the metric filter also.

Letter size scale to the size of the panel.
Custom font color.
Possibly if it's a number a different color if the value is greater than some value.

[turf00]

+1 This feature would be useful to our team as well. We would like to see the last value of a particular field grouped by another field.

[kjellski]

+1 for this, would be a great addition for our dashboard as well!

It's also usefull for everything that has a state and is logging that state.

[sahilthapar]

:+1:

[jeremybz]

+1

[webmstr]

I was hoping to work around this in a saved search by setting "size": 0 in the definition, but it's not respected there.

[elvarb]

Grafana has a single stat panel that is exactly what is needed in kibana

[jgogstad]

+1

[andrewvc]

+1

[andrewvc]

this is possible by combining filter, terms, and top_hits aggs, as in the example below. While the agg delivers the document, not the metric (top hits does not allow sub-aggs), that can be extracted in the JS code. See the example query below:

{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*",
          "analyze_wildcard": true
        }
      },
      "filter": {
        "bool": {
          "must": [
            {
              "query": {
                "query_string": {
                  "analyze_wildcard": true,
                  "query": "*"
                }
              }
            }
          ],
          "must_not": []
        }
      }
    }
  },
  "size": 0,
  "aggs": {
    "3": {
      "filter": {
        "term": {"tags": "apache_stats"}
      },
      "aggs": {
        "2": {
          "terms": {
            "field": "@host.raw",
            "size": 5
          },
          "aggs": {
            "1": {
              "top_hits": {
                "sort": [{
                    "@timestamp": {"order": "desc"}
                }],
                "size": 1

              }
            }
          }
        }
      }
    }
  }
}

[rashidkpc]

Once https://github.com/elastic/kibana/issues/3546 is in this can be closed.

[yehosef]

I'm curious why is this being approached as an aggregation? It's a simple query. We have a queue that processes log data and has a timestamp value stored that is the latests time processed (we then have a cron that takes the the next chunk of time to process and increases the time reference. For this case, the query would look like:

GET metrics-*/_search
{
  "size": 1, 
  "fields": ["redis.current_time","@timestamp"], 
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

What about an approach like this? It should have a different metric type. Maybe it could be called "single value" and it would have a sort field and direction or querystring field (like a filter). This would let it be more flexible and the interface and query would be very simple.

[andrewvc]

@yehosef the reason being that aggregations can be more easily composed
with other aggregations. You're right that the query is simpler it just
isn't as generic in the context of kibana

On Tuesday, August 25, 2015, yehosef [email protected] wrote:

I'm curious, why is this being approached as an aggregation? It's a simple
query. We have a queue that processes log data and has a timestamp value
stored that is the latests time processed (we then have a cron that takes
the the next chunk of time to process and increases the time reference. For
us the query would look like:

GET metrics-*/_search
{
"size": 1,
"fields": ["redis.current_time","@timestamp"],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}

What about an approach like this? It should have a different metric type -
maybe "latest".

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/kibana/issues/678#issuecomment-134511199.

[yehosef]

@andrewvc - this could be - but this is a very specific situation where I need a single value from the original data, the opposite of aggregations. It's seems heavy/silly to make it be an aggregation just because everywhere else it's an aggregation - when a single value solves this problem better.

[Trane9991]

I was able to configure latest response for host with configuration similar as @andrewvc . Check image below.

https://cloud.githubusercontent.com/assets/9948629/9663658/27bedefc-526e-11e5-839f-f64c827a27d2.png

[AnkurMathur14]

It would be really nice if this feature is included in up commimg kibana version

[bvorosadmin]

+1 for the ability to display the latest value.

[richard-hulm]

+1

[steveainuk]

+1

[anoinoz]

+1

[raffis]

+1

[markwalkom]

+1 for this from me, it'd be a great metric addition.

[jeromefroe]

+1

[aalexgabi]

+1

[toddiuszho]

+1

[bryantcj52]

+1

[alfeg]

+1

[richardkennedy]

+1

[richard-keller]

+1

[muchall]

+1

[mihai-dinu-hs]

+1

[chakrayadavalli]

+1

[dilip2048]

+1

[rmolabanti]

+1

[jadireddi]

+1

[karthikdivi]

+1

[tbragin]

From #5989:

At present I can show only the Max, Min, Percentile etc aggregations on a Metric. However there is no support to show the most recent value of the metric available in ElasticSearch at that point in time. Note that most recent value may not be one of the predefined aggregations available in Kibana currently. This will help in rendering say the metrics I am capturing with codahale metric library.

Another use case that I am struggling to support in Kibana is to show the time our server was last started/restarted i.e., "Up Since" time. When I choose the time range in the time picker to be last 15 mins, the "Up Since" metric visualization show the time if the server was started in that time range. I also would like to have a feature in Kibana that shows the metric value irrespective of the time range selected in time picker. Think of it like a visualization that is fixed to show the most recent value ALWAYS. For this special case, we should have a checkbox or something that we can choose in the options when we create a metric visualization to mark it as "Always show most recent value at current time irrespective of the time picker selection"

[athondapu]

+1

[jadireddi]

+1

[zphj1987]

+1
i need single stat panel

[yehosef]

http://siren.solutions/showing-the-latest-values-in-kibikibana-say-hi-to-the-rest-datasource/

[jccq]

Thanks Yehosef for posting :)! it's a "workaround", waiting for the fully worked out feature in Kibana.. but it might be interesting as it can solve a lot also a lot of other interesting scenarios

[yehosef]

@jccq - I think your approach is better/more generic. I'm not sure why Kibana is stuck on this being an aggregation - as it's inherently not. Hopefully your solution will help them see an alternative.

@tbragin - The kibi solution elegantly solve this problem - it seems. I think the normal use case for this kind of query is time-window independent. But this problem goes back to something I've mentioned before, that individual visualization on a dashboard should have the option of not inheriting the time-window of the dashboard. It's a common/generic problem that affects other dashboards/visualizations besides this case. I can have a graph where I want to zoom in on a region of activity but still since one of the graphs with a different range. If you solve that problem, then you'll be able to have this kind of metric time-independent also.

[richardkennedy]

+1

[nivoc]

+1

[oCroso]

+1

[bioffe01]

+1

[LeeSyd]

+1

[macteo]

+1

[damianharvey]

+1

[damianharvey]

I've worked around this in Kibana using Elasticsearch document versions. My use-case was showing a traffic light with the current status of each API. I create a separate "health" document type when indexing into Elasticsearch where I tie each API Resource to a document "_id". Then there is only ever one value and a pie chart shows the latest value (in my case red for "ERR" and green for "OK")

This won't work for all use-cases described here, but it's passable for mine.

[elvarb]

@damianharvey could you explain better how you do this? it actually looks to be very usable in some cases, looks fantastic visually also.

[bbeda]

+1

[rashidkpc]

We're implementing this as a metric agg, vs a whole new panel. Replacing with https://github.com/elastic/kibana/issues/6877

[mahepate]

+1

[hilocark]

+1

[karthikbaputj]

+1, Do we know when this enhancement will officially be made available ?

[Bargs]

@karthikbaputj 5.3 https://github.com/elastic/kibana/pull/7302

[karthikbaputj]

Thank you @Bargs , Is top_hits only supposed to work on Data table visualization ? If I try to use it on Pie charts it's able to query data but not rendering the chart. Would we be supporting other visualizations in future ?

[Bargs]

@karthikbaputj top hits should work on pie charts. Could you please open a new ticket with details about the issue you're seeing?

[jbgi]

@Bargs it looks like top_hits on non-number fields only works for table and metric (hard-coded!!) : https://github.com/elastic/kibana/blob/84ea50220535925ed3acd12b62c1681dbf435e01/src/ui/public/agg_types/metrics/top_hit.js#L28
Would be nice to allows other visualization plugins to use it...

[Bargs]

@jbgi I agree, it would be nice if each vis could tell top_hits what type of data it supports. Feel free to create a new ticket with more info about your use case and we might be able to figure something out.

[slavik57]

I created a visualization using the demo Kibana app to show the last status for ports:

• I used the table visualization
  
  
  
  • Metric:
  
  
  • Aggregation: Top Hit
  
  
  • Field: status
  
  
  • Aggregate With: Concatenate
  
  
  • Size: 1
  
  
  • Sort On: @timestamp
  
  
  • Order: Descending
  
  
  • Split Rows:
  
  
  • Aggregation: Terms
  
  
  • Field: port
  
  
  • Order By: Term
  
  

The only thing that I'm missing is to sort the results by the status in case I have a lot of ports and I'm just showing the top 10 results in which case I might miss the errors. But if I show all the ports there is no problem

[Sjaak01]

Came here searching for a way to do this and rather shocked that over 4 years later something basic as this still doesn't appear to be possible without jumping through many many hoops.

[Bargs]

@Sjaak01 what do you need to do that the Top Hit metric does not provide?

[yehosef]

I think the issue is that you have to do some extra steps each time:

• Aggregation: Top Hit
• Field: status
• Aggregate With: Concatenate
• Size: 1
• Sort On: @timestamp
• Order: Descending

When you could just have

• Aggregation: Last Value
• Field: status

For a common use case, it's nice to have "short-hand" or helper widgets so I can use it in the simple way I think about it. I just know that I want the last value, instead of "I want the last value so I have to use one of the aggs to get it, and I have to set some options, so it's the right value, etc."

Additionally, if you then wanted to rewrite it at some point to use a simple query instead of aggregation, it wouldn't affect the user experience - they just want the "Last Value".

[Bargs]

@yehosef after you select Top Hit, the only value you have to specify is "Field". Once you select your field, "aggregate with", "size", "sort on", and "order" all get set to sensible defaults automatically. So I don't see how the current Top Hit metric is any more burdensome than your proposed "Last Value" shorthand. If the issue is with the form simply being too complex for 90% of use cases, we could consider hiding some of the options in the "Advanced" drop down.

[yehosef]

I see what your saying - it seems the current top hits implementation is
sort of optimized for the Last Value case. As a note, if you the field you
choose is number, the aggregate is not pre-selected to "concatenate" - but
all the other types it is (not sure if that's intentional).

I don't think I would would hide the options under advanced - I think that
confuses the other valid uses of top hits.

Personally I think it still would be better to have a short-hand since it
seems to be a common use case - it's not intuitive that if I want the
latest value I would choose a top hits aggregation. After I know it's
there, you're right, it's not that much work to do. But I still think it
not so clear. Take a few new Kibana users and ask them to make a metric
with the current/latest value of a certain field - I'll be very surprised
if they would say "oh, I think I should use the Top Hits aggregation for
that." It's clever that it works, but it's not intuitive.

On Wed, Jan 24, 2018 at 8:25 PM, Matt Bargar [e-mail]
wrote:

@yehosef https://github.com/yehosef after you select Top Hit, the only
value you have to specify is "Field". Once you select your field,
"aggregate with", "size", "sort on", and "order" all get set to sensible
defaults automatically. So I don't see how the current Top Hit metric is
any more burdensome than your proposed "Last Value" shorthand. If the issue
is with the form simply being too complex 90% of use cases, we could
consider hiding some of the options in the "Advanced" drop down.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/678#issuecomment-360227604, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJHBR-fd79bHBxYpnXEAdWpgBPDKOFjks5tN3WPgaJpZM4BL_9n
.

[Bargs]

@yehosef I'd recommend creating a new Github issue so the team that works on Visualize can consider it. This sounds like an enhancement request to me, and it won't get much attention buried in this closed issue.

[yehosef]

good point - thanks.

On Thu, Jan 25, 2018 at 1:00 AM, Matt Bargar [e-mail]
wrote:

@yehosef https://github.com/yehosef I'd recommend creating a new Github
issue so the team that works on Visualize can consider it. This sounds like
an enhancement request to me, and it won't get much attention buried in
this closed issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/678#issuecomment-360303216, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJHBUbZ96SyYVMau3AV9ECSjclrB7fhks5tN7YIgaJpZM4BL_9n
.

[Sjaak01]

@Bargs It doesn't work on most fields for some reason.

I got various string fields with a text and keyword mapping and for some reason only one of them works (both text and keyword) but some other fields I have, also with a text and keyword mapping, don't display. The fields that don't display work fine in other visualizations. Of course there is data for the time range.

However as somebody else mentioned earlier in some cases you also want to only show the last value, regardless of the time range so even if this would work, it isn't a complete solution.

Not showing anything.

edit: tried with a different index and that seems to be working fine (different data but various field types, all work). Could this be some kind of bug?

[Bargs]

@Sjaak01 I would open a separate issue for your field problems with top hits so the vis team can investigate