Kibana: Add a field formatter for bits using existing numeral support

[loekvangool]

Please consider adding a Bits formatter, next to the existing Bytes formatter. In networking metrics, expressing things in Bits is very common and it would be great if Kibana supports it, too.

[Bargs]

Sort of related to this: https://github.com/elastic/kibana/issues/7543 though the bits/bytes conversion would be a bit easier. But we'd probably want to tackle these things at the same time.

[nathancheek]

I would really like to see this format as well. It is very helpful and often expected to see data expressed in bits per second when dealing with network utilization. A scripted field can be created that multiplies bits by 8, but then it loses the kilo, mega, giga, etc formatting. 19000000 bits/second isn't nearly as easy to read as 19 mbits/second.

[pblasquez]

Hello,

I am very interested in this as well so I thought I would share what I have found:

The original Numeral-js project stopped updating in 2014. At the time, there was a merge request to fix the decimal vs binary formatting. This merge request is in fact still open.

This merge request was mentioned in another pull request for a project named Numbro. The request was merged and in fact has been iterated upon since then.

Numbro is in fact the living fork of Numeral-js!

I hope that it is not too much trouble to change the dependency from Numeral-js to Numbro so that we can take advantage of the past 2 years of advancement.

[Bargs]

There was another ticket about switching to numbro and a couple of concerns were raised: https://github.com/elastic/kibana/issues/7049

Another one I would add is API backwards compatibility. We've allowed users to write formatters using numeral.js syntax so numbro either needs to provide backwards compatibility or we'll have to implement a compatibility layer, which is something we'd like to do anyway but it would take more effort.

I think it's worth considering though.

[epixa]

We really need to address this change only after finishing https://github.com/elastic/kibana/issues/8583, otherwise we risk breaking an untold number of Kibana installs. The reason this is so complicated is exactly why that other issue exists in the first place: switching out our number formatting dependency should be irrelevant to end users of Kibana, but it's not, and we have nothing in place to increase confidence that making such a change doesn't break tons of existing behaviors.

[Sawyer612]

+1
We leverage the Byte format to great effect on our storage dashboards. However, our data transmission metrics are left to a static Gbps, Mbps or Kbps. It would be a great addition to allow bps with the 1000 multiplier rather than the 1024 for storage. We are forced to choose which multiplier we need most for each chart/metric visualization. A more dynamic field would be greatly appreciated! Thanks.

[shaharmor]

Super important for us as well. Also need to make sure it gets to TSVB as well.

[davidban77]

+1

[MatJohnson]

+1

[robcowart]

As Kibana should support Numeral.js formatting, I should be able to use the bits formatting seen in the docs here...

However, adding that to TSVB doesn't work...

It just produces the same value as Bytes but messes up the number of decimal points.

Is a current version of Numeral.js being used?

[robcowart]

Actually, let's look at a different example from the numeral.js docs.

Given the value 3072, a format of 0.0 b should produce 3.1 KB, i.e. it divides by 1000. This is similar to the format 0.0 a, however we want the bytes-specific configuration because when the value gets above the millions, we want to see giga (G), not billion (B).

If we were to use the 0.0 ib format, it should divide by 1024 and the result would be 3.0 KiB (similar to the example above in the docs that produces 3.154 TiB).

Now let's look at what happens in Kibana (6.5.1 was used for this example).

Let's start with bytes as a raw number:

Using the provides Bytes option clearly produces the 1024-based ibi result. This is fine for representing things like storage volume, but not network bandwidth.

We also have the option to setup a custom format, using numeral.js syntax. 0.0 b should provide the 1000-based calculation, similar to 0.0 a, but with K/M/G/T units. However, it is clearly producing the 1024-based result.

Finally, 0.0 ib, which should produce a 1024-based result, does the right math, but messes with the decimals.

The question is, what is causing this issue. Is it Kibana code, or the underlying numeral.js library?

BTW, 0.0 a does the correct math we would want from 0.0 b, however the units are k/m/b instead of the desired K/M/G.

[BobBlank12]

Sorry if this is just noise on this thread... but in the meantime, does using the MATH aggregation help some people out? My test data input "bytes" is cumulative - (but not very random, sorry) so I use the max, derivative, then a math agg to convert to megabits

[wylieconlon]

If I'm reading this issue correctly, there are two potential solutions:

• You have a number measured in bytes, and want to display it using bits. So in this case, you might have a field like bytes_in with the value 1000 and you would display it as either 8000 bits or 8kbit
• You have a number measured in bits, and want to display it using bits with abbreviations. So you have the number 8000 and want to display 8kbit

It's unclear to me which one is preferable here. The talk of multiplying by 8 suggests that you are mostly measuring bytes, but want to convert from bytes -> bits.

[BobBlank12]

Network devices often record in bits... but with high speed routers/switches - it is often easier to read in Mbits or Gbits (or Mbytes, or Gbytes). So the ability to know it is a network speed measurement and then be able to select what makes sense for your graph. whether it's bits/bytes or some multiplier like mega or giga... The divide by 8 was to convert from bits to bytes for this one particular case.

[robcowart]

@wylieconlon there are two issues here.

The first is the bits vs bytes. Any conversion between bits and bytes could be done in TSVB with math. However the display of units should be different between the two, with the common convention being that bytes uses an uppercase B and bits uses lowercase b.

The second issue is regarding the abbreviation of the value. Currently the bytes format uses 1024-based (binary) abbreviation, which is the common convention for storage. However networks use 1000-based (decimal) abbreviation. For example, a 1Gb/s network interface is 1000Mb/s, not 1024Mb/s. The difference can be seen here...

(from https://en.wikipedia.org/wiki/Kibibyte)

Combining these two means that TSVB needs to support four options: binary bytes, binary bits, decimal bytes, decimal bits.

Consider the value 10,000,000 bytes. Using these formatting options would produce the following values:

binary bytes: 9.54MiB
decimal bytes: 10.0MB

Using math to multiple by 8 and using the bits formatting options proposed above would produce the following values:

binary bits: 76.29Mib
decimal bits: 80.0Mb

It is up to the user to understand the original content of the data (bits or bytes) and apply any math needed for relevant conversion. This is no different than it is today. However the formatters should be available that provided for the proper display of units in bits or bytes, as well as decimal or binary abbreviation.

[wylieconlon]

@robcowart Thanks for being specific about what you are looking for with bits, that is helpful. It sounds like you do not want a formatter that multiplies or divides by 8 for you.

I have already started working on binary vs decimal bytes separately, and I think we might be forced to provide 3 options instead of 2 because of inconsistency in the real world. Please see the discussion here: https://github.com/elastic/kibana/issues/7543

In particular, I am implementing new formats in our numeral fork here: https://github.com/elastic/numeral-js/pull/17

[robcowart]

I have tested out the changes made by @wylieconlon (docs are here... https://www.elastic.co/guide/en/kibana/current/numeral.html#_bytes_and_bits) and they work fine. I think this can be closed.

[maintain3r]

So what changes should I make to fix the thing?
Thank you.

[wylieconlon]

As documented in the Kibana numeral formatting docs, you can now use the format string 0.00bitb or 0.00bitd to get bits formatting in Kibana. This has been supported since 7.7, but is hard to find using the Kibana UI without reading documentation.

I've changed the title of the issue now that it's unblocked. We could still provide a separate formatter for bits in the Kibana management UI.