Victoriametrics: changing precisionBits doesn't result in change in precision of stored value

[gseddon]

Describe the bug
I want to store the value of 64 bit unsigned integers (from network interface packet counters) into VictoriaMetrics, while I am evaluating using it vs Influxdb. I am aware that some loss of precision will be incurred by converting the integer to a float, however adjusting the precisionBits argument does not seem to have any effect on the precision of the stored value.

To Reproduce
Steps to reproduce the behavior
Start the victoriaMetrics docker image.
docker run -it --rm -p 8429:8428 victoriametrics/victoria-metrics
Write a value, with e.g.
curl -d 'test,tag1=value1,tag2=value2 value=18446744073670737131 1571113986843174000' -X POST 'http://localhost:8429/write'
start the docker image with a different precision, e.g.
docker run -it --rm -p 8429:8428 victoriametrics/victoria-metrics -precisionBits 20
write the same value to it
curl -d 'test,tag1=value1,tag2=value2 value=18446744073670737131 1571113986843174000' -X POST 'http://localhost:8429/write'

Any time you query these values back, it gives you the same response.

curl -G 'http://localhost:8429/api/v1/export' -d 'match={__name__!=""}'
{"metric":{"__name__":"test_value","tag1":"value1","tag2":"value2"},"values":[18446744073670000000],"timestamps":[1571113986843]}

Expected behavior
The precision of the returned value should vary with the provided precisionBits value every time I read it back. However, it does not.

Version

$ docker run -it --rm -p 8429:8428 victoriametrics/victoria-metrics --version
victoria-metrics-20191009-135259-tags-v1.28.0-0-gf6334bff

Additional context
While influxdb can store these as uint64s, even when I store it as a float in Influx, it only rounds it to the nearest 1000. VictoriaMetrics is rounding my values to the nearest 10,000,000, which makes calculating rates from these counters very inaccurate.
Is the precisionBits flag working at the moment? I need the precision to be able to match Influx at least.

[valyala]

VictoriaMetrics stores all the values as float64, so these value can lose precision when the value contains more than 12-15 decimal digits.

-precisionBits works slightly differently - it removes trailing bits in consecutive numbers in order to improve compression ratio during the encoding. It isn't recommended modifying -precisionBits command-line flag in general case.

[d2km]

That doesn't quite explain it

package main

import (
    "fmt"
    "strconv"
)

func main() {
    str := "18446744073670737131"
    u64, _ := strconv.ParseUint(str, 10, 64)
    f64, _ := strconv.ParseFloat(str, 64)
    f32, _ := strconv.ParseFloat(str, 32)
    fmt.Printf("uint64  %d\nfloat64 %.0f\nfloat32 %.0f\ndiff %.0f\n", u64, f64, f32, f64-f32)
}

outputs

uint64  18446744073670737131
float64 18446744073670737920
float32 18446744073709551616
diff -38813696

I believe converting uint64 to float64 should preserve at least 54 bits, while the above looks more like it was converted to float32 with lower bits cleared.

https://play.golang.org/p/CWE9o6S_bmc

[valyala]

VictoriaMetrics performs double conversion for input numbers:

• Initially it calls strconv.ParseFloat(str, 64). This loses precision in lower 3 decimal digits as you mentioned above.
• Then the resulting float64 number is converted into decimal int64 number and decimal exponent at positiveFloatToDecimal function. This can result in additional precision loss in lower decimal digits.

Input integer values in the range [-2^54 ... 2^54-1] shouldn't lose precision, while all the integer values outside this range may lose precision due to conversions explained above.

[d2km]

Thanks for pointing to the place in the code!

I played a bit with it and found that when I change conversionPrecision to 1e15 the precision is pretty much the same we'd get with conv.ParseFloat(str, 64) with the added cost of a couple of extra loops.

Now the question is, would it be possible for you to consider making it a configurable parameter, instead of a hard-coded value? I can imagine that for some applications 10e12 might be a great trade-off. In our case, however, where we deal predominately with uint64 counters, and where counters more than 2^54 are infrequent but not exceptional, loosing 7 decimal digits is a deal-breaker, while 3 decimal digits (or 10-12 bits) is OK.

[valyala]

Sounds great! I'll make the conversionPrecision configurable in the next release

[gseddon]

Fantastic! Thank you

[valyala]

Let's keep this issue open until the next release.

[valyala]

@gseddon , could you try building VictoriaMetrics from the commit a42b5db39f079c75a59e37957549d33af186a730 ? It has improved precision when converting big float64 values to decimal. This commit will be included in the next release.

[gseddon]

Hi valyala, thank you for the update. I have done some testing with three different versions, storing the value 18446744073670737131.

1. At the previous revision https://github.com/VictoriaMetrics/VictoriaMetrics/commit/32600ba4fc725a32cac3574968589064885cdc24
2. At the above revision, but with a custom patch applied. Here is the patch

diff --git a/lib/decimal/decimal.go b/lib/decimal/decimal.go
index 60c1d30..44caee6 100644
--- a/lib/decimal/decimal.go
+++ b/lib/decimal/decimal.go
@@ -351,4 +351,4 @@ func positiveFloatToDecimal(f float64) (int64, int16) {
        return int64(u), scale
 }

-const conversionPrecision = 1e12
+const conversionPrecision = 1e15

1. At https://github.com/VictoriaMetrics/VictoriaMetrics/commit/a42b5db39f079c75a59e37957549d33af186a730.
   Here are the results:

18446744073670737131 | original number
18446744073670000000 | unmodified code
18446744073670736000 | with our patch
18446744073670730000 | with the commit you linked

So the new behaviour is much much better than it was, but still isn't as precise as it could be. For now we are using VictoriaMetrics with the custom patch. What are your thoughts?

[valyala]

Further increased precision for float64 -> decimal conversion at cfa5e279c2b2e1b284799e6c19a649fb3fc12c68 . Now it should return 18446744073670737000 for the input number.

[valyala]

Release v1.28.1 provides increased precision when converting float64 to decimal.

Closing the issue as fixed.
@gseddon , feel free re-opening the issue if additional questions arise.