Powershell: Float sum precision regression on 7.0.0-preview1

[vors]

Steps to reproduce

@(1.1) * 10 | measure -Sum  

Expected behavior

This is 6.2.0 behavior

Count             : 10
Average           : 
Sum               : 11
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 

Actual behavior

This is 7.0.0-preview.1

Count             : 10
Average           : 
Sum               : 10.999999999999998
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 

Environment data

Name                           Value
----                           -----
PSVersion                      7.0.0-preview.1
PSEdition                      Core
GitCommitId                    7.0.0-preview.1
OS                             Darwin 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[vors]

I guess it may be the dotnet 3.0 thing because it repros even for normal addition

PS > 1.1 + 1.1 + 1.1
3.3000000000000003

[SteveL-MSFT]

Yeah, .NET Core 3.0 made changes to how it handles precision.

[iSazonov]

It seems come from https://github.com/dotnet/coreclr/pull/22040

Perhaps we need to change output format (like ToString("g"))

This variant is chosen for "R" and when no precision specifier is given.

[vors]

I think G9 would be the old behavior equivalent

https://docs.microsoft.com/en-us/dotnet/standard/base-types/standard-numeric-format-strings?view=netframework-4.8

For Double types, use "G17"; for Single types, use "G9".

PS /Users/vors> [string]::Format("{0:G9}", (1.1 + 1.1 + 1.1)) 
3.3
PS /Users/vors> [string]::Format("{0:G17}", (1.1 + 1.1 + 1.1))
3.3000000000000003

[iSazonov]

It seems it is for PowerShell Committee review.

[jzeiders]

I discussed this with @rjmholt , @JamesWTruher and @TravisEz13.

Here's an example from PS Core 6.2:

> 1.1 + 1.1 + 1.1
3.3

> 3.3
3.3

> 1.1 + 1.1 + 1.1 -eq 3.3
False

Safe to say that's confusing at best.

Compare this to the current behavior in PS 7:

> 1.1 + 1.1 + 1.1
3.3000000000000003

> 3.3
3.3

> 1.1 + 1.1 + 1.1 -eq 3.3
False

It seems like the new behavior is less misleading and no functionality except formatting has changed (i.e. this is not a break). So this change is desirable.

We also discussed a bit how to mitigate it, with one suggestion being to default to decimal for real number representations. But that would be a pretty big breaking change, and quite unusual; no other mainstream language uses fixed-point by default, and many scripting languages only have floating-point arithmetic.

[rkeithhill]

This is the nature of floating point arithmetic and such issues have existed for decades in various languages that allow you to compare floating point numbers. Back in the 90's I worked on a graphical programming language that had a special operator for this ~= (read as "almost equals") which performed this comparison for "close enough equality": abs(lhs - rhs) < epsilon. I don't remember the exact value we used for epsilon but it was in the area of 1E-13.

I doubt the parser would support this syntax but ~eq would be pretty nifty. :-) If not that, then maybe an -fpeq or -eqfp operator (for floating point equal).

[vexx32]

Hmm. -eq~ isn't bad either syntax wise... Might be worth looking at, for sure. Technically a separate issue though.

I think that pretty much puts this issue solidly in the WontFix category. Though the change is confusing at first, I'm glad there is some visual confirmation of the results we would see anyway when checking with comparison operators. :)

[rjmholt]

There's always -feq... 😄

[vors]

Of course floating point arithmetic needs to use abs(a - b) < eps for the equality check.
I don't see how it justifies the new behavior, which I'd consider somewhat breaking.
I have few simple powershell scripts that run reporting and sum a bunch of floats like 1.1, 0.8. Now I'm just constantly have to round things up in my head. I'd say that a float format.xml or some analog should be provided so precision in the double / float display could be controlled.

[rjmholt]

I'd say the @PowerShell/powershell-committee should review this issue to be sure, but I think there are a few factors in favour of not trying to go back to the old behaviour:

• The most noticeable part of the change is interactive, which isn't classed as breaking change
• This change happened at the .NET Core 3 level, would be quite involved to mitigate, and I suspect we'd never quite restore the old behaviour. (We looked into this today and this is beyond a formatter issue; there are places that PowerShell doesn't hook into where this change happens, in fact (3*1.1).ToString() has changed behaviour (the real breaking change) and I very much doubt we could fix that in a consistent way)
• The old behaviour was totally misleading. Floating point math is hard enough for those trained to be wary of it, and PowerShell needs to be accessible to casual scripters

For a reporting script, that seems like a good candidate for '{0:g5}' -f $value

[vors]

PowerShell needs to be accessible to casual scripters

That's why 1.1 + 1.1 + 1.1 should be 3.3 :D

Thanks for the -f suggestion, is there a way to apply it to all the output of @(1.1) * 10 | measure -Sum ? In the sense of, I still want to see both Count and Sum. The only way I see with -f is to print them by hands, which is tedious.

[iSazonov]

@vors Above I referenced CoreFX PR where the change come from. But I did not read all discussions about this. While Core 3.0 is preview we could make a request to enhance an API in CoreFX to set default behavior for the formatting.

[vors]

@iSazonov I like that idea.

Let me add a little more color to the original bug report, so it's in the context.
Here is the output of my self-made-reporting script on

6.2.0

== HIGH ==

Count             : 14
Average           : 
Sum               : 5.9
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 



count points project
----- ------ -------
    1    0.2 
    5    2.1 xxxx
    1    1.5 xxxxxxxxxx
    6    1.7 xxxx
    1    0.4 xxxxxxx


== ALL ==

Count             : 22
Average           : 
Sum               : 8.8
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 




== NORMAL ==
Count             : 8
Average           : 
Sum               : 2.9
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 

7.0.0-preview1

== HIGH ==

Count             : 14
Average           : 
Sum               : 5.9
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 



            points project       count
            ------ -------       -----
               0.2                   1
               2.1 xxxx              5
               1.5 xxxxxxxxxxxxx     1
1.6999999999999997 xxxxx             6
               0.4 xxxxxxx           1


== ALL ==

Count             : 22
Average           : 
Sum               : 8.8
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 




== NORMAL ==
Count             : 8
Average           : 
Sum               : 2.9000000000000004
Maximum           : 
Minimum           : 
StandardDeviation : 
Property          : 

Inside of the script that generates it, I just use (... | measure-object -Sum).Sum for all the things. I hope that adds a little bit of color to the issue. I'd expect that to come up in many other people workflows, but don't have any data to prove it.

[SteveL-MSFT]

@PowerShell/powershell-committee discussed this, we agree to change the formatting of PSObject be consistent with Windows PowerShell 5.1. It appears from the corefx change, it used to be 15.

[vors]

@SteveL-MSFT awesome, thank you

[daxian-dbw]

Just want to add that this is a by design change in .NET Core to fix the previously broken round-trippable string returned by double.ToString().
Example:

$a = 1.1 * 3
$b = [double]::Parse($a.ToString())
$a -eq $b

In pwsh-7.0.0-preview.1, you will see $a -eq $b returns True, while in pwsh-6.2 you will see False.

To mitigate the unfriendly string representation in PowerShell, we will need to change the ToString method in MshObject.cs, so that during formatting, all double/float values can be converted to string using the old precision specifiers (G15 for double and G7 for float).
Also, the explicit conversion from double/float to [string] also needs to be changed to honor the old precision specifiers.

[msftbot[bot]]

:tada:This issue was addressed in #9893, which has now been successfully released as v7.0.0-preview.2.:tada:

Handy links:

• Release Notes