Rounding Bug?

This smells like a bug to me. If Dist = 9.995, valStr should round up to 10.00. It does not. However, 9.9951 does round up. I believe it is common practice that 0.5 rounds up.

Dist = 9.9951;
if (Dist < 10.0) { valStr = Dist.format("%.2f"); } else { valStr = Dist.format("%.1f"); }

Former Member over 6 years ago

I don't have a ton of knowledge on how this actually works, but I believe it may be rounding this way because 9.995 cannot be represented exactly by the floating point specification. It will end up being stored as:

9.99499988555908203125

9.9951 gets stored as 9.9951000213623046875

Reference: https://www.h-schmidt.net/FloatConverter/IEEE754.html

Also this sprintf tester online produces 9.99 in this case: http://php.fnlist.com/string/sprintf

DaveBrillhart over 6 years ago

I don't have a ton of knowledge on how this actually works, but I believe it may be rounding this way because 9.995 cannot be represented exactly by the floating point specification. It will end up being stored as:

9.99499988555908203125

9.9951 gets stored as 9.9951000213623046875

Reference: https://www.h-schmidt.net/FloatConverter/IEEE754.html

Also this sprintf tester online produces 9.99 in this case: http://php.fnlist.com/string/sprintf

Ok... that makes sense. Thanks!

flocsy🤠 over 1 year ago

actually according to what I remember form math class 0.5 rounds down, so this seems to be perfectly OK.