Ticket Created
over 3 years ago

CIQQA-1005

float.format() works bad

as I understand format with precision works like round(float,precision), so this code run bad (from time to time)

        var f1 = 1.545, f2=1.445;
        SYS.println(MAT.round(f1) + " " + MAT.round(f2));
        SYS.println(f1.format("%.0f") + " " + f2.format("%.0f"));
        SYS.println(f1.format("%.1f") + " " + f2.format("%.1f"));
        SYS.println(f1.format("%.2f") + " " + f2.format("%.2f"));
        SYS.println(f1.format("%.3f") + " " + f2.format("%.3f"));
        f1 = 1.51;
        for(var i = 0; i < 10; i++)
        {
            f2 = f1 + 0.001*i;
            SYS.println(f2 + " " + f2.format("%.2f"));
        }

console

2.000000 1.000000
2 1
1.5 1.4
1.54 1.45             bug - should be 1.55 1.45 (but why 1.445 is rounding well but not 1.545)
1.545 1.445
1.510000 1.51
1.511000 1.51
1.512000 1.51
1.513000 1.51
1.514000 1.51
1.515000 1.51  bug should be 1.52
1.516000 1.52
1.517000 1.52
1.518000 1.52
1.519000 1.52

_psx_
  • over 3 years ago in reply to flowstate

    Of course I want how to fix it but I don't plan to write calculator app Slight smile.

  • over 3 years ago in reply to _psx_

    Just my 2 cents (not worth much)

    1) You can mitigate this problem somewhat by using doubles instead floats

    2) This is how floating point numbers work on every platform, so this time we really can't fault Garmin

    3) Calculators on big platforms like Mac OS and Windows are well known to have the same kind of issue, also due to floating point math

    If you want perfect accuracy, you'll have to use some sort of bespoke exact representation data type like BigDecimal in Java. Of course the big tradeoff there is lack of speed, memory concerns, and lack of native hardware arithmetical operations. But for some applications (like banking), it may be worth it. If we're just displaying measured physical data like GPS pace or HR which is already imprecise, floating point errors may be less of a big deal.

    But while we're at it, we may as well complain that no CPU in existence does exact representation of fractional numbers.

    retrocomputing.stackexchange.com/.../why-not-use-fractions-instead-of-floating-point

  • over 3 years ago in reply to Travis.ConnectIQ

    In this case, IEEE /numeric representation of a number/etc has no meaning. The format/round/ etc function are the interface between the device and the user and should do what the user expects.

    In many situation this bug has no meaning because user doesn't know what number I format unless the difference can significant (e.g. show him that his body temperature is 52C).

    If I wanted to write a calculator application and I would add the round function I couldn't use the system one because it works badly. The user would enter 1.515 and received 1.51 and of course report an error. Should I explain to the user the nuances of IEEEE? Therefore, I think this is a bug.

  • over 3 years ago

    There is nothing wrong with that output.

        https://godbolt.org/z/fqeTeaeG1

    This is just another reminder that floating point numbers are not perfect. IEEE745 single precision floating point (float) cannot represent 1.545 exactly. The closest representable values are as follows...

    1. 1.54499983788
    2. 1.54499995708 (this is the closest to 1.545)
    3. 1.54500007629

    When you round 1.5449999570 to two digits after the decimal point, you get 1.54.