Ticket Created
over 3 years ago

CIQQA-1005

float.format() works bad

as I understand format with precision works like round(float,precision), so this code run bad (from time to time)

        var f1 = 1.545, f2=1.445;
        SYS.println(MAT.round(f1) + " " + MAT.round(f2));
        SYS.println(f1.format("%.0f") + " " + f2.format("%.0f"));
        SYS.println(f1.format("%.1f") + " " + f2.format("%.1f"));
        SYS.println(f1.format("%.2f") + " " + f2.format("%.2f"));
        SYS.println(f1.format("%.3f") + " " + f2.format("%.3f"));
        f1 = 1.51;
        for(var i = 0; i < 10; i++)
        {
            f2 = f1 + 0.001*i;
            SYS.println(f2 + " " + f2.format("%.2f"));
        }

console

2.000000 1.000000
2 1
1.5 1.4
1.54 1.45             bug - should be 1.55 1.45 (but why 1.445 is rounding well but not 1.545)
1.545 1.445
1.510000 1.51
1.511000 1.51
1.512000 1.51
1.513000 1.51
1.514000 1.51
1.515000 1.51  bug should be 1.52
1.516000 1.52
1.517000 1.52
1.518000 1.52
1.519000 1.52

_psx_
Parents
  • over 3 years ago in reply to Travis.ConnectIQ

    In this case, IEEE /numeric representation of a number/etc has no meaning. The format/round/ etc function are the interface between the device and the user and should do what the user expects.

    In many situation this bug has no meaning because user doesn't know what number I format unless the difference can significant (e.g. show him that his body temperature is 52C).

    If I wanted to write a calculator application and I would add the round function I couldn't use the system one because it works badly. The user would enter 1.515 and received 1.51 and of course report an error. Should I explain to the user the nuances of IEEEE? Therefore, I think this is a bug.

Comment
  • over 3 years ago in reply to Travis.ConnectIQ

    In this case, IEEE /numeric representation of a number/etc has no meaning. The format/round/ etc function are the interface between the device and the user and should do what the user expects.

    In many situation this bug has no meaning because user doesn't know what number I format unless the difference can significant (e.g. show him that his body temperature is 52C).

    If I wanted to write a calculator application and I would add the round function I couldn't use the system one because it works badly. The user would enter 1.515 and received 1.51 and of course report an error. Should I explain to the user the nuances of IEEEE? Therefore, I think this is a bug.

Children
  • over 3 years ago in reply to flowstate

    Of course I want how to fix it but I don't plan to write calculator app Slight smile.

  • over 3 years ago in reply to _psx_

    Just my 2 cents (not worth much)

    1) You can mitigate this problem somewhat by using doubles instead floats

    2) This is how floating point numbers work on every platform, so this time we really can't fault Garmin

    3) Calculators on big platforms like Mac OS and Windows are well known to have the same kind of issue, also due to floating point math

    If you want perfect accuracy, you'll have to use some sort of bespoke exact representation data type like BigDecimal in Java. Of course the big tradeoff there is lack of speed, memory concerns, and lack of native hardware arithmetical operations. But for some applications (like banking), it may be worth it. If we're just displaying measured physical data like GPS pace or HR which is already imprecise, floating point errors may be less of a big deal.

    But while we're at it, we may as well complain that no CPU in existence does exact representation of fractional numbers.

    retrocomputing.stackexchange.com/.../why-not-use-fractions-instead-of-floating-point