Ticket Created

CIQQA-1005

float.format() works bad

as I understand format with precision works like round(float,precision), so this code run bad (from time to time)

        var f1 = 1.545, f2=1.445;
        SYS.println(MAT.round(f1) + " " + MAT.round(f2));
        SYS.println(f1.format("%.0f") + " " + f2.format("%.0f"));
        SYS.println(f1.format("%.1f") + " " + f2.format("%.1f"));
        SYS.println(f1.format("%.2f") + " " + f2.format("%.2f"));
        SYS.println(f1.format("%.3f") + " " + f2.format("%.3f"));
        f1 = 1.51;
        for(var i = 0; i < 10; i++)
        {
            f2 = f1 + 0.001*i;
            SYS.println(f2 + " " + f2.format("%.2f"));
        }

console

2.000000 1.000000
2 1
1.5 1.4
1.54 1.45             bug - should be 1.55 1.45 (but why 1.445 is rounding well but not 1.545)
1.545 1.445
1.510000 1.51
1.511000 1.51
1.512000 1.51
1.513000 1.51
1.514000 1.51
1.515000 1.51  bug should be 1.52
1.516000 1.52
1.517000 1.52
1.518000 1.52
1.519000 1.52

_psx_
Parents
  • in reply to _psx_

    Just my 2 cents (not worth much)

    1) You can mitigate this problem somewhat by using doubles instead floats

    2) This is how floating point numbers work on every platform, so this time we really can't fault Garmin

    3) Calculators on big platforms like Mac OS and Windows are well known to have the same kind of issue, also due to floating point math

    If you want perfect accuracy, you'll have to use some sort of bespoke exact representation data type like BigDecimal in Java. Of course the big tradeoff there is lack of speed, memory concerns, and lack of native hardware arithmetical operations. But for some applications (like banking), it may be worth it. If we're just displaying measured physical data like GPS pace or HR which is already imprecise, floating point errors may be less of a big deal.

    But while we're at it, we may as well complain that no CPU in existence does exact representation of fractional numbers.

    retrocomputing.stackexchange.com/.../why-not-use-fractions-instead-of-floating-point

Comment
  • in reply to _psx_

    Just my 2 cents (not worth much)

    1) You can mitigate this problem somewhat by using doubles instead floats

    2) This is how floating point numbers work on every platform, so this time we really can't fault Garmin

    3) Calculators on big platforms like Mac OS and Windows are well known to have the same kind of issue, also due to floating point math

    If you want perfect accuracy, you'll have to use some sort of bespoke exact representation data type like BigDecimal in Java. Of course the big tradeoff there is lack of speed, memory concerns, and lack of native hardware arithmetical operations. But for some applications (like banking), it may be worth it. If we're just displaying measured physical data like GPS pace or HR which is already imprecise, floating point errors may be less of a big deal.

    But while we're at it, we may as well complain that no CPU in existence does exact representation of fractional numbers.

    retrocomputing.stackexchange.com/.../why-not-use-fractions-instead-of-floating-point

Children