[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-bugs-dist
Subject:    [Bug 51684] Math constant rounding error
From:       zack-weg () gmx ! de
Date:       2003-07-04 14:01:41
[Download RAW message or body]

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
     
http://bugs.kde.org/show_bug.cgi?id=51684     




------- Additional Comments From zack-weg@gmx.de  2003-07-04 16:01 -------
Note that 76.59999999999999 == 76.6 is true in JavaScript. Both have a binary
representation of 1001100.1001100110011001100110011001100110011001100110.

The ECMAScript algorithm to construct a string from a number is:

--8<----

9.8.1 ToString Applied to the Number Type

The operator ToString converts a number m to string format as follows:

[...]

5. Otherwise, let n, k, and s be integers such that k >= 1,
   10^(k-1) <= s < 10^k, the number value for s × 10^(n-k) is m, and k is as
   small as possible. Note that k is the number of digits in the decimal
   representation of s, that s is not divisible by 10, and that the least
   significant digit of s is not necessarily uniquely determined by these
   criteria.

[...]

7. If 0 < n <= 21, return the string consisting of the most significant n digits
   of the decimal representation of s, followed by a decimal point '.', followed
   by the remaining k-n digits of the decimal representation of s.

--8<----

n= 2, k= 16, s= 7659999999999999 (as kjs chooses), satisfies most of the
requirements, but k is not as small as possible. The correct choice would be
n= 2, k= 3, s= 766.

If the choice of s is still ambiguous, ECMAScript allows both representations,
but recommends to choose the one with an even last digit.
So 76.59999999999997.toString() should give "76.59999999999996", but kjs does
the opposite and gives "76.59999999999997" for 76.59999999999996.toString().
[prev in list] [next in list] [prev in thread] [next in thread]