[prev in list] [next in list] [prev in thread] [next in thread] 

List:       koffice-devel
Subject:    Re: patch to fix bug #18083
From:       John Dailey <dailey () vt ! edu>
Date:       2002-08-20 2:24:58
[Download RAW message or body]

Looks good to me as a workaround for this release, commit if you haven't.

The problem overall isn't a problem with libc or anything else -- it's simply 
a limitation of using the machine 'double' to store numeric values.  These 
numbers that round evenly to thousandths in base 10 do not round cleanly in 
binary so 63.145 might be stored internally as 63.14500000003 while 64.145 
might be stored internally as 64.1449999999999996.  This accounts for why one 
rounds up and one rounds down.  This can't be overcome without using either 
infinite precision number storage, or by storing in base 10.

I am hoping we can integrate some system that is already written and well 
tested into KSpread 1.3.

-John

On Monday 19 August 2002 04:11 pm, Norbert Andres wrote:
> Works for me.
>
> On Monday 19 August 2002 18:31, Ariya Hidayat wrote:
> > The following patch fix bug #18083 (kspread rounding error).
> >
> > Problem: in most system (including mine), printf("%.2f",x) results in
> > "63.15" if x is 63.145 but "64.14" if x is 64.145. For the latter, it
> > should be "64.15" (I checked with MS Excel). So it's not KLocale's or
> > QString's fault. I don't know exactly what might cause this (broken
> > libc?)
> >
> > Solution: round the number before passing it to KLocale::formatNumber().
> > Hopefully it doesn't introduce noticable penalty.
> >
> > OK to commit ?
> >
> > (John: sorry if this duplicates your work/investigation).
>
> _______________________________________________
> koffice-devel mailing list
> koffice-devel@mail.kde.org
> http://mail.kde.org/mailman/listinfo/koffice-devel

_______________________________________________
koffice-devel mailing list
koffice-devel@mail.kde.org
http://mail.kde.org/mailman/listinfo/koffice-devel
[prev in list] [next in list] [prev in thread] [next in thread]