[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mad-dev
Subject:    [mad-dev] III_requantize() accuracy
From:       rob () mars ! org (Rob Leslie)
Date:       2004-03-24 10:12:59
Message-ID: D39B4DEC-7D7B-11D8-BAC0-000393817488 () mars ! org
[Download RAW message or body]

On Mar 24, 2004, at 12:36 AM, Grigory A. wrote:
> I am again about accuracy.
> input parameters
> exp=0xffffff6d=-147 -> -(36.75) right?
> value d2=210
>
>>> 210^(4/3)*2^(-36.75)*2^28
> ans =
>    2.89921171527700
>
> my answer 3
>
> MAD 0.15 - #define FPM_INTEL
> answer 2
>
> What is the problem?

Keeping in mind that 2^(-36.75) = 2^(-36) × 2^(-3/4), you will find 
that III_requantize(210, -147) calculates:

              210^(4/3)    ⇒ 0x04e0393e × 2^12
   0x04e0393e × 2^(-36+12) ⇒ 0x00000005
                2^(-3/4)   ⇒ 0x09837f05
   0x00000005 × 0x09837f05 ⇒ 0x00000002

I think the difference is due to rounding errors from the limited 
precision of the fixed-point type. You're talking about a difference of 
3.725e-9, pretty insignificant in the grand scheme of things.

If you configure with --enable-accuracy, you'll get a slightly better 
fixed-point multiplication in the last step that happens to yield the 
value you were expecting.

-- 
Rob Leslie
rob@mars.org



[prev in list] [next in list] [prev in thread] [next in thread]