[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Iterating SunSpider
From: Mike Belshe <mike () belshe ! com>
Date: 2009-07-07 23:28:30
Message-ID: 2a10ed240907071628r79534747hab02857b7c5e1fe2 () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Tue, Jul 7, 2009 at 4:20 PM, Maciej Stachowiak <mjs@apple.com> wrote:
>
> On Jul 7, 2009, at 4:01 PM, Mike Belshe wrote:
>
>
>> I'd like benchmarks to:
>> a) have meaning even as browsers change over time
>> b) evolve. as new areas of JS (or whatever) become important, the
>> benchmark should have facilities to include that.
>>
>> Fair? Good? Bad?
>>
>
> I think we can't rule out the possibility of a benchmark becoming less
> meaningful over time. I do think that we should eventually produce a new and
> rebalanced set of test content. I think it's fair to say that time is
> approaching for SunSpider.
I certainly agree that updating the benchmark over time is necessary :-)
>
>
> In particular, I don't think geometric means are a magic bullet.
Yes, using a geometric mean does not mean that you never need to update the
test suite. But it does give you a lot of mileage :-) And I think its
closer to an "industry standard" than anything else (spec.org).
> When SunSpider was first created, regexps were a small proportion of the
> total execution in what were the fastest publicly available at the time.
> Eventually, everything else got much faster. So at some point, SunSpider
> said "it might be a good idea to quadruple the speed of regexp matching
> now". But if it used a geometric mean, it would always say it's a good idea
> to quadruple the speed of regexp matching, unless it omitted regexp tests
> entirely. From any starting point, and regardless of speed of other
> facilities, speeding up regexps by a factor of N would always show the same
> improvement in your overall score. SunSpider, on the other hand, was
> deliberately designed to highlight the area where an engine most needs
> improvement.
I don't think the optimization of regex would have been effected by using a
different scoring mechanism. In both scoring methods, the score of the
slowest test is the best pick for improving your overall score. So vendors
would still need to optimize it to keep up.
Mike
>
> I think the only real way to deal with this is to periodically revise and
> rebalance the benchmark.
>
> Regards,
> Maciej
>
>
[Attachment #5 (text/html)]
<div class="gmail_quote">On Tue, Jul 7, 2009 at 4:20 PM, Maciej Stachowiak <span \
dir="ltr"><<a href="mailto:mjs@apple.com">mjs@apple.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex;"> <div class="im"><br>
On Jul 7, 2009, at 4:01 PM, Mike Belshe wrote:<br>
<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>
I'd like benchmarks to:<br>
a) have meaning even as browsers change over time<br>
b) evolve. as new areas of JS (or whatever) become important, the benchmark \
should have facilities to include that.<br> <br>
Fair? Good? Bad?<br>
</blockquote>
<br></div>
I think we can't rule out the possibility of a benchmark becoming less meaningful \
over time. I do think that we should eventually produce a new and rebalanced set of \
test content. I think it's fair to say that time is approaching for \
SunSpider.</blockquote> <div><br></div><div>I certainly agree that updating the \
benchmark over time is necessary :-)</div><div> </div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br> <br>
In particular, I don't think geometric means are a magic bullet. \
</blockquote><div><br></div><div>Yes, using a geometric mean does not mean that you \
never need to update the test suite. But it does give you a lot of mileage :-) And \
I think its closer to an "industry standard" than anything else (<a \
href="http://spec.org">spec.org</a>).</div> <div><br></div><div> </div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex;">When SunSpider was first created, regexps were a small \
proportion of the total execution in what were the fastest publicly available at the \
time. Eventually, everything else got much faster. So at some point, SunSpider said \
"it might be a good idea to quadruple the speed of regexp matching now". \
But if it used a geometric mean, it would always say it's a good idea to \
quadruple the speed of regexp matching, unless it omitted regexp tests entirely. From \
any starting point, and regardless of speed of other facilities, speeding up regexps \
by a factor of N would always show the same improvement in your overall score. \
SunSpider, on the other hand, was deliberately designed to highlight the area where \
an engine most needs improvement.</blockquote> <div><br>I don't think the \
optimization of regex would have been effected by using a different scoring \
mechanism. In both scoring methods, the score of the slowest test is the best pick \
for improving your overall score. So vendors would still need to optimize it to keep \
up.</div> <div><br></div><div><br></div><div>Mike</div><div><br></div><div><br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex;"><br> <br>
I think the only real way to deal with this is to periodically revise and rebalance \
the benchmark.<br> <br>
Regards,<br><font color="#888888">
Maciej<br>
<br>
</font></blockquote></div><br>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic