[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Iterating SunSpider
From: Mike Belshe <mike () belshe ! com>
Date: 2009-07-07 22:01:42
Message-ID: 2a10ed240907071501y1e0a0a23m82fbb99a40b7b961 () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Mon, Jul 6, 2009 at 10:11 AM, Geoffrey Garen <ggaren@apple.com> wrote:
> So, what you end up with is after a couple of years, the slowest test in
>> the suite is the most significant part of the score. Further, I'll predict
>> that the slowest test will most likely be the least relevant test, because
>> the truly important parts of JS engines were already optimized. This has
>> happened with Sunspider 0.9 - the regex portions of the test became the
>> dominant factor, even though they were not nearly as prominent in the real
>> world as they were in the benchmark. This leads to implementors optimizing
>> for the benchmark - and that is not what we want to encourage.
>>
>
> How did you determine that regex performance is "not nearly as prominent in
> the real world?"
>
For a while regex was 20-30% of the benchmark on most browsers even though
it didn't consume 20-30% of the time that browsers spent inside javascript.
So, I determined this through profiling. If you profile your browser while
browsing websites, you won't find that it spends 20-30% of its javascript
execution time running regex (even with the old pcre). It's more like 1%.
If this is true, then it's a shame to see this consume 20-30% of any
benchmark, because it means the benchmark scoring is not indicative of the
real world. Maybe I just disagree with the mix ever having been very
representative? Or maybe it changed over time? I don't know because I
can't go back in time :-) Perhaps one solution is to better document how a
mix is chosen.
I don't really want to make this a debate about regex and he-says/she-says
how expensive it is. We should talk about the framework. If the framework
is subject to this type of skew, where it can disproportionately weight a
test, is that something we should avoid?
Keep in mind I'm not recommending any change to existing SunSpider 0.9 -
just changes to future versions.
Maciej pointed out a case where he thought the geometric mean was worse; I
think thats a fair consideration if you have the perfect benchmark with an
exactly representative workload. But we don't have the ability make a
perfectly representative benchmark workload, and even if we did it would
change over time - eventually making the benchmark useless...
Mike
[Attachment #5 (text/html)]
<div class="gmail_quote">On Mon, Jul 6, 2009 at 10:11 AM, Geoffrey Garen <span \
dir="ltr"><<a href="mailto:ggaren@apple.com" \
target="_blank">ggaren@apple.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> So, what you end up with is after a couple of years, the \
slowest test in the suite is the most significant part of the score. Further, \
I'll predict that the slowest test will most likely be the least relevant test, \
because the truly important parts of JS engines were already optimized. This has \
happened with Sunspider 0.9 - the regex portions of the test became the dominant \
factor, even though they were not nearly as prominent in the real world as they were \
in the benchmark. This leads to implementors optimizing for the benchmark - and that \
is not what we want to encourage.<br>
</blockquote>
<br></div>
How did you determine that regex performance is "not nearly as prominent in the \
real world?"<br></blockquote><div><br></div><div>For a while regex was 20-30% of \
the benchmark on most browsers even though it didn't consume 20-30% of the time \
that browsers spent inside javascript.</div>
<div><br></div><div>So, I determined this through profiling. If you profile your \
browser while browsing websites, you won't find that it spends 20-30% of its \
javascript execution time running regex (even with the old pcre). It's more like \
1%. If this is true, then it's a shame to see this consume 20-30% of any \
benchmark, because it means the benchmark scoring is not indicative of the real \
world. Maybe I just disagree with the mix ever having been very representative? Or \
maybe it changed over time? I don't know because I can't go back in time :-) \
Perhaps one solution is to better document how a mix is chosen.</div>
<div><br></div><div>I don't really want to make this a debate about regex and \
he-says/she-says how expensive it is. We should talk about the framework. If the \
framework is subject to this type of skew, where it can disproportionately weight a \
test, is that something we should avoid?</div>
<div><br></div><div>Keep in mind I'm not recommending any change to existing \
SunSpider 0.9 - just changes to future versions.</div><div><br></div><div>Maciej \
pointed out a case where he thought the geometric mean was worse; I think thats a \
fair consideration if you have the perfect benchmark with an exactly representative \
workload. But we don't have the ability make a perfectly representative \
benchmark workload, and even if we did it would change over time - eventually making \
the benchmark useless...</div>
<div><br></div><div>Mike</div><div> </div></div>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic