[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Iterating SunSpider
From: Mike Belshe <mike () belshe ! com>
Date: 2009-07-07 23:22:39
Message-ID: 2a10ed240907071622v43c16ae2v33c998ae053c2e3f () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Tue, Jul 7, 2009 at 3:58 PM, Geoffrey Garen <ggaren@apple.com> wrote:
> Are you saying that you did see Regex as being such a high percentage of
>> javascript code? If so, we're using very different mixes of content for our
>> tests.
>>
>
> I'm saying that I don't buy your claim that regular expression performance
> should only count as 1% of a JavaScript benchmark.
I never said that.
> I don't buy your premise -- that regular expressions are only 1% of
> JavaScript execution time on the web -- because I think your sample size is
> small, and, anecdotally, I've seen significant web applications that make
> heavy use of regular expressions for parsing and validation.
If you've got data, please post it - that would be fantastic.
> I also don't buy your conclusion -- that if regular expressions account for
> 1% of JavaScript time on the Internet overall, they need not be optimized.
I never said that.
> First, generally, I think it's dubious to say that there's any major
> feature of JavaScript that need not be optimized. Second, it's important for
> all web apps to be fast in WebKit -- not just the ones that do what's common
> overall. Third, we want to enable not only the web applications of today,
> but also the web applications of tomorrow.
I never said that either.
I'm talking about how a score is computed. You're throwing a lot of red
herrings in here, and I don't know why. Is this turning into an "I'm mad at
Google vs Webkit thing?" I'm trying to have an engineering discussion about
the merits of two different scoring mechanisms.
> To some extent, I think you must agree with me, since v8 copied
> JavaScriptCore in implementing a regular expression JIT, and the v8
> benchmark copied SunSpider in including regular expressions as a test
> component.
I like SunSpider because of its balance. I think SunSpider, unlike some
> other benchmarks, tends to encourage broad thinking about all the different
> parts of the JavaScript language, and design tradeoffs between them, while
> discouraging tunnel vision. You can't just implement fast integer math, or
> fast property access, and call it a day on SunSpider. Instead, you need to
> consider many different language features, and do them all well.
You're completely missing the point.
Of course benchmarks should cover a broad set of features; that is not being
debated here. I'd love to see the tests *expanded*. Using a sum, it is
very very difficult to add "many different language features" without
causing the benchmark scoring to be very misleading. (See my other reply).
The only thing I'm debating here is how the score should be computed. There
is the summation method, which is variable over time, or you can use a
geometric mean, which is balanced.
To think of this another way - how is JavaScript benchmarking different from
other types of benchmarks? If we could answer this question, that would
help me a lot.
What do you think of spec.org? They've been using geometric mean to do
scoring for years: http://www.spec.org/spec/glossary/#geometricmean Other
benchmarks all seem to choose geometric means as well. Peacemaker also uses
a geometric mean: http://service.futuremark.com/peacekeeper/faq.action I'm
actually hard pressed to find any benchmark which uses summation. Are there
any non-sunspider benchmarks using sums?
Thanks,
Mike
Mike
>
>
> Geoff
>
[Attachment #5 (text/html)]
<div class="gmail_quote">On Tue, Jul 7, 2009 at 3:58 PM, Geoffrey Garen <span \
dir="ltr"><<a href="mailto:ggaren@apple.com">ggaren@apple.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex;"> <div class="im"><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Are you saying \
that you did see Regex as being such a high percentage of javascript code? If so, \
we're using very different mixes of content for our tests.<br> </blockquote>
<br></div>
I'm saying that I don't buy your claim that regular expression performance \
should only count as 1% of a JavaScript benchmark.</blockquote><div><br></div><div>I \
never said that.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 \
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <br>
I don't buy your premise -- that regular expressions are only 1% of JavaScript \
execution time on the web -- because I think your sample size is small, and, \
anecdotally, I've seen significant web applications that make heavy use of \
regular expressions for parsing and validation.</blockquote> <div><br></div><div>If \
you've got data, please post it - that would be fantastic.</div><div> \
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex;">I also don't buy your conclusion -- that if regular \
expressions account for 1% of JavaScript time on the Internet overall, they need not \
be optimized.</blockquote> <div><br></div><div>I never said that.</div><div> \
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex;">First, generally, I think it's dubious to say that \
there's any major feature of JavaScript that need not be optimized. Second, \
it's important for all web apps to be fast in WebKit -- not just the ones that do \
what's common overall. Third, we want to enable not only the web applications of \
today, but also the web applications of tomorrow.</blockquote> <div><br></div><div>I \
never said that either.</div><div><br></div><div>I'm talking about how a score is \
computed. You're throwing a lot of red herrings in here, and I don't know \
why. Is this turning into an "I'm mad at Google vs Webkit thing?" \
I'm trying to have an engineering discussion about the merits of two different \
scoring mechanisms. </div> <div> </div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">To some \
extent, I think you must agree with me, since v8 copied JavaScriptCore in \
implementing a regular expression JIT, and the v8 benchmark copied SunSpider in \
including regular expressions as a test component.</blockquote> \
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex;">I like SunSpider because of its \
balance. I think SunSpider, unlike some other benchmarks, tends to encourage broad \
thinking about all the different parts of the JavaScript language, and design \
tradeoffs between them, while discouraging tunnel vision. You can't just \
implement fast integer math, or fast property access, and call it a day on SunSpider. \
Instead, you need to consider many different language features, and do them all \
well.</blockquote> <div><br></div><div>You're completely missing the \
point.</div><div><br></div><div>Of course benchmarks should cover a broad set of \
features; that is not being debated here. I'd love to see the tests *expanded*. \
Using a sum, it is very very difficult to add "many different language \
features" without causing the benchmark scoring to be very misleading. (See my \
other reply).</div> <div><br></div><div>The only thing I'm debating here is how \
the score should be computed. There is the summation method, which is variable over \
time, or you can use a geometric mean, which is balanced. </div><div><br> \
</div><div>To think of this another way - how is JavaScript benchmarking different \
from other types of benchmarks? If we could answer this question, that would help me \
a lot.</div><div><br></div><div>What do you think of <a \
href="http://spec.org">spec.org</a>? They've been using geometric mean to do \
scoring for years: <a \
href="http://www.spec.org/spec/glossary/#geometricmean">http://www.spec.org/spec/glossary/#geometricmean</a> \
Other benchmarks all seem to choose geometric means as well. Peacemaker also uses a \
geometric mean: <a href="http://service.futuremark.com/peacekeeper/faq.action">http://service.futuremark.com/peacekeeper/faq.action</a> \
I'm actually hard pressed to find any benchmark which uses summation. Are there \
any non-sunspider benchmarks using sums?</div> \
<div><br></div><div>Thanks,</div><div>Mike</div><div><br></div><div><br></div><div><br \
></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Mike</div><div><br></div><div> \
> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
> solid;padding-left:1ex;">
<br>
<br>
Geoff<br>
</blockquote></div><br>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic