[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Iterating SunSpider
From: Peter Kasting <pkasting () google ! com>
Date: 2009-07-07 23:19:57
Message-ID: d62cf1d10907071619y1f038928sb511f43bdd8afdf9 () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
I'm more verbose than Mike, but it seems like people are talking past each
other.
On Tue, Jul 7, 2009 at 3:25 PM, Oliver Hunt <oliver@apple.com> wrote:
> If we see one section of the test taking dramatically longer than another
> then we can assume that we have not been paying enough attention to
> performance in that area,
>
It depends on what your goal with perf is. If the goal is to balance
optimizations such that operation A always consumes the same time as
operation B, you are correct. But is this always best? The current design
says "yes". The open question is whether that is the best possible design.
On Tue, Jul 7, 2009 at 3:58 PM, Geoffrey Garen <ggaren@apple.com> wrote:
>
> I also don't buy your conclusion -- that if regular expressions account for
> 1% of JavaScript time on the Internet overall, they need not be optimized.
I didn't see Mike say that regexes "did not need to be optimized".
If given an operation that occurs 20% of the time and another that occurs 1%
of the time, I certainly think it _might_ be appropriate to spend more
engineering effort on optimizing the first operation. Knowing for sure
depends on how much you value the rarer cases, for reasons such as you give
next:
Second, it's important for all web apps to be fast in WebKit -- not just the
> ones that do what's common overall. Third, we want to enable not only the
> web applications of today, but also the web applications of tomorrow.
I strongly agree with these principles, but I don't see why the current
design necessarily does a better job of preserving them than all other
designs. For example, let's say at the time SunSpider was created (and
everything was roughly equal-weighted) that one of the subtests tested a
horribly slow operation that would greatly benefit future web apps if it
improved substantially. Unfortunately, the original equal-weighting
enshrines the slowness of this operation, relative to the others being
tested, such that if you begin to make it faster, the subtests become
unbalanced and you conclude that no further work on it is needed for the
time being. This is a suboptimal outcome.
So in general, the question is: when some operation is slower than others,
what criteria can we use to make the best decisions about where to spend
developer effort? Surely our greatest cost here is opportunity cost.
I accept Maciej's statement that the current design was intentional. I also
accept that sums and geomeans each have drawbacks in guiding
decision-making. I simply want to focus on finding the best possible design
for the framework.
For example, the framework could compute both sums _and_ geomeans, if people
thought both were valuable. We could agree on a way of benchmarking a
representative sample of current sites to get an idea of how widespread
certain operations currently are. We could talk with the maintainers of
jQuery, Dojo, etc. to see what sorts of operations they think would be
helpful to future apps to make faster. We could instrument browsers to have
some sort of (opt-in) sampling of real-world workloads. etc. Surely
together we can come up with ways to make Sunspider even better, while
keeping its current strengths in mind.
PK
[Attachment #5 (text/html)]
<div class="gmail_quote">I'm more verbose than Mike, but it seems like people are \
talking past each other.</div><div class="gmail_quote"><br></div><div \
class="gmail_quote">On Tue, Jul 7, 2009 at 3:25 PM, Oliver Hunt <span \
dir="ltr"><<a href="mailto:oliver@apple.com">oliver@apple.com</a>></span> \
wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div>If we see one \
section of the test taking dramatically longer than another then we can assume that \
we have not been paying enough attention to performance in that area,</div> \
</div></blockquote><div><br></div><div>It depends on what your goal with perf is. If \
the goal is to balance optimizations such that operation A always consumes the same \
time as operation B, you are correct. But is this always best? The current design \
says "yes". The open question is whether that is the best possible \
design.</div> <div><br></div><div>On Tue, Jul 7, 2009 at 3:58 PM, Geoffrey Garen \
<span dir="ltr"><<a href="mailto:ggaren@apple.com">ggaren@apple.com</a>></span> \
wrote:<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: \
rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; "> I also don't \
buy your conclusion -- that if regular expressions account for 1% of JavaScript time \
on the Internet overall, they need not be \
optimized.</blockquote><div><br></div><div>I didn't see Mike say that regexes \
"did not need to be optimized".</div> <div><br></div><div>If given an \
operation that occurs 20% of the time and another that occurs 1% of the time, I \
certainly think it _might_ be appropriate to spend more engineering effort on \
optimizing the first operation. Knowing for sure depends on how much you value the \
rarer cases, for reasons such as you give next:</div> <div><br></div><blockquote \
class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); \
border-left-style: solid; padding-left: 1ex; "> Second, it's important for all \
web apps to be fast in WebKit -- not just the ones that do what's common overall. \
Third, we want to enable not only the web applications of today, but also the web \
applications of tomorrow.</blockquote> <div><br></div><div>I strongly agree with \
these principles, but I don't see why the current design necessarily does a \
better job of preserving them than all other designs. For example, let's say at \
the time SunSpider was created (and everything was roughly equal-weighted) that one \
of the subtests tested a horribly slow operation that would greatly benefit future \
web apps if it improved substantially. Unfortunately, the original equal-weighting \
enshrines the slowness of this operation, relative to the others being tested, such \
that if you begin to make it faster, the subtests become unbalanced and you conclude \
that no further work on it is needed for the time being. This is a suboptimal \
outcome.</div> <div><br></div></div><div>So in general, the question is: when some \
operation is slower than others, what criteria can we use to make the best decisions \
about where to spend developer effort? Surely our greatest cost here is opportunity \
cost.</div> <div><br></div><div>I accept Maciej's statement that the current \
design was intentional. I also accept that sums and geomeans each have drawbacks in \
guiding decision-making. I simply want to focus on finding the best possible design \
for the framework.</div> <div><br></div><div>For example, the framework could compute \
both sums _and_ geomeans, if people thought both were valuable. We could agree on a \
way of benchmarking a representative sample of current sites to get an idea of how \
widespread certain operations currently are. We could talk with the maintainers of \
jQuery, Dojo, etc. to see what sorts of operations they think would be helpful to \
future apps to make faster. We could instrument browsers to have some sort of \
(opt-in) sampling of real-world workloads. etc. Surely together we can come up with \
ways to make Sunspider even better, while keeping its current strengths in \
mind.</div> <div><br></div><div>PK</div></div>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic