[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: RFR(XS): 8200187: Exclude 3 long-running tests from tier1
From: Leonid Mesnik <leonid.mesnik () oracle ! com>
Date: 2018-03-28 0:22:43
Message-ID: 5B7068DF-7FFC-41A8-AFC5-4D8DFDFE2B3D () oracle ! com
[Download RAW message or body]
Chris, David
Thank you for review.
> On Mar 26, 2018, at 5:48 PM, Chris Plummer <chris.plummer@oracle.com> wrote:
>
> Hi Leonid,
>
> The exclusion of these 3 tests from tier1 looks ok to me. It did lead me to some \
> other questions however. The first was that if you exclude them from tier1, how to \
> they get included in a later tier. The answer, for the SA tests, is that all of \
> serviceability is included in tier2. So the next question was whether or not we \
> want to repeat testing in higher tiers. I suppose since higher tiers include new \
> platforms and build options, the answer is yes, but I hope the upper tiers are \
> constructed well enough that we aren't wasting too much time repeating the same \
> test(s) on the same platforms with the same options.
It depends from component . Runtime the groups organized by excluding previous tiers \
like:
hotspot_tier2_runtime = \
runtime/ \
serviceability/ \
-runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java \
-runtime/Thread/TestThreadDumpMonitorContention.java \
-runtime/containers/ \
-:tier1_runtime \
-:tier1_serviceability \
-:hotspot_tier2_runtime_platform_agnostic
…
hotspot_tier3_runtime = \
runtime/ \
serviceability/ \
-runtime/containers/ \
-:tier1_runtime \
-:tier1_serviceability \
-:hotspot_tier2_runtime_platform_agnostic \
-:hotspot_tier2_runtime
…
So once test is excluded from tier1 it automatically is executed as a part of tier \
2/3. While these tiers doesn't execute tests from tier1. The other teams organize \
their testing slightly different. However the whole idea is that test excluded from \
tier1 should be executed in latest tiers and test should not re-executed in fast \
tiers once it was done already. However during testing like PIT we might just run \
whole hotspot_[component] team even some subset was run always in CI.
Leonid
> thanks,
>
> Chris
>
> On 3/26/18 4:32 PM, Leonid Mesnik wrote:
> > Hi
> >
> > There is no strict rules and time budgets for hotspot tiers. It is assumed that \
> > faster tests should be executed earlier tiers. The order and total impact of of \
> > these test depends on how they are executed.
> > Here is time of execution of tier1 groups with and without these excluded tests:
> > The whole :tier1 time reduced from 40 to 30 minutes (on dedicated HW with 32 \
> > core) Time for :tier1_serviceability reduced from 15 to 6 min (in HS CI)
> > Time for :tier1_gc_1 reduced from 12 to 8-10 min (in HS CI)
> >
> > The benefits of exclusion of gc/g1/ihop/TestIHOPErgo.java are not so significant. \
> > However it is the only one GC stress test executed in tier1 while intention is to \
> > don't run stress testing in tier1.
> > Tests
> > serviceability/sa/ClhsdbScanOops.java
> > serviceability/sa/TestHeapDumpForLargeArray.java
> > are now hotspot_tier3_runtime which includes all hotspot_serviceability tests \
> > which are not a part of tier1. They are executed as a part of tier3 now as well \
> > as most of serviceability tests.
> > Test gc/g1/ihop/TestIHOPErgo.java in now in group :hotspot_gc only and also it is \
> > marked as "stress" test. So it is executed with all other GC stress tests
> > Leonid
> >
> > > On Mar 25, 2018, at 3:48 PM, David Holmes <david.holmes@oracle.com> wrote:
> > >
> > > Hi Leonid,
> > >
> > > On 24/03/2018 9:31 AM, Leonid Mesnik wrote:
> > > > Hi
> > > > Could you please review following fix which exclude following tests from \
> > > > tier1 testing: serviceability/sa/ClhsdbScanOops.java
> > > > serviceability/sa/TestHeapDumpForLargeArray.java
> > > > gc/g1/ihop/TestIHOPErgo.java
> > > > Each of them takes more then 5 minutes to complete and significantly increase \
> > > > overall time to complete tier1.
> > > I'd need to see a much more detailed analysis of all the tests run, the order \
> > > they run and the execution times to ascertain what impact this actually has on \
> > > overall test execution time.
> > > But assuming 5 minutes is too long for tier1, this seems okay. But the tests \
> > > must still be run in some other tier, and I'm not clear where that would be \
> > > now? I would expect them to move to tier 3 perhaps, depending on what the time \
> > > criteria for tier 3 is.
> > > Thanks,
> > > David
> > >
> > > > Please let me know if there are any reasons to run these tests in tier1 \
> > > > despite on their execution time. \
> > > > webrev:http://cr.openjdk.java.net/~lmesnik/8200187/webrev.00/
> > > > bug: https://bugs.openjdk.java.net/browse/JDK-8200187
> > > > Leonid
>
>
>
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class="">Chris, David<div class=""><br \
class=""></div><div class="">Thank you for review. </div><div class=""><br \
class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On \
Mar 26, 2018, at 5:48 PM, Chris Plummer <<a href="mailto:chris.plummer@oracle.com" \
class="">chris.plummer@oracle.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><div class="">Hi Leonid,<br \
class=""><br class="">The exclusion of these 3 tests from tier1 looks ok to me. It \
did lead me to some other questions however. The first was that if you exclude them \
from tier1, how to they get included in a later tier. The answer, for the SA tests, \
is that all of serviceability is included in tier2. So the next question was whether \
or not we want to repeat testing in higher tiers. I suppose since higher tiers \
include new platforms and build options, the answer is yes, but I hope the upper \
tiers are constructed well enough that we aren't wasting too much time repeating the \
same test(s) on the same platforms with the same options.<br class=""><br \
class=""></div></div></blockquote><div><br class=""></div>It depends from component . \
Runtime the groups organized by excluding previous tiers like:<div><div class=""><br \
class=""></div><div class=""><div class="">hotspot_tier2_runtime = \</div><div \
class=""><div class=""> runtime/ \</div><div class=""> serviceability/ \
\</div><div class=""> -runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java \
\</div><div class=""> -runtime/Thread/TestThreadDumpMonitorContention.java \
\</div><div class=""> -runtime/containers/ \</div><div class=""><b \
class=""> -:tier1_runtime \</b></div><div class=""><b \
class=""> -:tier1_serviceability \</b></div><div \
class=""> -:hotspot_tier2_runtime_platform_agnostic</div></div><div \
class="">…</div><div class=""><br class=""></div><div class=""><div \
class="">hotspot_tier3_runtime = \</div><div class=""> runtime/ \</div><div \
class=""> serviceability/ \</div><div class=""> -runtime/containers/ \
\</div><div class=""> <b class="">-:tier1_runtime \</b></div><div class=""><b \
class=""> -:tier1_serviceability \</b></div><div \
class=""> -:hotspot_tier2_runtime_platform_agnostic \</div><div class=""><b \
class=""> -:hotspot_tier2_runtime</b></div></div><div class=""><br \
class=""></div><div class="">… </div><div class=""><br class=""></div><div \
class="">So once test is excluded from tier1 it automatically is executed as a part \
of tier 2/3. While these tiers doesn't execute tests from tier1. The other teams \
organize their testing slightly different. </div><div class="">However the whole \
idea is that test excluded from tier1 should be executed in latest tiers and test \
should not re-executed in fast tiers once it was done already. However during testing \
like PIT </div><div class="">we might just run whole hotspot_[component] team \
even some subset was run always in CI.</div><div class=""><br class=""></div><div \
class=""><br class=""></div><div class="">Leonid</div></div></div><br \
class=""><blockquote type="cite" class=""><div class=""><div class="">thanks,<br \
class=""><br class="">Chris<br class=""><br class="">On 3/26/18 4:32 PM, Leonid \
Mesnik wrote:<br class=""><blockquote type="cite" class="">Hi<br class=""><br \
class="">There is no strict rules and time budgets for hotspot tiers. It is assumed \
that faster tests should be executed earlier tiers. The order and total impact of of \
these test depends on how they are executed.<br class=""><br class="">Here is time of \
execution of tier1 groups with and without these excluded tests:<br class="">The \
whole :tier1 time reduced from 40 to 30 minutes (on dedicated HW with 32 core)<br \
class="">Time for :tier1_serviceability reduced from 15 to 6 min (in HS CI)<br \
class="">Time for :tier1_gc_1 reduced from 12 to 8-10 min (in HS CI)<br \
class=""><br class="">The benefits of exclusion of gc/g1/ihop/TestIHOPErgo.java are \
not so significant. However it is the only one GC stress test executed in tier1 while \
intention is to don't run stress testing in tier1.<br class=""><br class="">Tests<br \
class="">serviceability/sa/ClhsdbScanOops.java<br \
class="">serviceability/sa/TestHeapDumpForLargeArray.java<br class="">are now \
hotspot_tier3_runtime which includes all hotspot_serviceability tests which are not a \
part of tier1. They are executed as a part of tier3 now as well as most of \
serviceability tests.<br class=""><br class="">Test gc/g1/ihop/TestIHOPErgo.java in \
now in group :hotspot_gc only and also it is marked as "stress" test. So it is \
executed with all other GC stress tests<br class=""><br class="">Leonid<br \
class=""><br class=""><blockquote type="cite" class="">On Mar 25, 2018, at 3:48 PM, \
David Holmes <<a href="mailto:david.holmes@oracle.com" \
class="">david.holmes@oracle.com</a>> wrote:<br class=""><br class="">Hi \
Leonid,<br class=""><br class="">On 24/03/2018 9:31 AM, Leonid Mesnik wrote:<br \
class=""><blockquote type="cite" class="">Hi<br class="">Could you please review \
following fix which exclude following tests from tier1 testing:<br \
class="">serviceability/sa/ClhsdbScanOops.java<br \
class="">serviceability/sa/TestHeapDumpForLargeArray.java<br \
class="">gc/g1/ihop/TestIHOPErgo.java<br class="">Each of them takes more then 5 \
minutes to complete and significantly increase overall time to complete tier1.<br \
class=""></blockquote>I'd need to see a much more detailed analysis of all the tests \
run, the order they run and the execution times to ascertain what impact this \
actually has on overall test execution time.<br class=""><br class="">But assuming 5 \
minutes is too long for tier1, this seems okay. But the tests must still be run in \
some other tier, and I'm not clear where that would be now? I would expect them to \
move to tier 3 perhaps, depending on what the time criteria for tier 3 is.<br \
class=""><br class="">Thanks,<br class="">David<br class=""><br class=""><blockquote \
type="cite" class="">Please let me know if there are any reasons to run these tests \
in tier1 despite on their execution time.<br class="">webrev:<a \
href="http://cr.openjdk.java.net/~lmesnik/8200187/webrev.00/" \
class="">http://cr.openjdk.java.net/~lmesnik/8200187/webrev.00/</a><br class="">bug: \
<a href="https://bugs.openjdk.java.net/browse/JDK-8200187" \
class="">https://bugs.openjdk.java.net/browse/JDK-8200187</a><br class="">Leonid<br \
class=""></blockquote></blockquote></blockquote><br class=""><br class=""><br \
class=""></div></div></blockquote></div><br class=""></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic