[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-hotspot-compiler-dev
Subject: Re: compiler monitorig/tuning
From: charlie hunt <charlie.hunt () oracle ! com>
Date: 2016-06-27 19:30:19
Message-ID: E6DE595D-CE20-4C39-A816-FC348FDEDC22 () oracle ! com
[Download RAW message or body]
Wrt …
> Might be useful to make queue lengths available by an MBean or whatever so that \
> service owners can integrate those metrics with existing monitoring systems, and \
> can understand when it might make sense to add more JIT threads to keep up with the \
> enqueue rate.
Contributions are welcome. :-]
charlie
> On Jun 22, 2016, at 4:56 PM, Brian Toal <brian.toal@gmail.com> wrote:
>
> Thanks Vladmir,
>
> Will look into -XX:+PrintTieredEvents. Might be useful to make queue lengths \
> available by an MBean or whatever so that service owners can integrate those \
> metrics with existing monitoring systems, and can understand when it might make \
> sense to add more JIT threads to keep up with the enqueue rate.
> We recently upgraded from u66 -> u72 and around the same time we've noticed that \
> 1-2 JVM's out of ~30-40 that are servicing the application tier run significantly \
> slower and as a result take on more load, until the host is completely over \
> saturated, exacerbating the problem worse. The workload is uniform across the \
> hosts. We are looking at addressing the feedback system to ensure the requests \
> are throttled, how ever we are trying to understand if some change in JIT behavior \
> is causing the system to take longer to warm up.
> Looking at stacks from the active threads on the load ramp up, there isn't one \
> hotspot that would explain the increase in load due to a feature regression, but \
> rather what we see are arbitrary code paths. Could be due to the host being \
> completely saturated and no real work progressing, or could be due to the code \
> still running interpreted.
> - Brian
>
>
>
>
>
> On Tue, Jun 21, 2016 at 9:01 PM, Vladimir Kozlov <vladimir.kozlov@oracle.com \
> <mailto:vladimir.kozlov@oracle.com>> wrote: Hi Brian
>
> Could you explain what problems you are observing? Which JDK 8 update version you \
> are testing?
> On 6/16/16 11:00 AM, Brian Toal wrote:
> Good day,
>
> - How do I monitor the length of the compiler queues (client and server)? I want \
> to understand the backlog (if any) to determine if the application could benefit \
> from more compilation threads.
> There is -XX:+PrintTieredEvents flag which outputs on display queues sizes for each \
> compilation request (a lot of output).
>
> - I remember in the past being able to get detailed compiler statistics from a \
> mbean or something where the metric was prefixed with java.ci <http://java.ci/> \
> <http://java.ci <http://java.ci/>> and it contained execution count, total time, \
> thread count exec for the client and server compilers. Where can I find this data \
> in realtime for monitoring purposes? If I know the total time and thread count I \
> suppose I can work out if the utilization of the compilation threads.
> May be it was jvmstat monitoring utility (now jstat). But it does not collect \
> queues sizes.
>
> - What other compilation metrics should I be paying close attention to?
>
> Very detailed compilation output file is created (on JVM exit) with \
> -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation. You can build this output \
> parser tool from sources in \
> http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/file/ba5ca8b6e48f/src/share/tools/LogCompilation \
> <http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/file/ba5ca8b6e48f/src/share/tools/LogCompilation> \
> (note it is jdk 9, for 8 update build from clone of \
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/ \
> <http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/>).
>
> - Other than tiered compilation becoming the default in Java 8, where there any \
> significant changes that would have slowed down compilation time? We've had \
> -XX:-TieredCompilation set in Java 7 and 8 and haven't touched any other JIT \
> related config. Was there some other setting default that changed which we would \
> have needed to change when going fro 7 to 8? Looking at the various compiler \
> parameters that I know about via PrintFlagsFinal from 7 and 8, there doesn't seem \
> to be any change other than -XX:-TieredCompilation.
> There were a lot of changes which could affected JIT compilation speed.
>
>
> Regards,
> - Brian
>
> Regards,
> Vladimir
>
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space;" class="">Wrt …<div class=""><br \
class=""><blockquote type="cite" class="">Might be useful to make queue lengths \
available by an MBean or whatever so that service owners can integrate those metrics \
with existing monitoring systems, and can understand when it might make sense to add \
more JIT threads to keep up with the enqueue rate.</blockquote><div class=""><br \
class=""></div><div class="">Contributions are welcome. :-]</div><div class=""><br \
class=""></div><div class="">charlie</div><br class=""><div><blockquote type="cite" \
class=""><div class="">On Jun 22, 2016, at 4:56 PM, Brian Toal <<a \
href="mailto:brian.toal@gmail.com" class="">brian.toal@gmail.com</a>> \
wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" \
class=""><div class="">Thanks Vladmir,</div><div class=""><br class=""></div><div \
class="">Will look into -XX:+PrintTieredEvents. Might be useful to make queue \
lengths available by an MBean or whatever so that service owners can integrate those \
metrics with existing monitoring systems, and can understand when it might make sense \
to add more JIT threads to keep up with the enqueue rate.</div><div class=""><br \
class=""></div>We recently upgraded from u66 -> u72 and around the same time we've \
noticed that 1-2 JVM's out of ~30-40 that are servicing the application tier run \
significantly slower and as a result take on more load, until the host is completely \
over saturated, exacerbating the problem worse. The workload is uniform across \
the hosts. We are looking at addressing the feedback system to ensure the \
requests are throttled, how ever we are trying to understand if some change in JIT \
behavior is causing the system to take longer to warm up.<div class=""><div \
class=""><br class=""></div><div class="">Looking at stacks from the active threads \
on the load ramp up, there isn't one hotspot that would explain the increase in load \
due to a feature regression, but rather what we see are arbitrary code paths. \
Could be due to the host being completely saturated and no real work progressing, or \
could be due to the code still running interpreted.</div><div class=""><br \
class=""></div><div class="">- Brian</div><div class=""><br class=""></div><div \
class=""><br class=""><div class=""><div class=""><br class=""></div><div \
class=""><br class=""></div></div></div></div></div><div class="gmail_extra"><br \
class=""><div class="gmail_quote">On Tue, Jun 21, 2016 at 9:01 PM, Vladimir Kozlov \
<span dir="ltr" class=""><<a href="mailto:vladimir.kozlov@oracle.com" \
target="_blank" class="">vladimir.kozlov@oracle.com</a>></span> wrote:<br \
class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex">Hi Brian<br class=""> <br class="">
Could you explain what problems you are observing? Which JDK 8 update version you are \
testing?<span class=""><br class=""> <br class="">
On 6/16/16 11:00 AM, Brian Toal wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> Good day,<br class="">
<br class="">
- How do I monitor the length of the compiler queues (client and server)? I \
want to understand the backlog (if any) to<br class=""> determine if the application \
could benefit from more compilation threads.<br class=""> </blockquote>
<br class=""></span>
There is -XX:+PrintTieredEvents flag which outputs on display queues sizes for each \
compilation request (a lot of output).<br class=""> <br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span class=""> <br class="">
- I remember in the past being able to get detailed compiler statistics from a mbean \
or something where the metric was<br class=""></span> prefixed with <a \
href="http://java.ci/" rel="noreferrer" target="_blank" class="">java.ci</a> <<a \
href="http://java.ci/" rel="noreferrer" target="_blank" \
class="">http://java.ci</a>> and it contained execution count, total time, thread \
count exec for the client<span class=""><br class=""> and server compilers. \
Where can I find this data in realtime for monitoring purposes? If I know the \
total time and<br class=""> thread count I suppose I can work out if the utilization \
of the compilation threads.<br class=""> </span></blockquote>
<br class="">
May be it was jvmstat monitoring utility (now jstat). But it does not collect queues \
sizes.<span class=""><br class=""> <br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <br class="">
- What other compilation metrics should I be paying close attention to?<br class="">
</blockquote>
<br class=""></span>
Very detailed compilation output file is created (on JVM exit) with \
-XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation.<br class=""> You can build this \
output parser tool from sources in <a \
href="http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/file/ba5ca8b6e48f/src/share/tools/LogCompilation" \
rel="noreferrer" target="_blank" \
class="">http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/file/ba5ca8b6e48f/src/share/tools/LogCompilation</a> \
(note it is jdk 9, for 8 update build from clone of <a \
href="http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/" rel="noreferrer" \
target="_blank" class="">http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/</a>).<span \
class=""><br class=""> <br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <br class="">
- Other than tiered compilation becoming the default in Java 8, where there any \
significant changes that would have<br class=""> slowed down compilation time? \
We've had -XX:-TieredCompilation set in Java 7 and 8 and haven't touched any other \
JIT<br class=""> related config. Was there some other setting default that \
changed which we would have needed to change when going fro 7<br class=""> to \
8? Looking at the various compiler parameters that I know about via \
PrintFlagsFinal from 7 and 8, there doesn't seem<br class=""> to be any change other \
than -XX:-TieredCompilation.<br class=""> </blockquote>
<br class=""></span>
There were a lot of changes which could affected JIT compilation speed.<br class="">
<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <br class="">
Regards,<br class="">
- Brian<br class="">
</blockquote>
<br class="">
Regards,<br class="">
Vladimir<br class="">
</blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic