[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: system profilers and incomplete stacks
From: "serguei.spitsyn () oracle ! com" <serguei ! spitsyn () oracle ! com>
Date: 2014-06-17 7:11:18
Message-ID: 539FEA16.2050607 () oracle ! com
[Download RAW message or body]
Brendan,
Thank you for the details, especially, about the perf_events!
I was not aware about the issue on Linux.
I agree, the Solaris jstack issue was not that bad back in 2005.
It needs to be fixed cooperatively with the OS.
We have an idea how to fix it, this work is at the prototyping stage now.
Thanks,
Serguei
On 6/16/14 11:52 PM, Brendan Gregg wrote:
> G'Day Serguei,
>
> On Mon, Jun 16, 2014 at 10:45 PM, serguei.spitsyn@oracle.com
> <mailto:serguei.spitsyn@oracle.com> <serguei.spitsyn@oracle.com
> <mailto:serguei.spitsyn@oracle.com>> wrote:
>
> Hi Brendan,
>
> We are aware of these issues and work with the Solaris team to fix
> them in JDK 9.
> One is the frame pointer is used by the server compiler as a
> general purpose register on intel.
> Another is about the virtual (or inlined) frames.
>
> There are a couple of related bugs:
> https://bugs.openjdk.java.net/browse/JDK-6617153
> https://bugs.openjdk.java.net/browse/JDK-6276264
>
> There can be more issues filed on this.
>
>
> Ah, thanks, it's JDK-6276264.
>
> As Tom Rodriguez said at the time (2005): "The server VM uses the
> frame pointer as an allocatable register and there's no way to turn
> that off." I was really hoping there was a way to turn that off, like
> -fno-omit-frame-pointer.
>
> This also means DTrace jstack() has never worked fully. For the
> applications I tried it on, 50% of stacks were incomplete. Perhaps it
> wasn't that bad in 2005. I've been getting more mileage today from
> Java profilers.
>
> Please, note, that the jstack action is not implemented on Linux yet.
>
>
> Linux doesn't have DTrace jstack(), no, but its perf_events does has
> support for loading an auxiliary file of symbols, which can created
> via a Java agent for that purpose (eg,
> https://github.com/jrudolph/perf-map-agent). But that hasn't been
> working fully for the same reason - incomplete stacks.
>
> Brendan
>
>
> Thanks,
> Serguei
>
>
>
> On 6/16/14 5:14 PM, Brendan Gregg wrote:
> > Thanks but no, I'm aware of that bug and workarounds (I'm using
> > the LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so
> > workaround, which isn't mentioned in the bug comments, but
> > probably should be). That bug is about missing symbols, but the
> > stacks shown in that bug still go all the way to thread_start. My
> > stacks often don't.
> >
> > For simple programs, the stacks are complete. But something
> > complex (eg, vert.x with event loops), and the stacks are often
> > incomplete, one frame only. Very much like what I see with
> > -fomit-frame-pointer, although this is hotspot, not gcc. Such
> > incomplete stacks are seen using either DTrace or perf_events.
> >
> > It was suggested to me to email the hotspot developers, because
> > this may well be a hotspot optimization they are familiar with.
> > It may also be something really obvious, like that the JVM breaks
> > native stacks due to optimized frames / green threads / etc, and
> > there is absolutely no way around it (no way to disable it). If
> > that's true, it may also mean that the DTrace jstack() action has
> > always had this issue. I'm still reading the source...
> >
> > Brendan
> >
> >
> >
> > On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
> > <staffan.larsen@oracle.com <mailto:staffan.larsen@oracle.com>> wrote:
> >
> > I think this is the bug you are looking at:
> > https://bugs.openjdk.java.net/browse/JDK-7187999, but I'll
> > defer to someone else to confirm.
> >
> > /Staffan
> >
> >
> > On 16 jun 2014, at 12:47, Roland Westrelin
> > <roland.westrelin@oracle.com
> > <mailto:roland.westrelin@oracle.com>> wrote:
> >
> > > Forwarding to serviceability alias where this question
> > > belongs I think.
> > >
> > > Begin forwarded message:
> > >
> > > > *From: *Brendan Gregg <brendan.d.gregg@gmail.com
> > > > <mailto:brendan.d.gregg@gmail.com>>
> > > > *Subject: **system profilers and incomplete stacks*
> > > > *Date: *June 12, 2014 at 7:15:54 PM GMT+2
> > > > *To: *hotspot-compiler-dev@openjdk.java.net
> > > > <mailto:hotspot-compiler-dev@openjdk.java.net>
> > > >
> > > > G'Day,
> > > >
> > > > Is there a way to run hotspot so that a system profiler
> > > > (eg, DTrace, or Linux perf_events) can measure complete
> > > > stacks? I often get incomplete, partial stacks, with one or
> > > > a few frames only. I'm not worried about symbols right now,
> > > > what I'd like is to walk stacks all the way down to thread
> > > > start.
> > > >
> > > > I've been browsing the hotspot code, but haven't found out
> > > > how yet. I suspect it's related to Java optimized frames,
> > > > and has ditched the frame pointer. I was looking for an
> > > > equivalent -fno-omit-frame-pointer option.
> > > >
> > > > Here's an example:
> > > >
> > > > # dtrace -n 'profile-99 /execname == "java"/ {
> > > > @[jstack(100, 8000)] = count(); }'
> > > > [...]
> > > > org/mozilla/javascript/
> > > > ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
> > > > 0x884acce8200002da
> > > > 1
> > > >
> > > > sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
> > > > 0xffffffff20007f4b
> > > > 1
> > > >
> > > > org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljav \
> > > > a/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
> > > > 0xa20000041
> > > > 1
> > > > [...]
> > > >
> > > > I see similar incomplete stacks with Linux perf_events.
> > > > Oracle JDKs from 6 to 8, and OpenJDK.
> > > >
> > > > thanks,
> > > >
> > > > Brendan
> > > > --
> > > > http://www.brendangregg.com <http://www.brendangregg.com/>
> > >
> >
> >
> >
> >
> > --
> > http://www.brendangregg.com
>
>
>
>
> --
> http://www.brendangregg.com
[Attachment #3 (text/html)]
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Brendan,<br>
<br>
Thank you for the details, especially, about the perf_events!<br>
I was not aware about the issue on Linux.<br>
<br>
I agree, the Solaris jstack issue was not that bad back in 2005.<br>
It needs to be fixed cooperatively with the OS.<br>
We have an idea how to fix it, this work is at the prototyping
stage now.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
On 6/16/14 11:52 PM, Brendan Gregg wrote:<br>
</div>
<blockquote
cite="mid:CAE40pddqPY7e70maFqZm+41qCgw6P4Ck7N4u8UcTp4PQOuuh4A@mail.gmail.com"
type="cite">
<div dir="ltr">G'Day Serguei,<br>
<br>
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Jun 16, 2014 at 10:45 PM, <a
moz-do-not-send="true"
href="mailto:serguei.spitsyn@oracle.com">serguei.spitsyn@oracle.com</a>
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:serguei.spitsyn@oracle.com" \
target="_blank">serguei.spitsyn@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi Brendan,<br>
<br>
We are aware of these issues and work with the Solaris
team to fix them in JDK 9.<br>
One is the frame pointer is used by the server
compiler as a general purpose register on intel.<br>
Another is about the virtual (or inlined) frames.<br>
<br>
There are a couple of related bugs:<br>
<a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/browse/JDK-6617153"
target="_blank">https://bugs.openjdk.java.net/browse/JDK-6617153</a><br>
<a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/browse/JDK-6276264"
target="_blank">https://bugs.openjdk.java.net/browse/JDK-6276264</a><br>
<br>
There can be more issues filed on this.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Ah, thanks, it's JDK-6276264.<br>
<br>
</div>
<div>As Tom Rodriguez said at the time (2005): "The server
VM uses the frame pointer as an allocatable register and
there's no way to turn that off." I was really hoping
there was a way to turn that off, like
-fno-omit-frame-pointer.<br>
<br>
</div>
<div>This also means DTrace jstack() has never worked fully.
For the applications I tried it on, 50% of stacks were
incomplete. Perhaps it wasn't that bad in 2005. I've been
getting more mileage today from Java profilers.<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div> Please, note, that the jstack action is not
implemented on Linux yet.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Linux doesn't have DTrace jstack(), no, but its
perf_events does has support for loading an auxiliary file
of symbols, which can created via a Java agent for that
purpose (eg, <a moz-do-not-send="true"
href="https://github.com/jrudolph/perf-map-agent">https://github.com/jrudolph/perf-map-agent</a>).
But that hasn't been working fully for the same reason -
incomplete stacks.<br>
<br>
Brendan<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div> <br>
Thanks,<br>
Serguei
<div>
<div class="h5"><br>
<br>
<br>
On 6/16/14 5:14 PM, Brendan Gregg wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Thanks but no, I'm aware of that bug and
workarounds (I'm using the
LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so
workaround, which isn't mentioned in the bug
comments, but probably should be). That bug
is about missing symbols, but the stacks
shown in that bug still go all the way to
thread_start. My stacks often don't.<br>
<br>
</div>
For simple programs, the stacks are complete.
But something complex (eg, vert.x with event
loops), and the stacks are often incomplete,
one frame only. Very much like what I see with
-fomit-frame-pointer, although this is
hotspot, not gcc. Such incomplete stacks are
seen using either DTrace or perf_events.<br>
<br>
</div>
It was suggested to me to email the hotspot
developers, because this may well be a hotspot
optimization they are familiar with. It may also
be something really obvious, like that the JVM
breaks native stacks due to optimized frames /
green threads / etc, and there is absolutely no
way around it (no way to disable it). If that's
true, it may also mean that the DTrace jstack()
action has always had this issue. I'm still
reading the source...<br>
<div><br>
Brendan<br>
<br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Jun 16, 2014 at
4:04 AM, Staffan Larsen <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:staffan.larsen@oracle.com"
\
target="_blank">staffan.larsen@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">I think
this is the bug you are looking at: <a
moz-do-not-send="true"
\
href="https://bugs.openjdk.java.net/browse/JDK-7187999"
\
target="_blank">https://bugs.openjdk.java.net/browse/JDK-7187999</a>,
but I'll defer to someone else to confirm.<span><font
color="#888888">
<div> <br>
</div>
<div>/Staffan</div>
</font></span>
<div>
<div>
<div><br>
</div>
<div><br>
<div>
<div>On 16 jun 2014, at 12:47,
Roland Westrelin <<a
moz-do-not-send="true"
href="mailto:roland.westrelin@oracle.com"
\
target="_blank">roland.westrelin@oracle.com</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div
style="word-wrap:break-word">Forwarding
to serviceability alias where
this question belongs I think.
<div><br>
</div>
<div>Begin forwarded message:<br>
<div><br>
<blockquote type="cite">
<div style="margin:0px"><span
style="font-family:Helvetica"><b>From: </b></span><span
\
style="font-family:'Helvetica'">Brendan
Gregg <<a
moz-do-not-send="true"
href="mailto:brendan.d.gregg@gmail.com" \
target="_blank">brendan.d.gregg@gmail.com</a>><br> </span></div>
<div style="margin:0px"><span
style="font-family:Helvetica"><b>Subject: </b></span><span
\
style="font-family:'Helvetica'"><b>system
profilers and
incomplete stacks</b><br>
</span></div>
<div style="margin:0px"><span
style="font-family:Helvetica"><b>Date: </b></span><span
\
style="font-family:'Helvetica'">June
12, 2014 at 7:15:54
PM GMT+2<br>
</span></div>
<div style="margin:0px"><span
style="font-family:Helvetica"><b>To: </b></span><span
\
style="font-family:'Helvetica'"><a moz-do-not-send="true"
\
href="mailto:hotspot-compiler-dev@openjdk.java.net"
\
target="_blank">hotspot-compiler-dev@openjdk.java.net</a><br> </span></div>
<br>
<div>
<div dir="ltr">
<div>G'Day,<br>
<br>
</div>
Is there a way to
run hotspot so that
a system profiler
(eg, DTrace, or
Linux perf_events)
can measure complete
stacks? I often get
incomplete, partial
stacks, with one or
a few frames only.
I'm not worried
about symbols right
now, what I'd like
is to walk stacks
all the way down to
thread start.<br>
<br>
I've been browsing
the hotspot code,
but haven't found
out how yet. I
suspect it's related
to Java optimized
frames, and has
ditched the frame
pointer. I was
looking for an
equivalent
-fno-omit-frame-pointer
option.<br
clear="all">
<div>
<div><br>
</div>
<div>Here's an
example:<br>
<br>
# dtrace -n
'profile-99
/execname ==
"java"/ {
@[jstack(100,
8000)] =
count(); }'<br>
[...]<br>
org/mozilla/javascript/
\
<div>ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*<br>
0x884acce8200002da<br>
\
1<br>
<br>
sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*<br>
0xffffffff20007f4b<br>
\
1<br>
<br>
org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/O \
bject;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*<br>
0xa20000041<br>
\
1<br>
[...]<br>
<br>
</div>
<div>I see
similar
incomplete
stacks with
Linux
perf_events.
Oracle JDKs
from 6 to 8,
and OpenJDK.<br>
</div>
<br>
</div>
<div>thanks,<br>
<br>
</div>
<div>Brendan<br>
</div>
<div>-- <br>
<div dir="ltr"><a
moz-do-not-send="true" href="http://www.brendangregg.com/"
\
target="_blank">http://www.brendangregg.com</a><br> </div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div dir="ltr"><a moz-do-not-send="true"
href="http://www.brendangregg.com"
target="_blank">http://www.brendangregg.com</a><br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div dir="ltr"><a moz-do-not-send="true"
href="http://www.brendangregg.com" \
target="_blank">http://www.brendangregg.com</a><br> </div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic