[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openjdk-serviceability-dev
Subject:    Re: system profilers and incomplete stacks
From:       Brendan Gregg <brendan.d.gregg () gmail ! com>
Date:       2014-10-22 0:10:31
Message-ID: CAE40pdfOSGu1g=iVODaMteYtM3_XTNEJsq2JkcK4rw3JAVF4zg () mail ! gmail ! com
[Download RAW message or body]

G'Day,

I checked the JDK 9 early access releases, but didn't see anything for
JDK-6276264. I've also since learned that Twitter has an OpenJDK fork
with frame pointers disabled, for the same purpose: stack profiling
(using Linux perf_events). Might this be worked on for JDK 9? I can
help test. thanks,

Brendan

On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg
<brendan.d.gregg@gmail.com> wrote:
> G'Day Serguei,
> 
> On Mon, Jun 16, 2014 at 10:45 PM, serguei.spitsyn@oracle.com
> <serguei.spitsyn@oracle.com> wrote:
> > 
> > Hi Brendan,
> > 
> > We are aware of these issues and work with the Solaris team to fix them in
> > JDK 9.
> > One is the frame pointer is used by the server compiler as a general
> > purpose register on intel.
> > Another is about the virtual (or inlined) frames.
> > 
> > There are a couple of related bugs:
> > https://bugs.openjdk.java.net/browse/JDK-6617153
> > https://bugs.openjdk.java.net/browse/JDK-6276264
> > 
> > There can be more issues filed on this.
> 
> 
> Ah, thanks, it's JDK-6276264.
> 
> As Tom Rodriguez said at the time (2005): "The server VM uses the frame
> pointer as an allocatable register and there's no way to turn that off." I
> was really hoping there was a way to turn that off, like
> -fno-omit-frame-pointer.
> 
> This also means DTrace jstack() has never worked fully. For the applications
> I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in
> 2005. I've been getting more mileage today from Java profilers.
> 
> > Please, note, that the jstack action is not implemented on Linux yet.
> 
> 
> Linux doesn't have DTrace jstack(), no, but its perf_events does has support
> for loading an auxiliary file of symbols, which can created via a Java agent
> for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that
> hasn't been working fully for the same reason - incomplete stacks.
> 
> Brendan
> 
> > 
> > Thanks,
> > Serguei
> > 
> > 
> > 
> > On 6/16/14 5:14 PM, Brendan Gregg wrote:
> > 
> > Thanks but no, I'm aware of that bug and workarounds (I'm using the
> > LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
> > isn't mentioned in the bug comments, but probably should be). That bug is
> > about missing symbols, but the stacks shown in that bug still go all the way
> > to thread_start. My stacks often don't.
> > 
> > For simple programs, the stacks are complete. But something complex (eg,
> > vert.x with event loops), and the stacks are often incomplete, one frame
> > only. Very much like what I see with -fomit-frame-pointer, although this is
> > hotspot, not gcc. Such incomplete stacks are seen using either DTrace or
> > perf_events.
> > 
> > It was suggested to me to email the hotspot developers, because this may
> > well be a hotspot optimization they are familiar with. It may also be
> > something really obvious, like that the JVM breaks native stacks due to
> > optimized frames / green threads / etc, and there is absolutely no way
> > around it (no way to disable it). If that's true, it may also mean that the
> > DTrace jstack() action has always had this issue. I'm still reading the
> > source...
> > 
> > Brendan
> > 
> > 
> > 
> > On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
> > <staffan.larsen@oracle.com> wrote:
> > > 
> > > I think this is the bug you are looking at:
> > > https://bugs.openjdk.java.net/browse/JDK-7187999, but I'll defer to someone
> > > else to confirm.
> > > 
> > > /Staffan
> > > 
> > > 
> > > On 16 jun 2014, at 12:47, Roland Westrelin <roland.westrelin@oracle.com>
> > > wrote:
> > > 
> > > Forwarding to serviceability alias where this question belongs I think.
> > > 
> > > Begin forwarded message:
> > > 
> > > From: Brendan Gregg <brendan.d.gregg@gmail.com>
> > > Subject: system profilers and incomplete stacks
> > > Date: June 12, 2014 at 7:15:54 PM GMT+2
> > > To: hotspot-compiler-dev@openjdk.java.net
> > > 
> > > G'Day,
> > > 
> > > Is there a way to run hotspot so that a system profiler (eg, DTrace, or
> > > Linux perf_events) can measure complete stacks? I often get incomplete,
> > > partial stacks, with one or a few frames only. I'm not worried about symbols
> > > right now, what I'd like is to walk stacks all the way down to thread start.
> > > 
> > > I've been browsing the hotspot code, but haven't found out how yet. I
> > > suspect it's related to Java optimized frames, and has ditched the frame
> > > pointer. I was looking for an equivalent -fno-omit-frame-pointer option.
> > > 
> > > Here's an example:
> > > 
> > > # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] =
> > > count(); }'
> > > [...]
> > > org/mozilla/javascript/
> > > 
> > > ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
> > >  0x884acce8200002da
> > > 1
> > > 
> > > sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
> > > 0xffffffff20007f4b
> > > 1
> > > 
> > > 
> > > org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/ \
> > > lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
> > >  0xa20000041
> > > 1
> > > [...]
> > > 
> > > I see similar incomplete stacks with Linux perf_events. Oracle JDKs from
> > > 6 to 8, and OpenJDK.
> > > 
> > > thanks,
> > > 
> > > Brendan
> > > --
> > > http://www.brendangregg.com
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > --
> > http://www.brendangregg.com
> > 
> > 
> 
> 
> 
> --
> http://www.brendangregg.com


[prev in list] [next in list] [prev in thread] [next in thread]