[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: system profilers and incomplete stacks
From: Volker Simonis <volker.simonis () gmail ! com>
Date: 2014-10-22 20:29:39
Message-ID: CA+3eh10yDrFHqNbNSuicvO4VZERfFA7yA4AqhTHr6qXDGS+CXQ () mail ! gmail ! com
[Download RAW message or body]
Hi,
I think the main problem here is not only that compiled code uses the
frame pointer as a general purpose register, but the fact that even
the VM itself currently can not reliably take a stack trace of a
thread at any arbitrary PC. An external sampling profiler like perf
can interrupt the VM at any place (i.e. while the VM sets up a new
frame or while inside an adapter). So while keeping the frame pointer
alive will definitely help to improve the situation, I'm not it will
help in every situation.
Regards,
Volker
On 10/22/14, serguei.spitsyn@oracle.com <serguei.spitsyn@oracle.com> wrote:
> Hi Brendan,
>
> We are working with the Solaris on prototyping an assisted approach to
> resolve the issue.
> In this approach the jhelper.d (the dtrace jstack action provider on VM
> side) is assisting
> the DTrace framework to do the stack walking cooperatively.
>
> I've assigned the bug JDK-6617153 to myself and will post updates about
> the progress.
> Please, let us know if you have any questions.
>
> Thanks,
> Serguei
>
>
> On 10/21/14 5:10 PM, Brendan Gregg wrote:
> > G'Day,
> >
> > I checked the JDK 9 early access releases, but didn't see anything for
> > JDK-6276264. I've also since learned that Twitter has an OpenJDK fork
> > with frame pointers disabled, for the same purpose: stack profiling
> > (using Linux perf_events). Might this be worked on for JDK 9? I can
> > help test. thanks,
> >
> > Brendan
> >
> > On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg
> > <brendan.d.gregg@gmail.com> wrote:
> > > G'Day Serguei,
> > >
> > > On Mon, Jun 16, 2014 at 10:45 PM, serguei.spitsyn@oracle.com
> > > <serguei.spitsyn@oracle.com> wrote:
> > > > Hi Brendan,
> > > >
> > > > We are aware of these issues and work with the Solaris team to fix them
> > > > in
> > > > JDK 9.
> > > > One is the frame pointer is used by the server compiler as a general
> > > > purpose register on intel.
> > > > Another is about the virtual (or inlined) frames.
> > > >
> > > > There are a couple of related bugs:
> > > > https://bugs.openjdk.java.net/browse/JDK-6617153
> > > > https://bugs.openjdk.java.net/browse/JDK-6276264
> > > >
> > > > There can be more issues filed on this.
> > >
> > > Ah, thanks, it's JDK-6276264.
> > >
> > > As Tom Rodriguez said at the time (2005): "The server VM uses the frame
> > > pointer as an allocatable register and there's no way to turn that off."
> > > I
> > > was really hoping there was a way to turn that off, like
> > > -fno-omit-frame-pointer.
> > >
> > > This also means DTrace jstack() has never worked fully. For the
> > > applications
> > > I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad
> > > in
> > > 2005. I've been getting more mileage today from Java profilers.
> > >
> > > > Please, note, that the jstack action is not implemented on Linux yet.
> > >
> > > Linux doesn't have DTrace jstack(), no, but its perf_events does has
> > > support
> > > for loading an auxiliary file of symbols, which can created via a Java
> > > agent
> > > for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But
> > > that
> > > hasn't been working fully for the same reason - incomplete stacks.
> > >
> > > Brendan
> > >
> > > > Thanks,
> > > > Serguei
> > > >
> > > >
> > > >
> > > > On 6/16/14 5:14 PM, Brendan Gregg wrote:
> > > >
> > > > Thanks but no, I'm aware of that bug and workarounds (I'm using the
> > > > LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
> > > > isn't mentioned in the bug comments, but probably should be). That bug
> > > > is
> > > > about missing symbols, but the stacks shown in that bug still go all the
> > > > way
> > > > to thread_start. My stacks often don't.
> > > >
> > > > For simple programs, the stacks are complete. But something complex
> > > > (eg,
> > > > vert.x with event loops), and the stacks are often incomplete, one
> > > > frame
> > > > only. Very much like what I see with -fomit-frame-pointer, although this
> > > > is
> > > > hotspot, not gcc. Such incomplete stacks are seen using either DTrace
> > > > or
> > > > perf_events.
> > > >
> > > > It was suggested to me to email the hotspot developers, because this
> > > > may
> > > > well be a hotspot optimization they are familiar with. It may also be
> > > > something really obvious, like that the JVM breaks native stacks due to
> > > > optimized frames / green threads / etc, and there is absolutely no way
> > > > around it (no way to disable it). If that's true, it may also mean that
> > > > the
> > > > DTrace jstack() action has always had this issue. I'm still reading the
> > > > source...
> > > >
> > > > Brendan
> > > >
> > > >
> > > >
> > > > On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
> > > > <staffan.larsen@oracle.com> wrote:
> > > > > I think this is the bug you are looking at:
> > > > > https://bugs.openjdk.java.net/browse/JDK-7187999, but I'll defer to
> > > > > someone
> > > > > else to confirm.
> > > > >
> > > > > /Staffan
> > > > >
> > > > >
> > > > > On 16 jun 2014, at 12:47, Roland Westrelin
> > > > > <roland.westrelin@oracle.com>
> > > > > wrote:
> > > > >
> > > > > Forwarding to serviceability alias where this question belongs I
> > > > > think.
> > > > >
> > > > > Begin forwarded message:
> > > > >
> > > > > From: Brendan Gregg <brendan.d.gregg@gmail.com>
> > > > > Subject: system profilers and incomplete stacks
> > > > > Date: June 12, 2014 at 7:15:54 PM GMT+2
> > > > > To: hotspot-compiler-dev@openjdk.java.net
> > > > >
> > > > > G'Day,
> > > > >
> > > > > Is there a way to run hotspot so that a system profiler (eg, DTrace,
> > > > > or
> > > > > Linux perf_events) can measure complete stacks? I often get
> > > > > incomplete,
> > > > > partial stacks, with one or a few frames only. I'm not worried about
> > > > > symbols
> > > > > right now, what I'd like is to walk stacks all the way down to thread
> > > > > start.
> > > > >
> > > > > I've been browsing the hotspot code, but haven't found out how yet. I
> > > > > suspect it's related to Java optimized frames, and has ditched the
> > > > > frame
> > > > > pointer. I was looking for an equivalent -fno-omit-frame-pointer
> > > > > option.
> > > > >
> > > > > Here's an example:
> > > > >
> > > > > # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] =
> > > > > count(); }'
> > > > > [...]
> > > > > org/mozilla/javascript/
> > > > >
> > > > > ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
> > > > > 0x884acce8200002da
> > > > > 1
> > > > >
> > > > >
> > > > > sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
> > > > > 0xffffffff20007f4b
> > > > > 1
> > > > >
> > > > >
> > > > > org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Lj \
> > > > > ava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
> > > > > 0xa20000041
> > > > > 1
> > > > > [...]
> > > > >
> > > > > I see similar incomplete stacks with Linux perf_events. Oracle JDKs
> > > > > from
> > > > > 6 to 8, and OpenJDK.
> > > > >
> > > > > thanks,
> > > > >
> > > > > Brendan
> > > > > --
> > > > > http://www.brendangregg.com
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > http://www.brendangregg.com
> > > >
> > > >
> > >
> > >
> > > --
> > > http://www.brendangregg.com
>
>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic