[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openjdk-serviceability-dev
Subject:    Re: PowerPC: core file option not available with serviceability tools
From:       Volker Simonis <volker.simonis () gmail ! com>
Date:       2014-11-14 19:23:59
Message-ID: CA+3eh10BSLGj8=p9m9vToKVyrT+A+Gu7d+z_2adms8GPp-hUAA () mail ! gmail ! com
[Download RAW message or body]

On Fri, Nov 14, 2014 at 6:00 PM, Maynard Johnson <maynardj@us.ibm.com> wrote:
> On 10/07/2014 08:35 AM, Maynard Johnson wrote:
> > On 10/07/2014 03:58 AM, Volker Simonis wrote:
> > > Hi Maynard,
> > > 
> > > I'm now back from JavaOne and can look at this issue. Could you please
> > > share your current implementation so I can reproduce your problem more
> > > easily.
> > See attachment.  The two patches in the attached tar file apply to a jdk9-dev \
> > snapshot from July.  I haven't even tried forward-porting to current upstream \
> > code, so I don't know how well they would apply.
> > > 
> > > By the way, you can find the ppc frame layout description of all the
> > > different frame types (native, interpreted, compiled) in
> > > /hotspot/src/cpu/ppc/vm/frame_ppc.hpp. The different frame::sender()
> > > implementations (in frame_x86.cpp, frame_ppc.cpp, ..) contain all the
> > Yes, I've been studying those files, but I freely admit I don't have a good grasp \
> > of the code yet.
> > > gory details about how to walk a frame. That's what you have to
> > > implement in Java in order to get a full stack trace from the
> > > serviceability tools. On x86 the frame pointer (i.e. the ebp register
> > > points to the last frame pointer while (frame pointer - 1) points to
> > > the return pc.
> > Ummm . . . Stack addresses grow downward, and I was under the impression 'return \
> > pc' was the word on the stack directly before the ebp, which would mean return pc \
> > is at 'frame pointer + 1'. Or am I off base here?  Nevertheless, my question \
> > below concerns the "pc", not the "return pc".  Perhaps I'm misunderstanding "pc" \
> > in this context; but even so, the x86 code still seems wrong: 
> > this.pc = raw_sp.getAddressAt(-1 * VM.getVM().getAddressSize());
> > 
> > If, in fact, 'this.pc' is supposed to represent 'return pc' in this context, then \
> > I would think the code should be: 
> > this.pc = raw_fp.getAddressAt(VM.getVM().getAddressSize());
> > 
> > I hope you can help set me on the right track.  As you can see, I'm lost in the \
> > weeds right now.  :-)
> Hi, Volker,
> I presume you've been too busy to help out with this, so I had another go at it, \
> and was able to make quite a bit of progress.  The main issue was that I was \
> unwittingly building my openJDK source with the "--with-jvm-interpreter=cpp" \
> configure option!  Once I discovered that and removed that option, the stack \
> started making sense, since my new ppc64 agent's reflection of the stack assumes \
> the use of the template interpreter (as does the x86 agent code which the ppc64 \
> stuff is based on). 
> So now I have jmap, jsadebugd, and jstack working for ppc64 on both big endian and \
> little endian -- almost. There are a couple issues yet with jstack that I'm stuck \
> on.  I will post the patches as an RFC and document those issues.  If you could \
> find time to review the patches and perhaps help out with the open issues, I would \
> appreciate it.  Thanks! 

Hi Maynard,

sorry, I was quite busy indeed.
I'm really happy that you've made some progress in the meantime and I
promise to look at your changes next week.

Thank you and best regards,
Volker

> -Maynard
> > 
> > -Maynard
> > > 
> > > Regards,
> > > Volker
> > > 
> > > 
> > > On Tue, Sep 30, 2014 at 12:42 AM, Maynard Johnson <maynardj@us.ibm.com> wrote:
> > > > On 07/09/2014 12:38 PM, Volker Simonis wrote:
> > > > > On Wed, Jul 9, 2014 at 3:45 PM, Maynard Johnson <maynardj@us.ibm.com> \
> > > > > wrote:
> > > > > > On 07/04/2014 10:59 AM, Volker Simonis wrote:
> > > > > > > Hi Maynard,
> > > > > > > 
> > > > > > > we (i.e. SAP) do not currently support the SA agent on Linux/PPC64 and
> > > > > > > AIX (we have other proprietary servicibility tools). Because of that
> > > > > > > (and because SA isn't specified by the SE specification) porting the
> > > > > > > SA agent was no top priority until now. But there are no technical
> > > > > > > reasons why it should not work (it's just a lack of  resources). Of
> > > > > > > course contributions are always highly welcome:)
> > > > > > > 
> > > > > > > That said, the SA agent library and jar file actually gets build. If
> > > > > > > you do a complete build you'll find them under:
> > > > > > > 
> > > > > > > hotspot/linux_ppc64_compiler2/generated/sa-jdi.jar
> > > > > > > hotspot/linux_ppc64_compiler2/{product,fastdebug,debug}/libsaproc.so
> > > > > > > 
> > > > > > > in the build directory. They are just not copied into the jdk
> > > > > > > workspace and the created images because they don't work out of the
> > > > > > > box.
> > > > > > > 
> > > > > > > The following two patches for the jdk9 top-level and hotspot
> > > > > > > repositories respectively should fix the build such that the agent
> > > > > > > files will be correctly copied into the images.
> > > > > > > 
> > > > > > > http://cr.openjdk.java.net/~simonis/webrevs/sa_toplevel
> > > > > > > http://cr.openjdk.java.net/~simonis/webrevs/sa_hotspot/
> > > > > > > 
> > > > > > > They will get you to the point where for example 'jstack' will run up
> > > > > > > to the following point:
> > > > > > Ok, great.  This should be enough to get me started.  I should have time \
> > > > > > to begin on this later this week or early next week.
> > > > > 
> > > > > Hi Maynard,
> > > > > 
> > > > > great to welcome you in the ppc64 porting team:)
> > > > > 
> > > > > > I may come knocking at your "door" for some occasional help, but I'll try \
> > > > > > to keep that to a minimum.
> > > > Hi, Volker.  Knock, knock.  :-)
> > > > I was preoccupied for a while this summer rolling out the latest release of \
> > > > oprofile (for which I'm the maintainer), but am now coming back to this task. \
> > > > I've implemented what I believe are all of the necessary ppc64-specific Java \
> > > > files to enable the jstack and jmap tools to work on core files.  I've also \
> > > > updated hotspot/agent/src/os/linux/LinuxDebuggerLocal.c to implement the \
> > > > accumulation of register data on ppc64 vi ptrace.  But now I've run into a \
> > > > problem I need help with. 
> > > > When I run jstack on my POWER7 system, it gets stuck in a loop in \
> > > > sun.jvm.hotspot.tools.StackTrace::run.  There's an inner for-loop there where \
> > > > cur.getLastJavaVFrameDbg() is called ('cur' is a JavaThread). For the first \
> > > > JavaThread, we do return from getLastJavaVFrameDbg(), just as we do when \
> > > > running jstack on my Intel laptop.  But for the second JavaThread, we never \
> > > > return from getLastJavaVFrameDbg() on ppc64.  I believe the root of the \
> > > > problem is in my new  sun.jvm.hotspot.runtime.ppc64.PPC64Frame class. The \
> > > > getLastJavaVFrameDbg method calls getCurrentFrameGuess, which is implemented \
> > > > in the new sun.jvm.hotspot.runtime.linux_ppc64.LinuxPPC64JavaThreadPDAccess \
> > > > class. In both ppc64 and x86, this first level xxxCurrentFrameGuess object is \
> > > > instantiated with a 'pc' value of null, so getCurrentFrameGuess then new's up \
> > > > a xxxFrame object, passing in the SP and FP, but no PC. The implementation of \
> > > > the PPC64Frame(Address,Address) constructor is currently identical to the \
> > > > X86Frame !
> c!
> > ons!
> > > > tructor, b
> > > > ut is almost certainly incorrect. In this constructor, the 'pc' is set as \
> > > > follows: this.pc = raw_sp.getAddressAt(-1 * VM.getVM().getAddressSize());
> > > > 
> > > > This works fine on X86, but not on ppc64. But I'm not understanding how this \
> > > > even works on X86. From what I understand, the data below the stack pointer \
> > > > on X86 is the "red zone". How is that being used as a pc? But more \
> > > > importantly, do you know how I can ascertain what the 'pc' value should be \
> > > > for ppc64? 
> > > > Thanks in advance for any assistance you can give.
> > > > 
> > > > -Maynard
> > > > 
> > > > > 
> > > > > Please feel free to ask any questions. The OpenJDK project and
> > > > > especially the HotSpot part are known to take some getting used to.
> > > > > 
> > > > > > I was wondering if a bug report should be opened in JBS, just to record \
> > > > > > that the issue is being worked.  Thoughts?
> > > > > 
> > > > > I have opened "8049715: PPC64: First steps to enable SA on
> > > > > Linux/PPC64" (https://bugs.openjdk.java.net/browse/JDK-8049715) for
> > > > > the patch which I sent you with the last mail. I've already sent out
> > > > > webrevs for that change and hopefully it will be fixed within the next
> > > > > few days.
> > > > > 
> > > > > For the actual port of the ppc64-specific stuff I opened bug "8049716
> > > > > PPC64: Implement SA on Linux/PPC64"
> > > > > (https://bugs.openjdk.java.net/browse/JDK-8049716). I can also help
> > > > > with hosting the webrevs, once you have a running version.
> > > > > 
> > > > > Regards,
> > > > > Volker
> > > > > 
> > > > > > 
> > > > > > -Maynard
> > > > > > > 
> > > > > > > > images/j2sdk-image/bin/jstack ./jdk/bin/java core.13547
> > > > > > > Attaching to core core.13547 from executable ./jdk/bin/java, please \
> > > > > > >                 wait...
> > > > > > > WARNING: Hotspot VM version
> > > > > > > 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00 does not match with
> > > > > > > SA version 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00. You may
> > > > > > > see unexpected results.
> > > > > > > Debugger attached successfully.
> > > > > > > Server compiler detected.
> > > > > > > JVM version is 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00
> > > > > > > Deadlock Detection:
> > > > > > > 
> > > > > > > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > >  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > >  at java.lang.reflect.Method.invoke(Method.java:484)
> > > > > > > at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
> > > > > > > at sun.tools.jstack.JStack.main(JStack.java:106)
> > > > > > > Caused by: java.lang.ExceptionInInitializerError
> > > > > > > at sun.jvm.hotspot.runtime.VM.getThreads(VM.java:610)
> > > > > > > at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:54)
> > > > > > >  at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> > > > > > >  at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62)
> > > > > > > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> > > > > > > at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
> > > > > > > at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
> > > > > > > at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
> > > > > > > at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
> > > > > > > at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
> > > > > > > ... 6 more
> > > > > > > Caused by: java.lang.RuntimeException: OS/CPU combination linux/ppc64
> > > > > > > not yet supported
> > > > > > > at sun.jvm.hotspot.runtime.Threads.initialize(Threads.java:97)
> > > > > > > at sun.jvm.hotspot.runtime.Threads.access$000(Threads.java:42)
> > > > > > > at sun.jvm.hotspot.runtime.Threads$1.update(Threads.java:52)
> > > > > > > at sun.jvm.hotspot.runtime.VM.registerVMInitializedObserver(VM.java:394)
> > > > > > >  at sun.jvm.hotspot.runtime.Threads.<clinit>(Threads.java:50)
> > > > > > > ... 16 more
> > > > > > > 
> > > > > > > And that's the point where I was saying that "contributions are always
> > > > > > > highly welcome:)"
> > > > > > > 
> > > > > > > Now all the Linux/PPC64 specific class under
> > > > > > > hotspot/agent/src/share/classes/ would have to be implemented (e.g.
> > > > > > > sun/jvm/hotspot/runtime/amd64/AMD64CurrentFrameGuess). Are you
> > > > > > > interested in contributing to this project?
> > > > > > > 
> > > > > > > Regards,
> > > > > > > Volker
> > > > > > > 
> > > > > > > PS: I cc-ed serviceability-dev because I remember that they started a
> > > > > > > poll a while ago about who is using the SA tools. I'm therefore not
> > > > > > > quite sure what's the current status and what are the future plan for
> > > > > > > these libraries.
> > > > > > > 
> > > > > > > 
> > > > > > > On Thu, Jul 3, 2014 at 9:04 PM, Maynard Johnson <maynardj@us.ibm.com> \
> > > > > > > wrote:
> > > > > > > > Hi, all,
> > > > > > > > On my Intel laptop, I note that certain jdk9 serviceability tools -- \
> > > > > > > > jstack, jmap, jsadebugd -- have an option to pass a core file instead \
> > > > > > > > of a process ID; for example: 
> > > > > > > > $ jstack -h
> > > > > > > > Usage:
> > > > > > > > jstack [-l] <pid>
> > > > > > > > (to connect to running process)
> > > > > > > > jstack -F [-m] [-l] <pid>
> > > > > > > > (to connect to a hung process)
> > > > > > > > jstack [-m] [-l] <executable> <core>
> > > > > > > > (to connect to a core file)
> > > > > > > > jstack [-m] [-l] [server_id@]<remote server IP or hostname>
> > > > > > > > (to connect to a remote debug server)
> > > > > > > > 
> > > > > > > > But on my PowerLinux box, the core file option is missing from the \
> > > > > > > > usage output.  I see that \
> > > > > > > > jdk9-dev/jdk/src/share/classes/sun/tools/jstack/JStack.java requires \
> > > > > > > > the existence of sun.jvm.hotspot.tools.JStack for the core file \
> > > > > > > > option to be made available.  On my Intel system, the \
> > > > > > > > sun.jvm.hotspot.tools.JStack class is packaged in sa-jdi.jar in \
> > > > > > > > <jdk9Dev-install>/jvm/openjdk-1.9.0-internal/lib/.  But the \
> > > > > > > > sa-jdi.jar is missing on PowerPC.  Is there a technical reason for \
> > > > > > > > this or is it an oversight? 
> > > > > > > > The jsadebugd tool does not run at all on PowerLinux; it gets the \
> > > > > > > > following error: 
> > > > > > > > Error: Could not find or load main class \
> > > > > > > > sun.jvm.hotspot.jdi.SADebugServer 
> > > > > > > > On my Intel system, the SADebugServer class is packaged in the \
> > > > > > > > sa-jdi.jar mentioned above. 
> > > > > > > > I've spent the past day or so looking at makefiles until I'm \
> > > > > > > > cross-eyed, but haven't yet found where the issue might be.  Any tips \
> > > > > > > > would be appreciated. 
> > > > > > > > Thanks.
> > > > > > > > -Maynard
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 


[prev in list] [next in list] [prev in thread] [next in thread]