[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: RFR: 8201409: JDWP debugger initialization hangs intermittently
From: Andrew Leonard <andrew_m_leonard () uk ! ibm ! com>
Date: 2018-04-24 9:43:23
Message-ID: OF69362E40.A64D3EB1-ON80258279.00355659-80258279.00356934 () notes ! na ! collabserv ! com
[Download RAW message or body]
This is a multipart message in MIME format.
--=_alternative 003568D680258279_=
Content-Type: text/plain; charset="US-ASCII"
Hi Serguei,
Good find, i'll try it out and do some debugging.
Many thanks
Andrew
Andrew Leonard
Java Runtimes Development
IBM Hursley
IBM United Kingdom Ltd
Phone internal: 245913, external: 01962 815913
internet email: andrew_m_leonard@uk.ibm.com
From: "serguei.spitsyn@oracle.com" <serguei.spitsyn@oracle.com>
To: Andrew Leonard <andrew_m_leonard@uk.ibm.com>
Cc: serviceability-dev@openjdk.java.net
Date: 24/04/2018 01:03
Subject: Re: RFR: 8201409: JDWP debugger initialization hangs
intermittently
On 4/23/18 16:08, serguei.spitsyn@oracle.com wrote:
Hi Andrew,
There is a regression with this fix.
The following test is failed with timeout on all platforms except Windows:
Sorry, forgot to copy the test name:
open/test/jdk/com/sun/jdi/JITDebug.sh
Thanks,
Serguei
I'll try to get more details about this timeout.
Thanks,
Serguei
On 4/18/18 09:49, serguei.spitsyn@oracle.com wrote:
Hi Andrew,
Sorry, I did not reply earlier.
The fix need more testing. We also have some important tests in closed.
I'll run them but I'm a little bit busy at the moment.
You have two reviews which is enough for push after testing.
Thanks,
Serguei
On 4/18/18 08:23, Andrew Leonard wrote:
Hi Serguei,
Do you need me to try anything else for this review?
hotspot/jtreg/serviceability suite run successfully.
Many Thanks
Andrew
Andrew Leonard
Java Runtimes Development
IBM Hursley
IBM United Kingdom Ltd
Phone internal: 245913, external: 01962 815913
internet email: andrew_m_leonard@uk.ibm.com
From: "serguei.spitsyn@oracle.com" <serguei.spitsyn@oracle.com>
To: daniel.daugherty@oracle.com, Andrew Leonard
<andrew_m_leonard@uk.ibm.com>
Cc: serviceability-dev@openjdk.java.net
Date: 16/04/2018 07:10
Subject: Re: RFR: 8201409: JDWP debugger initialization hangs
intermittently
On 4/15/18 10:01, Daniel D. Daugherty wrote:
On 4/13/18 3:07 PM, serguei.spitsyn@oracle.com wrote:
Andrew and reviewers,
I'm re-sending this RFR with a corrected subject that includes the bug
number.
The issues is:
https://bugs.openjdk.java.net/browse/JDK-8201409
Webrev:
http://cr.openjdk.java.net/~sspitsyn/webrevs/2018/8201409-jdwp-initsync.ibm.1/
src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c
No comments.
src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h
No comments.
src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c
So now pauses in debugLoop_run() before the loop
that reads cmds. Looks good.
src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
So the VM_INIT event handler now signals that we have
received the VM_INIT event so that allows debugLoop_run()
to proceed.
Serguei, this fix needs to have the most of the Serviceability
stack of tests run against it (jdwp, JVM/TI, JDI and jdb tests).
Based on the email thread, I can't tell which tests have been
run with the fix in place.
Hi Dan,
I'm going to sponsor this fix and will run all the debugger tests.
Sorry that I did not announce it yet.
Thanks,
Serguei
Dan
The fix looks good to me.
Also, I've agreed to skip a unit test as creating it for this issue is not
easy.
At least, one more review is needed before the fix can be pushed.
Thanks,
Serguei
On 4/11/18 06:33, Andrew Leonard wrote:
Hi Serguei,
Thank you for raising the bug.
I had a chat with one of my colleagues who could recreate it, and it's
probably related to the handshaking that is done in the particular
scenario. So with the JCK harness:
com.sun.jck.lib.ExecJCKTestOtherJVMCmd LD_LIBRARY_PATH=/javatest/lib/jck
/jck8b/natives/linux_x86-64 /projects/jck/jdwp/j2sdk-image/bin/java
-Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache
-Xdump:snap:none -Xdump:snap:events=gpf+abort+traceassert+corruptcache
-Xdump:java:none -Xdump:java:events=gpf+abort+traceassert+corruptcache
-Xdump:heap:none -Xdump:heap:events=gpf+abort+traceassert+corruptcache -
Xfuture -agentlib:jdwp=server=y,transport=dt_socket,address=localhost
> 35000,suspend=y -classpath /javatest/lib/jck
/JCK8b-b03/JCK-runtime-8b/classes -Djava.security.policy=/javatest/lib/jck
/JCK8b-b03/JCK-runtime-8b/lib/jck.policy
javasoft.sqe.jck.lib.jpda.jdwp.DebuggeeLoader -waittime=600
-msgSwitch=ub1604x64vm10:38636 -componentName=
ArrayReference.GetValues.getvalues002
Note that the JCK test harness starts the target process, attaches to it,
and sends the resume command
in a very short time with no handshaking.
That may not help..but hopefully helps explain things a bit? It's the
timing of the resume command during the test that is crucial, resuming
before the VM initialization is complete will trigger it.
Thanks
Andrew
Andrew Leonard
Java Runtimes Development
IBM Hursley
IBM United Kingdom Ltd
Phone internal: 245913, external: 01962 815913
internet email: andrew_m_leonard@uk.ibm.com
From: "serguei.spitsyn@oracle.com" <serguei.spitsyn@oracle.com>
To: Andrew Leonard <andrew_m_leonard@uk.ibm.com>
Cc: serviceability-dev@openjdk.java.net
Date: 11/04/2018 09:57
Subject: Re: RFR: Fix race condition in jdwp
Hi Andrew,
I've filed the bug:
https://bugs.openjdk.java.net/browse/JDK-8201409
Also, this is a webrev with your patch:
http://cr.openjdk.java.net/~sspitsyn/webrevs/2018/8201409-jdwp-initsync.ibm.1/
I agree that creating a standalone test is tricky here.
I've added usleep(10000) into the eventHelper_reportVMInit()
and ran the JTreg com/sun/jdi tests with my JDK build.
However, none of the tests failed with the failure mode you described.
So that I'm puzzled a little bit.
I suspect that some specific debugLoop commands were used in your
scenario.
It is still possible that I've missed something here.
Will try to double check everything.
Thanks,
Serguei
On 4/11/18 01:29, Andrew Leonard wrote:
Thanks Serguei,
I terms of a standalone testcase it is quite tricky, as due to the nature
of the issue which took a lot of investigation to solve it's very timing
dependent and will only occur randomly. It can be forced as I indicated
below by adding a "sleep" in the VMInit report code but that's not a
testcase, however the issue was originally found in our JCK testing for
IBMJava8, testcase test.jck8b.runtime.vm.jdwp, but again only happened
intermittently. Sort of like "performance" type issues we're not always
going to be able to create a testcase that will always "fail" if the fix
is not present.
Your thoughts?
Cheers
Andrew
Andrew Leonard
Java Runtimes Development
IBM Hursley
IBM United Kingdom Ltd
Phone internal: 245913, external: 01962 815913
internet email: andrew_m_leonard@uk.ibm.com
From: "serguei.spitsyn@oracle.com" <serguei.spitsyn@oracle.com>
To: Andrew Leonard <andrew_m_leonard@uk.ibm.com>
Cc: serviceability-dev@openjdk.java.net
Date: 11/04/2018 01:02
Subject: Re: RFR: Fix race condition in jdwp
Hi Andrew,
Okay, I'll file a bug on this topic.
But do you have a standalone test demonstrating this issue?
Thanks,
Serguei
On 4/10/18 06:23, Andrew Leonard wrote:
Hi Serguei,
I don't have access to the bug database to raise one, are you able to
please?
Summary: JDWP debugger initialization hangs intermittently
Description: If during the JDWP setup initialization the VM initialization
takes slightly longer than the main debug initialization thread a "hang"
situation can occur. This has been seen in testcase
test.jck8b.runtime.vm.jdwp and can also be recreated easily by adding a 10
second sleep to the beginning of the
src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c method
eventHelper_reportVMInit() .
First seen: JDK8
Recreated: JDK11
Thanks
Andrew
Andrew Leonard
Java Runtimes Development
IBM Hursley
IBM United Kingdom Ltd
Phone internal: 245913, external: 01962 815913
internet email: andrew_m_leonard@uk.ibm.com
From: "serguei.spitsyn@oracle.com" <serguei.spitsyn@oracle.com>
To: Andrew Leonard <andrew_m_leonard@uk.ibm.com>,
serviceability-dev@openjdk.java.net
Date: 09/04/2018 23:03
Subject: Re: RFR: Fix race condition in jdwp
Hi Andrew,
The patch itself looks reasonable.
However, in order to proceed with it, a bug report with a standalone
test case demonstrating the issue is needed.
Thanks,
Serguei
On 4/9/18 09:07, Andrew Leonard wrote:
> Hi,
> We discovered in our testing with OpenJ9 that a race condition can
> occur in the jdwp under certain circumstances, and we were able to
> force the same issue with Hotspot. Normally, the event helper thread
> suspends all threads, then the debug loop in the listener thread
> receives a command to resume. The debugger may deadlock if the debug
> loop in the listener thread starts processing commands (e.g. resume
> threads) before the event helper completes the initialization (and
> suspends threads).
>
> This patch adds synchronization to ensure the event helper completes
> the initialization sequence before debugger commands are processed.
>
> Please can I find a sponsor for this contribution? Patch below..
>
> Many thanks
>
> Andrew
>
>
>
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c
> @@ -1,5 +1,5 @@
> /*
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
> reserved.
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> *
> * This code is free software; you can redistribute it and/or modify it
> @@ -58,6 +58,7 @@
> static jboolean vmInitialized;
> static jrawMonitorID initMonitor;
> static jboolean initComplete;
> +static jboolean VMInitComplete;
> static jbyte currentSessionID;
>
> /*
> @@ -617,6 +618,35 @@
> debugMonitorExit(initMonitor);
> }
>
> +/*
> + * Signal VM initialization is complete.
> + */
> +void
> +signalVMInitComplete(void)
> +{
> + /*
> + * VM Initialization is complete
> + */
> + LOG_MISC(("signal VM initialization complete"));
> + debugMonitorEnter(initMonitor);
> + VMInitComplete = JNI_TRUE;
> + debugMonitorNotifyAll(initMonitor);
> + debugMonitorExit(initMonitor);
> +}
> +
> +/*
> + * Wait for VM initialization to complete.
> + */
> +void
> +debugInit_waitVMInitComplete(void)
> +{
> + debugMonitorEnter(initMonitor);
> + while (!VMInitComplete) {
> + debugMonitorWait(initMonitor);
> + }
> + debugMonitorExit(initMonitor);
> +}
> +
> /* All process exit() calls come from here */
> void
> forceExit(int exit_code)
> @@ -672,6 +702,7 @@
> LOG_MISC(("Begin initialize()"));
> currentSessionID = 0;
> initComplete = JNI_FALSE;
> + VMInitComplete = JNI_FALSE;
>
> if ( gdata->vmDead ) {
> EXIT_ERROR(AGENT_ERROR_INTERNAL,"VM dead at initialize() time");
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h
> @@ -1,5 +1,5 @@
> /*
> - * Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
> reserved.
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> *
> * This code is free software; you can redistribute it and/or modify it
> @@ -39,4 +39,7 @@
> void debugInit_exit(jvmtiError, const char *);
> void forceExit(int);
>
> +void debugInit_waitVMInitComplete(void);
> +void signalVMInitComplete(void);
> +
> #endif
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c
> @@ -1,5 +1,5 @@
> /*
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
> reserved.
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> *
> * This code is free software; you can redistribute it and/or modify it
> @@ -98,6 +98,7 @@
> standardHandlers_onConnect();
> threadControl_onConnect();
>
> + debugInit_waitVMInitComplete();
> /* Okay, start reading cmds! */
> while (shouldListen) {
> if (!dequeue(&p)) {
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
> b/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
> @@ -1,5 +1,5 @@
> /*
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
> reserved.
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
> *
> * This code is free software; you can redistribute it and/or modify it
> @@ -580,6 +580,7 @@
> (void)threadControl_suspendThread(command->thread, JNI_FALSE);
> }
>
> + signalVMInitComplete();
> outStream_initCommand(&out, uniqueID(), 0x0,
> JDWP_COMMAND_SET(Event),
> JDWP_COMMAND(Event, Composite));
>
>
>
> Andrew Leonard
> Java Runtimes Development
> IBM Hursley
> IBM United Kingdom Ltd
> Phone internal: 245913, external: 01962 815913
> internet email: andrew_m_leonard@uk.ibm.com
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with
> number 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--=_alternative 003568D680258279_=
Content-Type: text/html; charset="US-ASCII"
<span style=" font-size:10pt;font-family:sans-serif">Hi Serguei,</span>
<br><span style=" font-size:10pt;font-family:sans-serif">Good find, i'll
try it out and do some debugging.</span>
<br><span style=" font-size:10pt;font-family:sans-serif">Many thanks</span>
<br><span style=" font-size:10pt;font-family:sans-serif">Andrew</span>
<br>
<br><span style=" font-size:12pt">Andrew Leonard<br>
Java Runtimes Development<br>
IBM Hursley<br>
IBM United Kingdom Ltd<br>
Phone internal: 245913, external: 01962 815913<br>
internet email: andrew_m_leonard@uk.ibm.com </span>
<br>
<br>
<br>
<br>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" \
font-size:9pt;font-family:sans-serif">"serguei.spitsyn@oracle.com" \
<serguei.spitsyn@oracle.com></span> <br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: \
</span><span style=" font-size:9pt;font-family:sans-serif">Andrew Leonard \
<andrew_m_leonard@uk.ibm.com></span> <br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc: \
</span><span style=" \
font-size:9pt;font-family:sans-serif">serviceability-dev@openjdk.java.net</span> \
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date: \
</span><span style=" \
font-size:9pt;font-family:sans-serif">24/04/2018 01:03</span>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" \
font-size:9pt;font-family:sans-serif">Re:
RFR: 8201409: JDWP debugger initialization hangs intermittently</span>
<br>
<hr noshade>
<br>
<br>
<br><span style=" font-size:12pt">On 4/23/18 16:08, </span><a \
href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>serguei.spitsyn@oracle.com</u></span></a><span style=" \
font-size:12pt"> wrote:</span>
<br><span style=" font-size:12pt">Hi Andrew,<br>
<br>
There is a regression with this fix.<br>
The following test is failed with timeout on all platforms except Windows:</span>
<br><span style=" font-size:12pt"><br>
Sorry, forgot to copy the test name:<br>
open/test/jdk/com/sun/jdi/JITDebug.sh<br>
<br>
<br>
Thanks,<br>
Serguei<br>
<br>
</span>
<br><span style=" font-size:12pt">I'll try to get more details about this
timeout.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/18/18 09:49, </span><a href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>serguei.spitsyn@oracle.com</u></span></a><span style=" \
font-size:12pt"> wrote:</span>
<br><span style=" font-size:12pt">Hi Andrew,<br>
<br>
Sorry, I did not reply earlier.<br>
The fix need more testing. We also have some important tests in closed.<br>
I'll run them but I'm a little bit busy at the moment.<br>
<br>
You have two reviews which is enough for push after testing.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/18/18 08:23, Andrew Leonard wrote:</span>
<br><span style=" font-size:12pt">Hi Serguei, <br>
Do you need me to try anything else for this review? hotspot/jtreg/serviceability
suite run successfully. <br>
Many Thanks <br>
Andrew <br>
<br>
Andrew Leonard<br>
Java Runtimes Development<br>
IBM Hursley<br>
IBM United Kingdom Ltd<br>
Phone internal: 245913, external: 01962 815913<br>
internet email: </span><a href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u>andrew_m_leonard@uk.ibm.com</u></span></a><span style=" \
font-size:12pt"> <br>
<br>
<br>
<br>
<br>
From: </span><a \
href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>"serguei.spitsyn@oracle.com"</u></span></a><span \
style=" font-size:12pt"> </span><a href=mailto:serguei.spitsyn@oracle.com><span \
style=" font-size:12pt;color:blue"><u><serguei.spitsyn@oracle.com></u></span></a><span \
style=" font-size:12pt"> <br>
To: </span><a \
href=mailto:daniel.daugherty@oracle.com><span style=" \
font-size:12pt;color:blue"><u>daniel.daugherty@oracle.com</u></span></a><span style=" \
font-size:12pt">, Andrew Leonard </span><a \
href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u><andrew_m_leonard@uk.ibm.com></u></span></a><span \
style=" font-size:12pt"> <br>
Cc: </span><a \
href="mailto:serviceability-dev@openjdk.java.net"><span style=" \
font-size:12pt;color:blue"><u>serviceability-dev@openjdk.java.net</u></span></a><span \
style=" font-size:12pt"> <br>
Date: 16/04/2018 07:10 <br>
Subject: Re: RFR: 8201409: JDWP debugger initialization
hangs intermittently <br>
</span>
<hr><span style=" font-size:12pt"><br>
<br>
<br>
On 4/15/18 10:01, Daniel D. Daugherty wrote: <br>
On 4/13/18 3:07 PM, </span><a href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>serguei.spitsyn@oracle.com</u></span></a><span style=" \
font-size:12pt">
wrote: <br>
Andrew and reviewers,<br>
<br>
I'm re-sending this RFR with a corrected subject that includes the bug
number.<br>
<br>
The issues is:<br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openj \
dk.java.net_browse_JDK-2D8201409&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=NaV8I \
y8Ld-vjpXZFDdTbgGlRTghGHnwM75wUPd5_NUQ&m=nYdDNT1pdP7gQK1JviPJ6zIxLWcwe7R-azh_NnvI0Ok&s=z0_3fj5wG3cofrwXExX0jmnKvUeBGxjAOsYE1Gz6xCg&e="><span \
style=" font-size:12pt;color:blue"><u>https://bugs.openjdk.java.net/browse/JDK-8201409</u></span></a><span \
style=" font-size:12pt"><br> <br>
Webrev:<br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk. \
java.net_-7Esspitsyn_webrevs_2018_8201409-2Djdwp-2Dinitsync.ibm.1_&d=DwMDaQ&c= \
jf_iaSHvJObTbx-siA1ZOg&r=NaV8Iy8Ld-vjpXZFDdTbgGlRTghGHnwM75wUPd5_NUQ&m=nYdDNT1 \
pdP7gQK1JviPJ6zIxLWcwe7R-azh_NnvI0Ok&s=8LH2-PYUqJbVkawhCgmuD5106lOuFKI-jAVhCpB_tYY&e="><span \
style=" font-size:12pt;color:blue"><u>http://cr.openjdk.java.net/~sspitsyn/webrevs/2018/8201409-jdwp-initsync.ibm.1/</u></span></a><span \
style=" font-size:12pt"> </span><tt><span style=" font-size:12pt"><br>
<br>
src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c<br>
No comments.<br>
<br>
src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h<br>
No comments.<br>
<br>
src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c<br>
So now pauses in debugLoop_run() before the loop<br>
that reads cmds. Looks good.<br>
<br>
src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c<br>
So the VM_INIT event handler now signals that we have<br>
received the VM_INIT event so that allows debugLoop_run()<br>
to proceed.<br>
<br>
Serguei, this fix needs to have the most of the Serviceability<br>
stack of tests run against it (jdwp, JVM/TI, JDI and jdb tests).<br>
Based on the email thread, I can't tell which tests have been<br>
run with the fix in place.</span></tt><span style=" font-size:12pt"> <br>
<br>
Hi Dan,<br>
<br>
I'm going to sponsor this fix and will run all the debugger tests.<br>
Sorry that I did not announce it yet.<br>
<br>
Thanks,<br>
Serguei<br>
</span><tt><span style=" font-size:12pt"><br>
<br>
Dan</span></tt><span style=" font-size:12pt"><br>
<br>
<br>
<br>
The fix looks good to me.<br>
Also, I've agreed to skip a unit test as creating it for this issue is
not easy.<br>
<br>
At least, one more review is needed before the fix can be pushed.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/11/18 06:33, Andrew Leonard wrote: <br>
Hi Serguei, <br>
Thank you for raising the bug. <br>
I had a chat with one of my colleagues who could recreate it, and it's
probably related to the handshaking that is done in the particular scenario.
So with the JCK harness: <br>
<br>
com.sun.jck.lib.ExecJCKTestOtherJVMCmd \
LD_LIBRARY_PATH=/<u>javatest</u>/<u>lib</u>/<u>jck</u>/jck8b/natives/linux_x86-64
/projects/<u>jck</u>/jdwp/j2sdk-image/bin/java -Xdump:system:none \
-Xdump:system:events=<u>gpf</u>+abort+<u>traceassert</u>+<u>corruptcache</u>
-Xdump:snap:none -Xdump:snap:events=<u>gpf</u>+abort+<u>traceassert</u>+<u>corruptcache</u>
-Xdump:java:none -Xdump:java:events=<u>gpf</u>+abort+<u>traceassert</u>+<u>corruptcache</u>
-Xdump:heap:none -Xdump:heap:events=<u>gpf</u>+abort+<u>traceassert</u>+<u>corruptcache</u>
-<u>Xfuture</u> -agentlib:jdwp=server=y,transport=dt_socket,address=<u>localhost</u>:35000,suspend=y
-<u>classpath</u> /<u>javatest</u>/<u>lib</u>/<u>jck</u>/JCK8b-b03/JCK-runtime-8b/classes
-Djava.security.policy=/<u>javatest</u>/<u>lib</u>/<u>jck</u>/JCK8b-b03/JCK-runtime-8b/<u>lib</u>/jck.policy
javasoft.sqe.jck.lib.jpda.jdwp.DebuggeeLoader -<u>waittime</u>=600 \
-msgSwitch=ub1604x64vm10:38636
-componentName=<b>ArrayReference.GetValues.getvalues002</b> <br>
<br>
Note that the JCK test harness starts the target process, attaches to it,
and sends the resume command <br>
in a very short time with no handshaking. <br>
<br>
That may not help..but hopefully helps explain things a bit? It's the timing
of the resume command during the test that is crucial, resuming before
the VM initialization is complete will trigger it. <br>
<br>
Thanks <br>
Andrew <br>
<br>
Andrew Leonard<br>
Java Runtimes Development<br>
IBM Hursley<br>
IBM United Kingdom Ltd<br>
Phone internal: 245913, external: 01962 815913<br>
internet email: </span><a href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u>andrew_m_leonard@uk.ibm.com</u></span></a><span style=" \
font-size:12pt"> <br>
<br>
<br>
<br>
<br>
From: </span><a \
href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>"serguei.spitsyn@oracle.com"</u></span></a><span \
style=" font-size:12pt"> </span><a href=mailto:serguei.spitsyn@oracle.com><span \
style=" font-size:12pt;color:blue"><u><serguei.spitsyn@oracle.com></u></span></a><span \
style=" font-size:12pt"> <br>
To: Andrew Leonard </span><a \
href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u><andrew_m_leonard@uk.ibm.com></u></span></a><span \
style=" font-size:12pt"> <br>
Cc: </span><a \
href="mailto:serviceability-dev@openjdk.java.net"><span style=" \
font-size:12pt;color:blue"><u>serviceability-dev@openjdk.java.net</u></span></a><span \
style=" font-size:12pt"> <br>
Date: 11/04/2018 09:57 <br>
Subject: Re: RFR: Fix race condition in jdwp
<br>
</span>
<hr><span style=" font-size:12pt"><br>
<br>
<br>
Hi Andrew,<br>
<br>
I've filed the bug:<br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openj \
dk.java.net_browse_JDK-2D8201409&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=NaV8I \
y8Ld-vjpXZFDdTbgGlRTghGHnwM75wUPd5_NUQ&m=nYdDNT1pdP7gQK1JviPJ6zIxLWcwe7R-azh_NnvI0Ok&s=z0_3fj5wG3cofrwXExX0jmnKvUeBGxjAOsYE1Gz6xCg&e="><span \
style=" font-size:12pt;color:blue"><u>https://bugs.openjdk.java.net/browse/JDK-8201409</u></span></a><span \
style=" font-size:12pt"><br> <br>
Also, this is a webrev with your patch:<br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk. \
java.net_-7Esspitsyn_webrevs_2018_8201409-2Djdwp-2Dinitsync.ibm.1_&d=DwMDaQ&c= \
jf_iaSHvJObTbx-siA1ZOg&r=NaV8Iy8Ld-vjpXZFDdTbgGlRTghGHnwM75wUPd5_NUQ&m=nYdDNT1 \
pdP7gQK1JviPJ6zIxLWcwe7R-azh_NnvI0Ok&s=8LH2-PYUqJbVkawhCgmuD5106lOuFKI-jAVhCpB_tYY&e="><span \
style=" font-size:12pt;color:blue"><u>http://cr.openjdk.java.net/~sspitsyn/webrevs/2018/8201409-jdwp-initsync.ibm.1/</u></span></a><span \
style=" font-size:12pt"><br> <br>
I agree that creating a standalone test is tricky here.<br>
<br>
I've added usleep(10000) into the eventHelper_reportVMInit()<br>
and ran the JTreg com/sun/jdi tests with my JDK build.<br>
However, none of the tests failed with the failure mode you described.<br>
So that I'm puzzled a little bit.<br>
I suspect that some specific debugLoop commands were used in your scenario.<br>
<br>
It is still possible that I've missed something here.<br>
Will try to double check everything.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/11/18 01:29, Andrew Leonard wrote: <br>
Thanks Serguei, <br>
I terms of a standalone testcase it is quite tricky, as due to the nature
of the issue which took a lot of investigation to solve it's very timing
dependent and will only occur randomly. It can be forced as I indicated
below by adding a "sleep" in the VMInit report code but that's
not a testcase, however the issue was originally found in our JCK testing
for IBMJava8, testcase test.jck8b.runtime.vm.jdwp, but again only happened
intermittently. Sort of like "performance" type issues we're
not always going to be able to create a testcase that will always "fail"
if the fix is not present. <br>
Your thoughts? <br>
Cheers <br>
Andrew <br>
<br>
Andrew Leonard<br>
Java Runtimes Development<br>
IBM Hursley<br>
IBM United Kingdom Ltd<br>
Phone internal: 245913, external: 01962 815913<br>
internet email: </span><a href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u>andrew_m_leonard@uk.ibm.com</u></span></a><span style=" \
font-size:12pt"> <br>
<br>
<br>
<br>
<br>
From: </span><a \
href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>"serguei.spitsyn@oracle.com"</u></span></a><span \
style=" font-size:12pt"> </span><a href=mailto:serguei.spitsyn@oracle.com><span \
style=" font-size:12pt;color:blue"><u><serguei.spitsyn@oracle.com></u></span></a><span \
style=" font-size:12pt"> <br>
To: Andrew Leonard </span><a \
href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u><andrew_m_leonard@uk.ibm.com></u></span></a><span \
style=" font-size:12pt"> <br>
Cc: </span><a \
href="mailto:serviceability-dev@openjdk.java.net"><span style=" \
font-size:12pt;color:blue"><u>serviceability-dev@openjdk.java.net</u></span></a><span \
style=" font-size:12pt"> <br>
Date: 11/04/2018 01:02 <br>
Subject: Re: RFR: Fix race condition in jdwp
<br>
</span>
<hr><span style=" font-size:12pt"><br>
<br>
<br>
Hi Andrew,<br>
<br>
Okay, I'll file a bug on this topic.<br>
But do you have a standalone test demonstrating this issue?<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/10/18 06:23, Andrew Leonard wrote: <br>
Hi Serguei, <br>
I don't have access to the bug database to raise one, are you able to please?
<br>
<br>
Summary: JDWP debugger initialization hangs intermittently <br>
Description: If during the JDWP setup initialization the VM initialization
takes slightly longer than the main debug initialization thread a "hang"
situation can occur. This has been seen in testcase test.jck8b.runtime.vm.jdwp
and can also be recreated easily by adding a 10 second sleep to the beginning
of the src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c method \
eventHelper_reportVMInit()
. <br>
First seen: JDK8 <br>
Recreated: JDK11 <br>
<br>
Thanks <br>
Andrew <br>
<br>
Andrew Leonard<br>
Java Runtimes Development<br>
IBM Hursley<br>
IBM United Kingdom Ltd<br>
Phone internal: 245913, external: 01962 815913<br>
internet email: </span><a href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u>andrew_m_leonard@uk.ibm.com</u></span></a><span style=" \
font-size:12pt"> <br>
<br>
<br>
<br>
<br>
From: </span><a \
href=mailto:serguei.spitsyn@oracle.com><span style=" \
font-size:12pt;color:blue"><u>"serguei.spitsyn@oracle.com"</u></span></a><span \
style=" font-size:12pt"> </span><a href=mailto:serguei.spitsyn@oracle.com><span \
style=" font-size:12pt;color:blue"><u><serguei.spitsyn@oracle.com></u></span></a><span \
style=" font-size:12pt"> <br>
To: Andrew Leonard </span><a \
href=mailto:andrew_m_leonard@uk.ibm.com><span style=" \
font-size:12pt;color:blue"><u><andrew_m_leonard@uk.ibm.com></u></span></a><span \
style=" font-size:12pt">, </span><a \
href="mailto:serviceability-dev@openjdk.java.net"><span style=" \
font-size:12pt;color:blue"><u>serviceability-dev@openjdk.java.net</u></span></a><span \
style=" font-size:12pt"> <br>
Date: 09/04/2018 23:03 <br>
Subject: Re: RFR: Fix race condition in jdwp
<br>
</span>
<hr><tt><span style=" font-size:12pt"><br>
<br>
<br>
Hi Andrew,<br>
<br>
The patch itself looks reasonable.<br>
However, in order to proceed with it, a bug report with a standalone<br>
test case demonstrating the issue is needed.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
<br>
On 4/9/18 09:07, Andrew Leonard wrote:<br>
> Hi,<br>
> We discovered in our testing with OpenJ9 that a race condition can
<br>
> occur in the jdwp under certain circumstances, and we were able to
<br>
> force the same issue with Hotspot. Normally, the event helper thread
<br>
> suspends all threads, then the debug loop in the listener thread <br>
> receives a command to resume. The debugger may deadlock if the debug
<br>
> loop in the listener thread starts processing commands (e.g. resume
<br>
> threads) before the event helper completes the initialization (and
<br>
> suspends threads).<br>
><br>
> This patch adds synchronization to ensure the event helper completes
<br>
> the initialization sequence before debugger commands are processed.<br>
><br>
> Please can I find a sponsor for this contribution? Patch below..<br>
><br>
> Many thanks<br>
><br>
> Andrew<br>
><br>
><br>
><br>
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c <br>
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c<br>
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c<br>
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c<br>
> @@ -1,5 +1,5 @@<br>
> /*<br>
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.<br>
> *<br>
> * This code is free software; you can redistribute it and/or
modify it<br>
> @@ -58,6 +58,7 @@<br>
> static jboolean vmInitialized;<br>
> static jrawMonitorID initMonitor;<br>
> static jboolean initComplete;<br>
> +static jboolean VMInitComplete;<br>
> static jbyte currentSessionID;<br>
><br>
> /*<br>
> @@ -617,6 +618,35 @@<br>
> debugMonitorExit(initMonitor);<br>
> }<br>
><br>
> +/*<br>
> + * Signal VM initialization is complete.<br>
> + */<br>
> +void<br>
> +signalVMInitComplete(void)<br>
> +{<br>
> + /*<br>
> + * VM Initialization is complete<br>
> + */<br>
> + LOG_MISC(("signal VM initialization complete"));<br>
> + debugMonitorEnter(initMonitor);<br>
> + VMInitComplete = JNI_TRUE;<br>
> + debugMonitorNotifyAll(initMonitor);<br>
> + debugMonitorExit(initMonitor);<br>
> +}<br>
> +<br>
> +/*<br>
> + * Wait for VM initialization to complete.<br>
> + */<br>
> +void<br>
> +debugInit_waitVMInitComplete(void)<br>
> +{<br>
> + debugMonitorEnter(initMonitor);<br>
> + while (!VMInitComplete) {<br>
> + debugMonitorWait(initMonitor);<br>
> + }<br>
> + debugMonitorExit(initMonitor);<br>
> +}<br>
> +<br>
> /* All process exit() calls come from here */<br>
> void<br>
> forceExit(int exit_code)<br>
> @@ -672,6 +702,7 @@<br>
> LOG_MISC(("Begin initialize()"));<br>
> currentSessionID = 0;<br>
> initComplete = JNI_FALSE;<br>
> + VMInitComplete = JNI_FALSE;<br>
><br>
> if ( gdata->vmDead ) {<br>
> EXIT_ERROR(AGENT_ERROR_INTERNAL,"VM dead
at initialize() time");<br>
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h <br>
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h<br>
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h<br>
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugInit.h<br>
> @@ -1,5 +1,5 @@<br>
> /*<br>
> - * Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.<br>
> *<br>
> * This code is free software; you can redistribute it and/or
modify it<br>
> @@ -39,4 +39,7 @@<br>
> void debugInit_exit(jvmtiError, const char *);<br>
> void forceExit(int);<br>
><br>
> +void debugInit_waitVMInitComplete(void);<br>
> +void signalVMInitComplete(void);<br>
> +<br>
> #endif<br>
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c <br>
> b/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c<br>
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c<br>
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c<br>
> @@ -1,5 +1,5 @@<br>
> /*<br>
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.<br>
> *<br>
> * This code is free software; you can redistribute it and/or
modify it<br>
> @@ -98,6 +98,7 @@<br>
> standardHandlers_onConnect();<br>
> threadControl_onConnect();<br>
><br>
> + debugInit_waitVMInitComplete();<br>
> /* Okay, start reading cmds! */<br>
> while (shouldListen) {<br>
> if (!dequeue(&p)) {<br>
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c
<br>
> b/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c<br>
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c<br>
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/eventHelper.c<br>
> @@ -1,5 +1,5 @@<br>
> /*<br>
> - * Copyright (c) 1998, 2017, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights
<br>
> reserved.<br>
> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.<br>
> *<br>
> * This code is free software; you can redistribute it and/or
modify it<br>
> @@ -580,6 +580,7 @@<br>
> (void)threadControl_suspendThread(command->thread,
JNI_FALSE);<br>
> }<br>
><br>
> + signalVMInitComplete();<br>
> outStream_initCommand(&out, uniqueID(), 0x0,<br>
> JDWP_COMMAND_SET(Event),<br>
> JDWP_COMMAND(Event, Composite));<br>
><br>
><br>
><br>
> Andrew Leonard<br>
> Java Runtimes Development<br>
> IBM Hursley<br>
> IBM United Kingdom Ltd<br>
> Phone internal: 245913, external: 01962 815913<br>
> internet email: </span></tt><a href=mailto:andrew_m_leonard@uk.ibm.com><tt><span \
style=" font-size:12pt;color:blue"><u>andrew_m_leonard@uk.ibm.com</u></span></tt></a><tt><span \
style=" font-size:12pt"><br> ><br>
><br>
> Unless stated otherwise above:<br>
> IBM United Kingdom Limited - Registered in England and Wales with
<br>
> number 741598.<br>
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
PO6 3AU</span></tt><span style=" font-size:12pt"><br>
<br>
<br>
<br>
<br>
<br>
Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number
741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU <br>
<br>
<br>
<br>
<br>
Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number
741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU <br>
<br>
<br>
<br>
<br>
Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number
741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU <br>
<br>
<br>
<br>
<br>
<br>
<br>
Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number
741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU</span>
<br>
<br>
<br>
<br>
<br>
<br><span style=" font-size:10pt;font-family:sans-serif"><br>
Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number
741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU<br>
</span>
--=_alternative 003568D680258279_=--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic