[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: For-Test: 8212933: Thread-SMR: requesting a VM operation whilst holding a ThreadsListHandle can
From: Robbin Ehn <robbin.ehn () oracle ! com>
Date: 2018-10-25 20:24:24
Message-ID: F6630121-4A43-4FB3-8D8F-4187E508DCD5 () oracle ! com
[Download RAW message or body]
Great, thanks.
I did some local stress tests and t1-5.
I'll rfr it tomorrow if KS passes.
/Robbin
Leonid Mesnik <leonid.mesnik@oracle.com> skrev: (25 oktober 2018 22:11:07 CEST)
> Testing is still in progress, out of 30min tests passed and 8 hours
> tasks are still running.
> http://java-dev.se.oracle.com:10067/mdash/jobs/lmesnik-ks-short-test-20181025-1842-7762?search=result.status%3APASSED
>
> I haven't run any other tests.
>
> Leonid
> > On Oct 25, 2018, at 6:15 AM, Robbin Ehn <robbin.ehn@oracle.com>
> wrote:
> >
> > Hi, here is a fix, which allows ThreadsList to be used over a VM
> operation.
> >
> >
> http://cr.openjdk.java.net/~rehn/8212933/v1_handshak_vm_cancel/webrev/
> >
> > Please test it out.
> >
> > /Robbin
> >
> > On 10/24/18 11:16 PM, serguei.spitsyn@oracle.com wrote:
> > > Leonid confirmed this deadlock is not reproducible if the
> Kitchensink agent_sampler is disabled.
> > > Also, applying the patch from Robbin (with agent_sampler enabled)
> hit new assert
> > > that has caught another case in JvmtiEnv::GetStackTrace with the
> same pattern:
> > > With proposed patch issue reproduced with hs_err (file in
> attachement):
> > > #
> > > # A fatal error has been detected by the Java Runtime Environment:
> > > #
> > > # Internal Error
> (/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:607),
> pid=26188, tid=26325
> > > # assert(!t->have_threads_list()) failed: Deadlock if we have
> exiting threads and if vm thread is running an VM op.
> > > #
> > > # JRE version: Java(TM) SE Runtime Environment (12.0) (slowdebug
> build 12-internal+0-2018-10-24-2022348.lmesnik.hs-bigapps)
> > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug
> 12-internal+0-2018-10-24-2022348.lmesnik.hs-bigapps, mixed mode,
> sharing, tiered, compressed oops, g1 gc, linux-amd64)
> > > # Core dump will be written. Default location: Core dumps may be
> processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I
> %h" (or dumping to
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/core.26188)
> #
> > > # If you would like to submit a bug report, please visit:
> > > # http://bugreport.java.com/bugreport/crash.jsp
> > > #
> > > --------------- S U M M A R Y ------------
> > > Command Line: -XX:MaxRAMPercentage=2 -XX:MaxRAMPercentage=50
> -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false
> -XX:-PrintVMOptions -XX:+DisplayVMOutputToStderr -XX:+UsePerfData
> -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags
> -XX:+DisableExplicitGC -XX:+PrintFlagsFinal -XX:+StartAttachListener
> -XX:NativeMemoryTracking=detail -XX:+FlightRecorder
> --add-exports=java.base/java.lang=ALL-UNNAMED
> --add-opens=java.base/java.lang=ALL-UNNAMED
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED
> -Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_c \
> losed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/java.io.tmpdir
>
> -Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed \
> _test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/user.home
>
> -agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so
> applications.kitchensink.process.stress.Main
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspo \
> t_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/kitchensink.final.properties
>
> Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32 cores, 235G, Oracle
> Linux Server release 7.3
> > > Time: Wed Oct 24 13:28:30 2018 PDT elapsed time: 3 seconds (0d 0h 0m
> 3s)
> > > --------------- T H R E A D ---------------
> > > Current thread (0x00002b9f68006000): JavaThread "Jvmti-AgentSampler"
> daemon [_thread_in_vm, id=26325,
> stack(0x00002b9f88808000,0x00002b9f88909000)]
> _threads_hazard_ptr=0x00002b9f68008e30,
> _nested_threads_hazard_ptr_cnt=0
> > > Stack: [0x00002b9f88808000,0x00002b9f88909000],
> sp=0x00002b9f88907440, free space=1021k
> > > Native frames: (J=compiled Java code, A=aot compiled Java code,
> j=interpreted, Vv=VM code, C=native code)
> > > V [libjvm.so+0x12a04bb] VMError::report_and_die(int, char const*,
> char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*,
> char const*, int, unsigned long)+0x6a5
> > > V [libjvm.so+0x129fdb3] VMError::report_and_die(Thread*, void*, char
> const*, int, char const*, char const*, __va_list_tag*)+0x57
> > > V [libjvm.so+0x8ca5ab] report_vm_error(char const*, int, char
> const*, char const*, ...)+0x152
> > > V [libjvm.so+0x12e485b] VMThread::execute(VM_Operation*)+0x99
> > > V [libjvm.so+0xdbec54] JvmtiEnv::GetStackTrace(JavaThread*, int,
> int, _jvmtiFrameInfo*, int*)+0xc0
> > > V [libjvm.so+0xd677cf] jvmti_GetStackTrace+0x2c2
> > > C [libJvmtiStressModule.so+0x302d] trace_stack+0xa9
> > > C [libJvmtiStressModule.so+0x3daf] agent_sampler+0x21f
> > > V [libjvm.so+0xddf595] JvmtiAgentThread::call_start_function()+0x67
> > > V [libjvm.so+0xddf52a]
> JvmtiAgentThread::start_function_wrapper(JavaThread*, Thread*)+0xf2
> > > V [libjvm.so+0x1218945] JavaThread::thread_main_inner()+0x17f
> > > V [libjvm.so+0x12187ad] JavaThread::run()+0x273
> > > V [libjvm.so+0x100e4ee] thread_native_entry(Thread*)+0x192
> > > Leonid attached the full hs_err log to the bug report.
> > > Thanks,
> > > Serguei
> > > On 10/24/18 02:00, serguei.spitsyn@oracle.com wrote:
> > > > Hi Robbin and David,
> > > >
> > > > There is no JvmtiEnv::SuspendThreadList call in the dumped stack
> traces.
> > > > But there is an instance of the JvmtiEnv::SuspendThread which seems
> to be supporting your theory:
> > > >
> > > > Thread 136 (Thread 0x2ae494100700 (LWP 28023)):
> > > > #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae454005800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
> > > > #2 0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae454005800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
> > > > #3 Monitor::IWait (this=this@entry=0x2ae398023c10,
> Self=Self@entry=0x2ae454004800, timo=timo@entry=0) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:768
> > > > #4 0x00002ae393b51f2e in Monitor::wait
> (this=this@entry=0x2ae398023c10, no_safepoint_check=<optimized out>,
> timeout=timeout@entry=0,
> as_suspend_equivalent=as_suspend_equivalent@entry=false) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
> > > > #5 0x00002ae393de7867 in VMThread::execute
> (op=op@entry=0x2ae4940ffb10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
> > > > #6 0x00002ae393d6a3bd in JavaThread::java_suspend
> (this=this@entry=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321
> > > > #7 0x00002ae3939ad7e1 in JvmtiSuspendControl::suspend
> (java_thread=java_thread@entry=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:847
> > > > #8 0x00002ae3939887ae in JvmtiEnv::SuspendThread
> (this=this@entry=0x2ae39801b270, java_thread=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:955
> > > > #9 0x00002ae39393a8c6 in jvmti_SuspendThread (env=0x2ae39801b270,
> thread=0x2ae49929fdf8) at
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:527
>
> > > > #10 0x00002ae394d973ee in agent_sampler (jvmti=0x2ae39801b270,
> env=<optimized out>, p=<optimized out>) at
> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:274
>
> > > > #11 0x00002ae3939ab24d in call_start_function (this=0x2ae454004800)
> at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85
> > > > #12 JvmtiAgentThread::start_function_wrapper
> (thread=0x2ae454004800, __the_thread__=<optimized out>) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79
> > > > #13 0x00002ae393d7338a in JavaThread::thread_main_inner
> (this=this@entry=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
> > > > #14 0x00002ae393d736c6 in JavaThread::run (this=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
> > > > #15 0x00002ae393ba0070 in thread_native_entry
> (thread=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
> > > > #16 0x00002ae3927b1e25 in start_thread () from
> /lib64/libpthread.so.0
> > > > #17 0x00002ae392cc234d in clone () from /lib64/libc.so.6
> > > >
> > > >
> > > > The JavaThread::suspend() call from the stack trace above also
> holds a ThreadsListHandle while executing
> VMThread::execute(&vm_suspend):
> > > >
> > > > void JavaThread::java_suspend() {
> > > > ThreadsListHandle tlh;
> > > > if (!tlh.includes(this) || threadObj() == NULL || is_exiting()) {
> > > > return;
> > > > }
> > > >
> > > > { MutexLockerEx ml(SR_lock(), Mutex::_no_safepoint_check_flag);
> > > > if (!is_external_suspend()) {
> > > > // a racing resume has cancelled us; bail out now
> > > > return;
> > > > }
> > > >
> > > > // suspend is done
> > > > uint32_t debug_bits = 0;
> > > > // Warning: is_ext_suspend_completed() may temporarily drop the
> > > > // SR_lock to allow the thread to reach a stable thread state
> if
> > > > // it is currently in a transient thread state.
> > > > if (is_ext_suspend_completed(false /* !called_by_wait */,
> > > > SuspendRetryDelay, &debug_bits)) {
> > > > return;
> > > > }
> > > > }
> > > >
> > > > VM_ThreadSuspend vm_suspend;
> > > > VMThread::execute(&vm_suspend);
> > > > }
> > > >
> > > >
> > > > I'll check with Leonid tomorrow if we can check if your patch below
> can fix this deadlock.
> > > >
> > > > Thanks,
> > > > Serguei
> > > >
> > > >
> > > > On 10/24/18 00:18, Robbin Ehn wrote:
> > > > > Hi, truncate:
> > > > >
> > > > > On 24/10/2018 02:00, serguei.spitsyn@oracle.com wrote:
> > > > > > > > One thing I noticed which Robbin should be able to expand upon
> is that Thread 101 is terminating and has called
> ThreadsSMRSupport::smr_delete and is blocked here:
> > > > > > > >
> > > > > > > > // Wait for a release_stable_list() call before we check
> again. No
> > > > > > > > // safepoint check, no timeout, and not as suspend equivalent
> flag
> > > > > > > > // because this JavaThread is not on the Threads list.
> > > > > > > >
> ThreadsSMRSupport::delete_lock()->wait(Mutex::_no_safepoint_check_flag,
> > > > > > > > 0,
> !Mutex::_as_suspend_equivalent_flag);
> > > > > > > >
> > > > > > > > As the comment says this thread is no longer on the
> Threads_list, but the VM_HandshakeAllThreads is not a safepoint
> operation and does not hold the Threads_lock, so is it possible this
> thread was captured by the JavaThreadIteratorWithHandle being used by
> VM_HandshakeAllThreads, before it got removed? If so we'd be hung
> waiting it for it handshake as it's not in a "safepoint-safe" or
> suspend-equivalent state.
> > > > >
> > > > > In short:
> > > > > # VM Thread
> > > > > VM Thread is in a loop, takes Threads_lock, takes a new snapshot
> of the Threads_list, scans the list and process handshakes on behalf of
> safe threads.
> > > > > Releases snapshot and Threads_lock and checks if all handshakes
> are completed
> > > > >
> > > > > # An exiting thread
> > > > > A thread exiting thread removes it self from _THE_ threads list,
> but must stick around if it is on any snapshots of alive. When it is no
> on any list it will cancel the handshake.
> > > > >
> > > > > Since VM thread during the handshake takes a new snapshot every
> iteration any exiting can proceed since it will not be a the new
> snapshot. Thus cancel the handshake and VM thread can exit the loop (if
> this was the last handshake).
> > > > >
> > > > > Constraint:
> > > > > If any thread grabs a snapshot of threads list and later tries to
> take a lock which is 'used' by VM Thread or inside the handshake we can
> deadlock.
> > > > >
> > > > > Considering that looking at e.g. : JvmtiEnv::SuspendThreadList
> > > > > Which calls VMThread::execute(&tsj); with a ThreadsListHandle
> alive, this could deadlock AFAICT. Since the thread will rest on
> VMOperationRequest_lock with a Threads list snapshot but VM thread
> cannot finishes handshake until that snapshot is released.
> > > > >
> > > > > I suggest first step is to add something like this patch below and
> fix the obvious ones first.
> > > > >
> > > > > Note, I have not verified that is the problem you are seeing, I'm
> saying that this seem to be real issue. And considering how the stack
> traces looks, it may be this.
> > > > >
> > > > > You want me going through this, just assign a bug if there is one?
> > > > >
> > > > > /Robbin
> > > > >
> > > > > diff -r 622fd3608374 src/hotspot/share/runtime/thread.hpp
> > > > > --- a/src/hotspot/share/runtime/thread.hpp Tue Oct 23 13:27:41
> 2018 +0200
> > > > > +++ b/src/hotspot/share/runtime/thread.hpp Wed Oct 24 09:13:17
> 2018 +0200
> > > > > @@ -167,2 +167,6 @@
> > > > > }
> > > > > + public:
> > > > > + bool have_threads_list();
> > > > > + private:
> > > > > +
> > > > > // This field is enabled via -XX:+EnableThreadSMRStatistics:
> > > > > diff -r 622fd3608374 src/hotspot/share/runtime/thread.inline.hpp
> > > > > --- a/src/hotspot/share/runtime/thread.inline.hpp Tue Oct 23
> 13:27:41 2018 +0200
> > > > > +++ b/src/hotspot/share/runtime/thread.inline.hpp Wed Oct 24
> 09:13:17 2018 +0200
> > > > > @@ -111,2 +111,6 @@
> > > > >
> > > > > +inline bool Thread::have_threads_list() {
> > > > > + return OrderAccess::load_acquire(&_threads_hazard_ptr) != NULL;
> > > > > +}
> > > > > +
> > > > > inline void Thread::set_threads_hazard_ptr(ThreadsList* new_list)
> {
> > > > > diff -r 622fd3608374 src/hotspot/share/runtime/vmThread.cpp
> > > > > --- a/src/hotspot/share/runtime/vmThread.cpp Tue Oct 23
> 13:27:41 2018 +0200
> > > > > +++ b/src/hotspot/share/runtime/vmThread.cpp Wed Oct 24
> 09:13:17 2018 +0200
> > > > > @@ -608,2 +608,3 @@
> > > > > if (!t->is_VM_thread()) {
> > > > > + assert(t->have_threads_list(), "Deadlock if we have exiting
> threads and if vm thread is running an VM op."); // fatal/guarantee
> > > > > SkipGCALot sgcalot(t); // avoid re-entrant attempts to
> gc-a-lot
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > > >
> > > > > > > > David
> > > > > > > > -----
> > > > > > > >
> > > > > > > > On 24/10/2018 9:18 AM, serguei.spitsyn@oracle.com wrote:
> > > > > > > > > Please, skip it - sorry for the noise.
> > > > > > > > > It is hard to prove anything with current dump.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Serguei
> > > > > > > > >
> > > > > > > > > On 10/23/18 9:09 AM, serguei.spitsyn@oracle.com wrote:
> > > > > > > > > > Hi David and Robbin,
> > > > > > > > > >
> > > > > > > > > > I have an idea that needs to be checked.
> > > > > > > > > > It can be almost the same deadlock scenario that I've already
> explained but more sophisticated.
> > > > > > > > > > I suspect a scenario with JvmtiThreadState_lock that the flag
> Monitor::_safepoint_check_always does not help much.
> > > > > > > > > > It can be verified by checking what monitors are used by the
> blocked threads.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Serguei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 10/23/18 07:38, Robbin Ehn wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > On 10/23/18 10:34 AM, David Holmes wrote:
> > > > > > > > > > > > Hi Serguei,
> > > > > > > > > > > >
> > > > > > > > > > > > The VMThread is executing VM_HandshakeAllThreads which is
> not a safepoint operation. There's no real way to tell from the stacks
> what it's stuck on.
> > > > > > > > > > >
> > > > > > > > > > > I cannot find a thread that is not considered safepoint safe
> or is_ext_suspended (thread 146). So the handshake should go through.
> The handshake will log a warning after a while, is there such warning
> from the handshake operation?
> > > > > > > > > > >
> > > > > > > > > > > There are several threads competing with e.g. Threads_lock,
> and threads waiting for GC and several other VM ops, could it just be
> really slow?
> > > > > > > > > > >
> > > > > > > > > > > /Robbin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > David
> > > > > > > > > > > >
> > > > > > > > > > > > On 23/10/2018 5:58 PM, serguei.spitsyn@oracle.com wrote:
> > > > > > > > > > > > > Hi David,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You are right, thanks.
> > > > > > > > > > > > > It means, this deadlock needs more analysis.
> > > > > > > > > > > > > For completeness, the stack traces are in attachments.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Serguei
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10/23/18 00:43, David Holmes wrote:
> > > > > > > > > > > > > > Hi Serguei,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The JvmtiThreadState_lock is always acquired with
> safepoint checks enabled, so all JavaThreads blocked trying to acquire
> it will be _thread_blocked and so safepoint-safe and so won't be
> holding up the safepoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > David
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 23/10/2018 5:21 PM, serguei.spitsyn@oracle.com wrote:
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've added the seviceability-dev mailing list.
> > > > > > > > > > > > > > > It can be interesting for the SVC folks. :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 10/22/18 22:14, Leonid Mesnik wrote:
> > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Seems last version also crashes with 2 other \
> > > > > > > > > > > > > > > > different
> symptoms.
> > > > > > > > > > > > > > > >
> http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status%3Afailed+AND+-state%3Ainvalid
> <http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status:failed+AND+-state:invalid>
>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Also it might hangs with stack attached. Seems that
> test might be blocked because it invoke 2 jvmti methods. Can jvmti
> agent invoke jvmti methods from different threads?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, in general.
> > > > > > > > > > > > > > > However, you have to be careful when using debugging
> features.
> > > > > > > > > > > > > > > Below, one thread is enabling single stepping while
> another thread is being suspended.
> > > > > > > > > > > > > > > Both are blocked at a safepoint which is Okay in \
> > > > > > > > > > > > > > > general
> but not Okay if they hold any lock.
> > > > > > > > > > > > > > > For instance, the thread #152 is holding the monitor
> JvmtiThreadState.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Also, I see a couple of more threads that are
> interesting as well:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thread 159 (Thread 0x2ae40b78f700 (LWP 27962)):
> > > > > > > > > > > > > > > #0 0x00002ae3927b5945 in \
> > > > > > > > > > > > > > > pthread_cond_wait@@GLIBC_2.3.2
> () from /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae3984c9100) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>
> > > > > > > > > > > > > > > #2 0x00002ae393b50920 in ParkCommon (timo=0,
> ev=0x2ae3984c9100) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>
> > > > > > > > > > > > > > > #3 Monitor::ILock (this=0x2ae398024f10,
> Self=0x2ae3984c7800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>
> > > > > > > > > > > > > > > #4 0x00002ae393b512c1 in lock
> (Self=0x2ae3984c7is_ext_suspended800, this=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>
> > > > > > > > > > > > > > > #5 Monitor::lock (this=this@entry=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>
> > > > > > > > > > > > > > > #6 0x00002ae39350510c in MutexLocker
> (mutex=0x2ae398024f10, this=<synthetic pointer>) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>
> > > > > > > > > > > > > > > #7 ciEnv::cache_jvmti_state
> (this=this@entry=0x2ae40b78eb30) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
> > > > > > > > > > > > > > > #8 0x00002ae3935d3294 in
> CompileBroker::invoke_compiler_on_method
> (task=task@entry=0x2ae48800ff40) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084
>
> > > > > > > > > > > > > > > #9 0x00002ae3935d4f48 in
> CompileBroker::compiler_thread_loop () at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798
>
> > > > > > > > > > > > > > > #10 0x00002ae393d7338a in JavaThread::thread_main_inner
> (this=this@entry=0x2ae3984c7800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>
> > > > > > > > > > > > > > > #11 0x00002ae393d736c6 in JavaThread::run
> (this=0x2ae3984c7800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>
> > > > > > > > > > > > > > > #12 0x00002ae393ba0070 in thread_native_entry
> (thread=0x2ae3984c7800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>
> > > > > > > > > > > > > > > #13 0x00002ae3927b1e25 in start_thread () from
> /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #14 0x00002ae392cc234d in clone () from \
> > > > > > > > > > > > > > > /lib64/libc.so.6
> > > > > > > > > > > > > > > Thread 158 (Thread 0x2ae40b890700 (LWP 27963)):
> > > > > > > > > > > > > > > #0 0x00002ae3927b5945 in \
> > > > > > > > > > > > > > > pthread_cond_wait@@GLIBC_2.3.2
> () from /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae3984cbb00) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>
> > > > > > > > > > > > > > > #2 0x00002ae393b50920 in ParkCommon (timo=0,
> ev=0x2ae3984cbb00) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>
> > > > > > > > > > > > > > > #3 Monitor::ILock (this=0x2ae398024f10,
> Self=0x2ae3984ca800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>
> > > > > > > > > > > > > > > #4 0x00002ae393b512c1 in lock (Self=0x2ae3984ca800,
> this=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>
> > > > > > > > > > > > > > > #5 Monitor::lock (this=this@entry=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>
> > > > > > > > > > > > > > > #6 0x00002ae39350510c in MutexLocker
> (mutex=0x2ae398024f10, this=<synthetic pointer>) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>
> > > > > > > > > > > > > > > #7 ciEnv::cache_jvmti_state
> (this=this@entry=0x2ae40b88fb30) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
> > > > > > > > > > > > > > > #8 0x00002ae3935d3294 in
> CompileBroker::invoke_compiler_on_method
> (task=task@entry=0x2ae49c00a670) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084
>
> > > > > > > > > > > > > > > #9 0x00002ae3935d4f48 in
> CompileBroker::compiler_thread_loop () at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798
>
> > > > > > > > > > > > > > > #10 0x00002ae393d7338a in JavaThread::thread_main_inner
> (this=this@entry=0x2ae3984ca800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>
> > > > > > > > > > > > > > > #11 0x00002ae393d736c6 in JavaThread::run
> (this=0x2ae3984ca800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>
> > > > > > > > > > > > > > > #12 0x00002ae393ba0070 in thread_native_entry
> (thread=0x2ae3984ca800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>
> > > > > > > > > > > > > > > #13 0x00002ae3927b1e25 in start_thread () from
> /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #14 0x00002ae392cc234d in clone () from \
> > > > > > > > > > > > > > > /lib64/libc.so.6
> > > > > > > > > > > > > > > Thread 51 (Thread 0x2ae49549b700 (LWP 29678)):
> > > > > > > > > > > > > > > #0 0x00002ae3927b5945 in \
> > > > > > > > > > > > > > > pthread_cond_wait@@GLIBC_2.3.2
> () from /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae460061c00) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>
> > > > > > > > > > > > > > > #2 0x00002ae393b50920 in ParkCommon (timo=0,
> ev=0x2ae460061c00) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>
> > > > > > > > > > > > > > > #3 Monitor::ILock (this=0x2ae398024f10,
> Self=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>
> > > > > > > > > > > > > > > #4 0x00002ae393b512c1 in lock (Self=0x2ae4600c2800,
> this=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>
> > > > > > > > > > > > > > > #5 Monitor::lock (this=this@entry=0x2ae398024f10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>
> > > > > > > > > > > > > > > #6 0x00002ae393999682 in MutexLocker
> (mutex=0x2ae398024f10, this=<synthetic pointer>) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>
> > > > > > > > > > > > > > > #7 thread_started (thread=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:668
>
> > > > > > > > > > > > > > > #8 JvmtiEventController::thread_started
> (thread=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:1027
>
> > > > > > > > > > > > > > > #9 0x00002ae39399f3a0 in \
> > > > > > > > > > > > > > > JvmtiExport::post_thread_start
> (thread=thread@entry=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiExport.cpp:1395
>
> > > > > > > > > > > > > > > #10 0x00002ae393d737d8 in JavaThread::run
> (this=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1764
>
> > > > > > > > > > > > > > > #11 0x00002ae393ba0070 in thread_native_entry
> (thread=0x2ae4600c2800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>
> > > > > > > > > > > > > > > #12 0x00002ae3927b1e25 in start_thread () from
> /lib64/libpthread.so.0
> > > > > > > > > > > > > > > #13 0x00002ae392cc234d in clone () from \
> > > > > > > > > > > > > > > /lib64/libc.so.6
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > These two thread are blocked on the monitor
> JvmtiThreadState_lock in the function ciEnv::cache_jvmti_state().
> > > > > > > > > > > > > > > Also, there are many threads (like #51) that are
> executing JvmtiExport::post_thread_start and blocked on the same
> monitor.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Now, the question is why this safepoint can not start?
> > > > > > > > > > > > > > > What thread is blocking it? Or in reverse, what thread
> this safepoint is waiting for?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think, this safepoint operation is waiting for all
> threads that are blocked on the JvmtiThreadState_lock.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Conclusion:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The deadlock is:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thread #152:
> > > > > > > > > > > > > > > - grabbed the monitor JvmtiThreadState_lock
> > > > > > > > > > > > > > > - blocked in the VM_GetCurrentLocation in the
> function JvmtiEnvThreadState::reset_current_location()
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Many other threads:
> > > > > > > > > > > > > > > - blocked on the monitor JvmtiThreadState_lock
> > > > > > > > > > > > > > > - can not reach the blocked at a safepoint state (all
> threads have to reach this state for this safepoint to happen)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It seems to me, this is a bug which has to be filed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My guess is that this will stop to reproduce after if
> you turn off the single stepping for thread #152.
> > > > > > > > > > > > > > > Please, let me know about the results.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Assuming that crashes look like VM bugs I think it \
> > > > > > > > > > > > > > > > make
> sense to integrate jvmti changes but *don't* enabled jvmti module by
> default.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This one is a deadlock.
> > > > > > > > > > > > > > > However, the root cause is a race condition that can
> potentially result in both deadlocks and crashes.
> > > > > > > > > > > > > > > So, I'm curious if you observed crashes as well.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > And add to more tests with jvmti enabled.
> > > > > > > > > > > > > > > > So anyone could easily run them to reproduce crashes. \
> > > > > > > > > > > > > > > >
> This test would be out of CI to don't introduce any bugs. Does it make
> sense?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Consider hang - I think that it might be product bug
> since I don't see any locking on my monitors. But I am not sure. Is it
> possible that any my code jvmti agent prevent VM to get into safepoint?
> > > > > > > > > > > > > > > > Could we discuss it tomorrow or his week when you \
> > > > > > > > > > > > > > > > have
> a time?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, of course.
> > > > > > > > > > > > > > > Let's find some time tomorrow.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Any suggestion how to diagnose deadlock would be \
> > > > > > > > > > > > > > > > great.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Analysis of stack traces is needed.
> > > > > > > > > > > > > > > It is non-trivial in this particular case as there are
> so many threads executed at the same time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Part of stack trace with 2 my threads only:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thread 136 (Thread 0x2ae494100700 (LWP 28023)):
> > > > > > > > > > > > > > > > #0 0x00002ae3927b5945 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> > > > > > > > > > > > > > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae454005800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>
> > > > > > > > > > > > > > > > #2 0x00002ae393b50cf8 in ParkCommon (timo=0,
> ev=0x2ae454005800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>
> > > > > > > > > > > > > > > > #3 Monitor::IWait (this=this@entry=0x2ae398023c10,
> Self=Self@entry=0x2ae454004800, timo=timo@entry=0) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\
>
> > > > > > > > > > > > > > > > 8
> > > > > > > > > > > > > > > > #4 0x00002ae393b51f2e in Monitor::wait
> (this=this@entry=0x2ae398023c10, no_safepoint_check=<optimized out>,
> timeout=timeout@entry=0,
> as_suspend_equivalent=as_suspend_equivalent@en\
> > > > > > > > > > > > > > > > try=false) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
>
> > > > > > > > > > > > > > > > #5 0x00002ae393de7867 in VMThread::execute
> (op=op@entry=0x2ae4940ffb10) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
>
> > > > > > > > > > > > > > > > #6 0x00002ae393d6a3bd in JavaThread::java_suspend
> (this=this@entry=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321
>
> > > > > > > > > > > > > > > > #7 0x00002ae3939ad7e1 in \
> > > > > > > > > > > > > > > > JvmtiSuspendControl::suspend
> (java_thread=java_thread@entry=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:8\
>
> > > > > > > > > > > > > > > > 47
> > > > > > > > > > > > > > > > #8 0x00002ae3939887ae in JvmtiEnv::SuspendThread
> (this=this@entry=0x2ae39801b270, java_thread=0x2ae3985f2000) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiE\
> > > > > > > > > > > > > > > > nv.cpp:955
> > > > > > > > > > > > > > > > #9 0x00002ae39393a8c6 in jvmti_SuspendThread
> (env=0x2ae39801b270, thread=0x2ae49929fdf8) at
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles\
>
> > > > > > > > > > > > > > > > /jvmtiEnter.cpp:527
> > > > > > > > > > > > > > > > #10 0x00002ae394d973ee in agent_sampler
> (jvmti=0x2ae39801b270, env=<optimized out>, p=<optimized out>) at
> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitc\
>
> > > > > > > > > > > > > > > >
> hensink/process/stress/modules/libJvmtiStressModule.c:274
> > > > > > > > > > > > > > > > #11 0x00002ae3939ab24d in call_start_function
> (this=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85
>
> > > > > > > > > > > > > > > > #12 JvmtiAgentThread::start_function_wrapper
> (thread=0x2ae454004800, __the_thread__=<optimized out>) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79
>
> > > > > > > > > > > > > > > > #13 0x00002ae393d7338a in \
> > > > > > > > > > > > > > > > JavaThread::thread_main_inner
> (this=this@entry=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>
> > > > > > > > > > > > > > > > #14 0x00002ae393d736c6 in JavaThread::run
> (this=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>
> > > > > > > > > > > > > > > > #15 0x00002ae393ba0070 in thread_native_entry
> (thread=0x2ae454004800) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>
> > > > > > > > > > > > > > > > #16 0x00002ae3927b1e25 in start_thread () from
> /lib64/libpthread.so.0
> > > > > > > > > > > > > > > > #17 0x00002ae392cc234d in clone () from
> /lib64/libc.so.6
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thread 152 (Thread 0x2ae427060700 (LWP 27995)):
> > > > > > > > > > > > > > > > #0 0x00002ae3927b5945 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> > > > > > > > > > > > > > > > #1 0x00002ae393ba8d63 in os::PlatformEvent::park
> (this=this@entry=0x2ae3985e7400) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>
> > > > > > > > > > > > > > > > #2 0x00002ae393b50cf8 in ParkCommon (timo=0,
> ev=0x2ae3985e7400) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>
> > > > > > > > > > > > > > > > #3 Monitor::IWait (this=this@entry=0x2ae398023c10,
> Self=Self@entry=0x2ae3985e6000, timo=timo@entry=0) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\
>
> > > > > > > > > > > > > > > > 8
> > > > > > > > > > > > > > > > #4 0x00002ae393b51f2e in Monitor::wait
> (this=this@entry=0x2ae398023c10, no_safepoint_check=<optimized out>,
> timeout=timeout@entry=0,
> as_suspend_equivalent=as_suspend_equivalent@en\
> > > > > > > > > > > > > > > > try=false) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
>
> > > > > > > > > > > > > > > > #5 0x00002ae393de7867 in VMThread::execute
> (op=0x2ae42705f500) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
>
> > > > > > > > > > > > > > > > #6 0x00002ae3939965f3 in
> JvmtiEnvThreadState::reset_current_location
> (this=this@entry=0x2ae6bc000d80,
> event_type=event_type@entry=JVMTI_EVENT_SINGLE_STEP,
> enabled=enabled@entry=tr\
> > > > > > > > > > > > > > > > ue) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnvThreadState.cpp:312
>
> > > > > > > > > > > > > > > > #7 0x00002ae393997acf in \
> > > > > > > > > > > > > > > > recompute_env_thread_enabled
> (state=0x2ae6bc000cd0, ets=0x2ae6bc000d80) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventControlle\
>
> > > > > > > > > > > > > > > > r.cpp:490
> > > > > > > > > > > > > > > > #8
> JvmtiEventControllerPrivate::recompute_thread_enabled
> (state=state@entry=0x2ae6bc000cd0) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp\
>
> > > > > > > > > > > > > > > > > 523
> > > > > > > > > > > > > > > > #9 0x00002ae393998168 in
> JvmtiEventControllerPrivate::recompute_enabled () at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:598
>
> > > > > > > > > > > > > > > > #10 0x00002ae39399a244 in set_user_enabled
> (enabled=true, event_type=JVMTI_EVENT_SINGLE_STEP, thread=0x0,
> env=0x2ae39801b270) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/sha\
> > > > > > > > > > > > > > > > re/prims/jvmtiEventController.cpp:818
> > > > > > > > > > > > > > > > #11 JvmtiEventController::set_user_enabled
> (env=0x2ae39801b270, thread=0x0, event_type=JVMTI_EVENT_SINGLE_STEP,
> enabled=<optimized out>) at /scratch/lmesnik/ws/hs-bigapps/open/src/\
> > > > > > > > > > > > > > > > hotspot/share/prims/jvmtiEventController.cpp:963
> > > > > > > > > > > > > > > > #12 0x00002ae393987d2d in
> JvmtiEnv::SetEventNotificationMode (this=this@entry=0x2ae39801b270,
> mode=mode@entry=JVMTI_ENABLE,
> event_type=event_type@entry=JVMTI_EVENT_SINGLE_STEP, eve\
> > > > > > > > > > > > > > > > nt_thread=event_thread@entry=0x0) at
> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:543
>
> > > > > > > > > > > > > > > > #13 0x00002ae3939414eb in
> jvmti_SetEventNotificationMode (env=0x2ae39801b270,
> mode=mode@entry=JVMTI_ENABLE,
> event_type=event_type@entry=JVMTI_EVENT_SINGLE_STEP,
> event_thread=event_\
> > > > > > > > > > > > > > > > thread@entry=0x0) at
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:5389
>
> > > > > > > > > > > > > > > > #14 0x00002ae394d97989 in enable_events () at
> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:519
>
> > > > > > > > > > > > > > > > #15 0x00002ae394d98070 in
> Java_applications_kitchensink_process_stress_modules_JvmtiStressModule_startIteration
> (env=<optimized out>, this=<optimized out>) at /scratch/lmesnik/ws/h\
> > > > > > > > > > > > > > > >
> s-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:697
>
> > > > > > > > > > > > > > > > #16 0x00002ae3a43ef257 in ?? ()
> > > > > > > > > > > > > > > > #17 0x00002ae3a43eede1 in ?? ()
> > > > > > > > > > > > > > > > #18 0x00002ae42705f878 in ?? ()
> > > > > > > > > > > > > > > > #19 0x00002ae40ad334e0 in ?? ()
> > > > > > > > > > > > > > > > #20 0x00002ae42705f8e0 in ?? ()
> > > > > > > > > > > > > > > > #21 0x00002ae40ad33c68 in ?? ()
> > > > > > > > > > > > > > > > #22 0x0000000000000000 in ?? ()
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Serguei
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Leonid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Oct 9, 2018, at 4:52 PM, \
> > > > > > > > > > > > > > > > > serguei.spitsyn@oracle.com
> <mailto:serguei.spitsyn@oracle.com> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Leonid,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There is an existing bug:
> > > > > > > > > > > > > > > > > https://bugs.openjdk.java.net/browse/JDK-8043571
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Serguei
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On 10/9/18 16:11, Leonid Mesnik wrote:
> > > > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > During fixing kitchensink I get
> > > > > > > > > > > > > > > > > > assert(_cur_stack_depth == count_frames()) \
> > > > > > > > > > > > > > > > > > failed:
> cur_stack_depth out of sync
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Do you know if i might be bug in my jvmti agent?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Leonid
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > #
> > > > > > > > > > > > > > > > > > # A fatal error has been detected by the Java \
> > > > > > > > > > > > > > > > > > Runtime
> Environment:
> > > > > > > > > > > > > > > > > > #
> > > > > > > > > > > > > > > > > > # Internal Error
> (/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiThreadState.cpp:277),
> pid=13926, tid=13962
> > > > > > > > > > > > > > > > > > # assert(_cur_stack_depth == count_frames()) \
> > > > > > > > > > > > > > > > > > failed:
> cur_stack_depth out of sync
> > > > > > > > > > > > > > > > > > #
> > > > > > > > > > > > > > > > > > # JRE version: Java(TM) SE Runtime Environment \
> > > > > > > > > > > > > > > > > > (12.0)
> (fastdebug build 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps)
> > > > > > > > > > > > > > > > > > # Java VM: Java HotSpot(TM) 64-Bit Server VM
> (fastdebug 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps, mixed
> mode, tiered, compressed oops, g1 gc, linux-amd64)
> > > > > > > > > > > > > > > > > > # Core dump will be written. Default location: \
> > > > > > > > > > > > > > > > > > Core
> dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g
> %t e %P %I %h" (or dumping to
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/core.13926)
>
> > > > > > > > > > > > > > > > > > #
> > > > > > > > > > > > > > > > > > # If you would like to submit a bug report, \
> > > > > > > > > > > > > > > > > > please
> visit:
> > > > > > > > > > > > > > > > > > # http://bugreport.java.com/bugreport/crash.jsp
> > > > > > > > > > > > > > > > > > #
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --------------- S U M M A R Y ------------
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Command Line: -XX:MaxRAMPercentage=2
> -XX:MaxRAMPercentage=50 -XX:+CrashOnOutOfMemoryError
> -Djava.net.preferIPv6Addresses=false -XX:-PrintVMOptions
> -XX:+DisplayVMOutputToStderr -XX:+UsePerfData
> -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags
> -XX:+DisableExplicitGC -XX:+PrintFlagsFinal -XX:+StartAttachListener
> -XX:NativeMemoryTracking=detail -XX:+FlightRecorder
> --add-exports=java.base/java.lang=ALL-UNNAMED
> --add-opens=java.base/java.lang=ALL-UNNAMED
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED
> -Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_c \
> losed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/java.io.tmpdir
>
> -Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed \
> _test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/user.home
>
> -agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so
> applications.kitchensink.process.stress.Main
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspo \
> t_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/kitchensink.final.properties
>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Host: scaaa118.us.oracle.com
> <http://scaaa118.us.oracle.com>, Intel(R) Xeon(R) CPU E5-2690 0 @
> 2.90GHz, 32 cores, 235G, Oracle Linux Server release 7.3
> > > > > > > > > > > > > > > > > > Time: Tue Oct 9 16:06:07 2018 PDT elapsed time: \
> > > > > > > > > > > > > > > > > > 31
> seconds (0d 0h 0m 31s)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --------------- T H R E A D ---------------
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Current thread (0x00002af3dc6ac800): VMThread \
> > > > > > > > > > > > > > > > > > "VM
> Thread" [stack: 0x00002af44f10a000,0x00002af44f20a000] [id=13962]
> _threads_hazard_ptr=0x00002af4ac090eb0,
> _nested_threads_hazard_ptr_cnt=0
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Stack: [0x00002af44f10a000,0x00002af44f20a000],
> sp=0x00002af44f208720, free space=1017k
> > > > > > > > > > > > > > > > > > Native frames: (J=compiled Java code, A=aot \
> > > > > > > > > > > > > > > > > > compiled
> Java code, j=interpreted, Vv=VM code, C=native code)
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x18c4923]
> VMError::report_and_die(int, char const*, char const*, __va_list_tag*,
> Thread*, unsigned char*, void*, void*, char const*, int, unsigned
> long)+0x2c3
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x18c56ef]
> VMError::report_and_die(Thread*, void*, char const*, int, char const*,
> char const*, __va_list_tag*)+0x2f
> > > > > > > > > > > > > > > > > > V [libjvm.so+0xb55aa0] report_vm_error(char \
> > > > > > > > > > > > > > > > > > const*,
> int, char const*, char const*, ...)+0x100
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x11f2cfe]
> JvmtiThreadState::cur_stack_depth()+0x14e
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x11f3257]
> JvmtiThreadState::update_for_pop_top_frame()+0x27
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x119af99]
> VM_UpdateForPopTopFrame::doit()+0xb9
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x1908982]
> VM_Operation::evaluate()+0x132
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x19040be]
> VMThread::evaluate_operation(VM_Operation*) [clone .constprop.51]+0x18e
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x1904960] VMThread::loop()+0x4c0
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x1904f53] VMThread::run()+0xd3
> > > > > > > > > > > > > > > > > > V [libjvm.so+0x14e8300]
> thread_native_entry(Thread*)+0x100
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VM_Operation (0x00002af4d8502910):
> UpdateForPopTopFrame, mode: safepoint, requested by thread
> 0x00002af4dc008800
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic