[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openjdk-serviceability-dev
Subject:    Re: JFR thread sampling mechanism
From:       Kirk Pepperdine <kirk.pepperdine () gmail ! com>
Date:       2019-06-30 22:08:18
Message-ID: 2E7CA8A6-ACBA-4AFD-80EF-C278F8A239B4 () gmail ! com
[Download RAW message or body]

Hi Gil,

I would support an improvement in sampling as there is an obvious bias which allows \
me to write benchmarks where JFR completely misses things it should find. That said, \
I'm not sure that waking a thread up every 10ms is a great idea as it is very \
disruptive to Linux thread scheduling. I'd very much like to experiment with lower \
sampling rates.

Kind regards,
Kirk


> On Jun 30, 2019, at 9:20 AM, Gil Tene <gil@azul.com> wrote:
> 
> I would like to discuss a potential improvement to the JFR thread
> sampling mechanism, and would like to see if the change we'd
> propose has already been considered in the past.
> 
> I believe that the current thread sampling mechanism (mostly via
> hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp) can be
> summarized as: A control thread wakes up periodically (e.g. 100
> times per second) and in each period chooses a number (e.g. 5)
> threads to sample (by rotating through the overall list of threads)
> only if they are "in java", and a number (e.g. 1) threads (by
> separately rotating through the overall list of threads) to sample
> "only if it is in native". For each thread targeted to sample, the
> control thread suspends the target thread (e.g. for linux this is
> done by preparing a suspend request a sending a SIGUSR2 to
> make the thread deal with it), takes a stacktrace of the suspended
> thread, adds the stacktrace to JfrStackTraceRepository, and
> resumes the thread (e.g. on linux resumption involves setting up
> a resume request and again sending a SIGUSR2 to the thread to
> get it to handle it and resume).
> 
> We've been contemplating a change to make thread sampling use
> Posix timers instead, such that each thread would use a separate
> timer, and threads would receive signals based on their CPU
> consumption (the timer, e.g. created with timer_create(2), would
> be clocked by the thread CPU time of their associated threads,
> and signal their threads when that CPU time reaches a level
> [of e.g. 10 msec]). The signal handler will then perform the
> stacktrace sampling locally on the  thread, and deliver the
> stacktrace to JfrStackTraceRepository (either directly or by
> enqueing through an intermediary).
> 
> There are multiple potential benefits that may arise from switching
> to such a scheme, including significant reduction of sampling cost,
> improvement of density and focus of samples (fewer lost samples,
> ensuring that enough activity in a given thread will end up leading
> to a sample for that thread, etc.), and, potentially, an ability to
> (with additional  changes) better account for time spent "outside
> of java" in e.g. native and runtime code.
> 
> Has this (using thread-cpu-time-based posix timer sampling) been
> considered before?
> 


[prev in list] [next in list] [prev in thread] [next in thread]