[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: Discuss the design of parallel and incremental jmap histo.
From: linzang(臧琳) <linzang () tencent ! com>
Date: 2019-12-17 2:57:08
Message-ID: C00B6677-2BD8-42E0-A6F3-7AEC00744891 () tencent ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Dear Chris,
I think I can first make the patch of parallel jmap. it seems to me that if \
parallel is fast enough, there is no need for incremental. So l will not work on it \
until I found new cases that show it is necessary, then we can discuss it again. \
Thanks!
BRs,
Lin
On Dec 17, 2019, at 3:09 AM, Chris Plummer \
<chris.plummer@oracle.com<mailto:chris.plummer@oracle.com>> wrote:
On 12/15/19 5:38 PM, linzang(臧琳) wrote:
Dear Chris,
> > why jmap is getting stuck or "killed by timer" as you mention below. Shouldn't \
> > this be considered a bug and addressed directly.
This "timer" is usually another process, my experience is HDFS and ZKFC, the ZKFC \
pings it's NameNode periodically, and when the NameNode's heap is large (~180GB in my \
case), the heap iteration by jmap can cause the process stuck, so ZFKC can not get \
response from NameNode, so the NameNode got killed.
This is the first I've heard mentioned of any of these. I assume NameNode is the \
process you are getting the heap dump from, and while dumping the heap it can't \
respond to ZKFC? That still sounds like something to me that should be addressed \
directly, and not worked around with the incremental solution. The parallel solution \
is ok because it also has a performance benefit, so if as a side affect it helps \
prevent the timeout issue, then that's ok.
> > How useful are intermediate results? How often can users come to reasonable \
> > conclusions about heap usage when the data is incomplete.
From my experience, I usually use jmap -histo to get the information about the \
distribution of objects, and I found usually the object distribution of part of the \
heap is similar about the distribution of the whole heap. I agree that this is not \
correct for all cases, but since jmap -histo give results only when it's exit \
normally at present, and I think maybe info of partial heap is better than nothing, \
especially for memory leak analysis.
Ok, but I still think avoiding the need for incremental dumps would be a better \
approach.
thanks,
Chris
> > Is there even an indication given of how much of the heap is accounted for in the \
> > output?
Yes, the incremental dump information shows the number of the objects and the totally \
bytes have been iterated.
Thanks!
BRs,
Lin
*From: *Chris Plummer <chris.plummer@oracle.com<mailto:chris.plummer@oracle.com>>
*Date: *Saturday, December 14, 2019 at 2:46 AM
*To: *"linzang(臧琳)" <linzang@tencent.com<mailto:linzang@tencent.com>>, \
"serviceability-dev@openjdk.java.net<mailto:serviceability-dev@openjdk.java.net>" \
<serviceability-dev@openjdk.java.net<mailto:serviceability-dev@openjdk.java.net>>
*Subject: *Re: Discuss the design of parallel and incremental jmap histo.(Internet \
mail)
Hi Lin,
I have a question regarding the need for incremental support. The CSR states:
Problem: Now, the "JMap -histo" tool can not dump intermediate result, which is \
useful if the heap is large and dumping the whole heap can be stuck.
Two questions. The first is why jmap is getting stuck or "killed by timer" as you \
mention below. Shouldn't this be considered a bug and addressed directly. Second \
question is how useful are intermediate results? How often can users come to \
reasonable conclusions about heap usage when the data is incomplete. Is there even an \
indication given of how much of the heap is accounted for in the output?
thanks,
Chris
On 12/12/19 10:22 PM, linzang(臧琳) wrote:
Dear All,
I want to re-activate the thread of discussion about the
implementation of parallel and incremental "Jmap -histo".
The target of these changes is to solve the problems that "jmap
-histo" may " timeout or killed by timer" when heap is large. And
the result of "jmap -histo" is "one or nothing", which means if it
gets killed before exit, user gets no information about the heap.
The "incremental" means that jmap -histo dumps the intermediate
results when it is iterating the heap, so if it is interrupted,
user can get some meaningful information.
The "parallel" targets to help speed up the heap iteration with
multi-threading.
Originally I have implemented the "incremental dump" that dump the
intermediate data into a separate file like
<IncrementalHisto.dump>, and the final result will be saved to
another file <HistoResult.dump>. so when jmap -histo get
interrupted, user can get information from
<IncrementalHisto.dump>, and if jmap -histo works fine, the final
result would be in <HistoResult.dump>.
And the parallel dump will have multiple thread working on heap
iteration, each thread generates intermediate data timely.
The main reason of using separate file for incremental dump is
due to the consideration of parallel incremental dump
implementation, so that every heap-iteration thread could dump its
own data in separate file, to avoid using file lock.
However, it seems that the original design might confuse user by
having two or more result files (intermediated result and final
result). So I want to ask your help to discuss it:
1. For incremental dump without parallel, Intermediate result and
the final result are dumped to the same file:
In this case, the intermediate data are generated in the middle of
heap iteration, they are written to file <HistoResult.dump> at the
same time. And if jmap -histo exits normally, the final result
will be also dump to <HistoResult.dump>, then all intermediate
data are flushed.
2. For parallel dump without incremental:
Every thread generates its own thread-local dump buffer, and all
thread local dump are merged and write to the <HistoResult.dump>
file at the end.
There is no incremental support, so the result is "one or nothing".
3. For parallel + incremental dump, I think it's a little
complicated because of intermediate data processing:
1. Every thread has its own thread-local intermediate data
buffer, and all the thread-local buffers will be written
to <HistoResult.dump> file while holding file lock. So
there is only one data file generated, and if jmap -histo
is interrupted, the intermediated data are save in the
same file.
The problem is that the file write lock can be heavy, which may
cause parallel heap dump slow.
2. Every thread has its own thread-local intermediate data
buffer, and every thread save its result in an temp file
named <IntermediatedResult_[tid].dump>.
So there is no file lock. The parallel can be fast. But the
problem is that there will be multiple files generated to save the
thread-local intermediate results. And this might confuse the user.
3. Every thread has its own thread-local intermediate data
buffer, and another "data-merging-thread" will be generated.
The parallel threads write data to its thread local buffer, and
enqueue the buffer when data reach some threshold. The
"data-merging-thread" consumes the queue, merge the data from
different thread, save the merged data to the result file.
In this case, there is only one <HistoResult.dump> file generated.
And there is no file lock needed, but there is queue lock, and a
separate "merging thread" impl. Do you think this is a reasonable
solution?
So may I ask your suggestion ?
Details of previous discussion can be found at
https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-June/028276.html
Thanks!
BRs,
Lin
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> Dear Chris,
<div class=""> I think I can first make the patch of parallel jmap. it \
seems to me that if parallel is fast enough, there is no need for incremental. So l \
will not work on it until I found new cases that show it is necessary, then we can \
discuss it again.</div> <div class=""> Thanks!</div>
<div class=""><br class="">
</div>
<div class="">BRs,</div>
<div class="">Lin</div>
<div class=""><br class="">
</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 17, 2019, at 3:09 AM, Chris Plummer <<a \
href="mailto:chris.plummer@oracle.com" class="">chris.plummer@oracle.com</a>> \
wrote:</div> <br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; \
font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" class="">On \
12/15/19 5:38 PM, linzang(臧琳) wrote:</span><br style="caret-color: rgb(0, 0, 0); \
font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote \
type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: \
auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br class="">
Dear Chris,<br class="">
<br class="">
>> why jmap is getting stuck or "killed by timer" as you mention \
below. Shouldn't this be considered a bug and addressed directly.<br class=""> <br \
class=""> This "timer" is usually another process, my experience is HDFS and ZKFC, \
the ZKFC pings it's NameNode periodically, and when the NameNode's heap is large \
(~180GB in my case), the heap iteration by jmap can cause the process stuck, so ZFKC \
can not get response from NameNode, so the NameNode got killed.<br class="">
<br class="">
</blockquote>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">This is the first I've heard \
mentioned of any of these. I assume NameNode is the process you are getting the heap \
dump from, and while dumping the heap it can't respond to ZKFC? That still sounds \
like something to me that should be addressed directly, and not worked around with \
the incremental solution. The parallel solution is ok because it also has a \
performance benefit, so if as a side affect it helps prevent the timeout issue, then \
that's ok.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; \
font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""> <blockquote type="cite" style="font-family: \
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; \
font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; \
text-indent: 0px; text-transform: none; white-space: normal; widows: auto; \
word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""> <br class="">
>> How useful are intermediate results? How often can users come to \
reasonable conclusions about heap usage when the data is incomplete.<br class=""> <br \
class=""> From my experience, I usually use jmap -histo to get the information \
about the distribution of objects, and I found usually the object distribution of \
part of the heap is similar about the distribution of the whole heap. I agree that \
this is not correct for all cases, but since jmap -histo give results only when it's \
exit normally at present, and I think maybe info of partial heap is better than \
nothing, especially for memory leak analysis.<br class=""> <br class="">
</blockquote>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
float: none; display: inline !important;" class="">Ok, but I still think avoiding \
the need for incremental dumps would be a better approach.</span><br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); \
font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline \
!important;" class="">thanks,</span><br style="caret-color: rgb(0, 0, 0); \
font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: \
normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: \
0px; text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br \
style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; \
font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" \
class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: \
12px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; float: none; display: inline !important;" \
class="">Chris</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; \
font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""> <blockquote type="cite" style="font-family: \
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; \
font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; \
text-indent: 0px; text-transform: none; white-space: normal; widows: auto; \
word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; \
text-decoration: none;" class=""> <br class="">
>> Is there even an indication given of how much of the heap is accounted for \
in the output?<br class=""> <br class="">
Yes, the incremental dump information shows the number of the objects and the totally \
bytes have been iterated.<br class=""> <br class="">
Thanks!<br class="">
<br class="">
BRs,<br class="">
<br class="">
Lin<br class="">
<br class="">
*From: *Chris Plummer <<a href="mailto:chris.plummer@oracle.com" \
class="">chris.plummer@oracle.com</a>><br class="">
*Date: *Saturday, December 14, 2019 at 2:46 AM<br class="">
*To: *"linzang(臧琳)" <<a href="mailto:linzang@tencent.com" \
class="">linzang@tencent.com</a>>, "<a \
href="mailto:serviceability-dev@openjdk.java.net" \
class="">serviceability-dev@openjdk.java.net</a>" <<a \
href="mailto:serviceability-dev@openjdk.java.net" \
class="">serviceability-dev@openjdk.java.net</a>><br class="">
*Subject: *Re: Discuss the design of parallel and incremental jmap histo.(Internet \
mail)<br class=""> <br class="">
Hi Lin,<br class="">
<br class="">
I have a question regarding the need for incremental support. The CSR states:<br \
class=""> <br class="">
Problem: Now, the "JMap -histo" tool can not dump intermediate result, \
which is useful if the heap is large and dumping the whole heap can be stuck.<br \
class=""> <br class="">
Two questions. The first is why jmap is getting stuck or "killed by timer" \
as you mention below. Shouldn't this be considered a bug and addressed directly. \
Second question is how useful are intermediate results? How often can users come to \
reasonable conclusions about heap usage when the data is incomplete. Is there even \
an indication given of how much of the heap is accounted for in the output?<br \
class=""> <br class="">
thanks,<br class="">
<br class="">
Chris<br class="">
<br class="">
On 12/12/19 10:22 PM, linzang(臧琳) wrote:<br class="">
<br class="">
Dear All,<br class="">
<br class="">
I want to re-activate the thread of discussion about \
the<br class=""> implementation of parallel and incremental "Jmap \
-histo".<br class=""> <br class="">
The target of these changes is to solve the \
problems that "jmap<br class=""> -histo" may " timeout or killed by \
timer" when heap is large. And<br class=""> the result of "jmap \
-histo" is "one or nothing", which means if it<br class=""> gets \
killed before exit, user gets no information about the heap.<br class=""> <br \
class=""> The "incremental" means that jmap -histo \
dumps the intermediate<br class=""> results when it is iterating \
the heap, so if it is interrupted,<br class=""> user can get some \
meaningful information.<br class=""> <br class="">
The "parallel" targets to help speed up the heap \
iteration with<br class=""> multi-threading.<br class="">
<br class="">
Originally I have implemented the "incremental dump" that dump \
the<br class=""> intermediate data into a separate file like<br \
class=""> <IncrementalHisto.dump>, and the final result will \
be saved to<br class=""> another file <HistoResult.dump>. so \
when jmap -histo get<br class=""> interrupted, user can get \
information from<br class=""> <IncrementalHisto.dump>, and if \
jmap -histo works fine, the final<br class=""> result would be in \
<HistoResult.dump>.<br class=""> <br class="">
And the parallel dump will have multiple thread \
working on heap<br class=""> iteration, each thread generates \
intermediate data timely.<br class=""> <br class="">
The main reason of using separate file for incremental \
dump is<br class=""> due to the consideration of parallel \
incremental dump<br class=""> implementation, so that every \
heap-iteration thread could dump its<br class=""> own data in \
separate file, to avoid using file lock.<br class=""> <br class="">
However, it seems that the original design might confuse user by<br \
class=""> having two or more result files (intermediated result and \
final<br class=""> result). So I want to ask your help to \
discuss it:<br class=""> <br class="">
1. For incremental dump without parallel, Intermediate result \
and<br class=""> the final result are \
dumped to the same file:<br class=""> <br class="">
In this case, the intermediate data are generated in the middle \
of<br class=""> heap iteration, they are written to file \
<HistoResult.dump> at the<br class=""> same time. And if jmap \
-histo exits normally, the final result<br class=""> will be also \
dump to <HistoResult.dump>, then all intermediate<br class=""> \
data are flushed.<br class=""> <br class="">
2. For parallel dump without incremental:<br class="">
<br class="">
Every thread generates its own thread-local dump buffer, and all<br \
class=""> thread local dump are merged and write to the \
<HistoResult.dump><br class=""> file at the end.<br class="">
<br class="">
There is no incremental support, so the result is "one or \
nothing".<br class=""> <br class="">
3. For parallel + incremental dump, I think it's a \
little<br class=""> complicated because of \
intermediate data processing:<br class=""> <br class="">
1. Every thread has its own \
thread-local intermediate data<br class=""> \
buffer, and all the \
thread-local buffers will be written<br class=""> \
to \
<HistoResult.dump> file while holding file lock. So<br class=""> \
there is only one \
data file generated, and if jmap -histo<br class=""> \
is interrupted, \
the intermediated data are save in the<br class=""> \
same file.<br \
class=""> <br class="">
The problem is that the file write lock can be heavy, which may<br \
class=""> cause parallel heap dump slow.<br class="">
<br class="">
2. Every thread has its own \
thread-local intermediate data<br class=""> \
buffer, and every \
thread save its result in an temp file<br class=""> \
named \
<IntermediatedResult_[tid].dump>.<br class=""> <br class="">
So there is no file lock. The parallel can be fast. But \
the<br class=""> problem is that there will be multiple files \
generated to save the<br class=""> thread-local intermediate \
results. And this might confuse the user.<br class=""> <br class="">
3. Every thread has its own \
thread-local intermediate data<br class=""> \
buffer, and another \
"data-merging-thread" will be generated.<br class=""> <br class="">
The parallel threads write data to its thread local buffer, and<br \
class=""> enqueue the buffer when data reach some threshold. The<br \
class=""> "data-merging-thread" consumes the queue, merge the data \
from<br class=""> different thread, save the merged data to the \
result file.<br class=""> <br class="">
In this case, there is only one <HistoResult.dump> file \
generated.<br class=""> And there is no file lock needed, but there \
is queue lock, and a<br class=""> separate "merging thread" impl. \
Do you think this is a reasonable<br class=""> solution?<br \
class=""> <br class="">
So may I ask your suggestion ?<br class="">
<br class="">
Details of previous discussion can be found at<br class="">
<a \
href="https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-June/028276.html" \
class="">https://mail.openjdk.java.net/pipermail/serviceability-dev/2019-June/028276.html</a><br \
class=""> <br class="">
Thanks!<br class="">
<br class="">
BRs,<br class="">
<br class="">
Lin</blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic