[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: =?gb2312?B?tPC4tDogQ2FsbCBmb3IgZGlzY3Vzc2lvbjogRXh0ZW5kIEpNYXAgdG8gc3Vw?= =?gb2312?Q?port_parallel_a
From: <zanglin5 () jd ! com>
Date: 2018-11-28 2:47:38
Message-ID: 24357a4ebb1347cb8dcd5e4c199402b4 () jd ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Add serviceability-dev.
Thanks.
________________________________
·¢¼þÈË: ê°ÁÕ
·¢ËÍʱ¼ä: 2018Äê11Ô 28ÈÕ 10:30
ÊÕ¼þÈË: Martijn Verburg
³ËÍ: discuss@openjdk.java.net
Ö÷Ìâ: Re: Call for discussion: Extend JMap to support parallel and incremental heap \
scanning.
Hi Martijn,
Thanks a lot for you guidance! I will go to serviceability-dev and try to join \
the Atlantis discussion.
Cheers,
Lin
·¢¼þÈË: Martijn Verburg [mailto:martijnverburg@gmail.com]
·¢ËÍʱ¼ä: 2018Äê11Ô 27ÈÕ 23:18
ÊÕ¼þÈË: ê°ÁÕ <zanglin5@jd.com>
³ËÍ: discuss@openjdk.java.net
Ö÷Ìâ: Re: Call for discussion: Extend JMap to support parallel and incremental heap \
scanning.
Hi Lin and welcome to OpenJDK!
There's an ongoing thread discussing project Atlantis \
(http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html) which I \
think will be looking at potential enhancements like this alongside the \
serviceability group.
In the meantime, I think the serviceability-dev mailing list is the right place to \
start your discussion.
Cheers,
Martijn
On Tue, 27 Nov 2018 at 15:08, ê°ÁÕ <zanglin5@jd.com<mailto:zanglin5@jd.com>> wrote:
Hello,
This is Zang Lin, JVM developer in JD.COM<http://JD.COM>. My department just \
signed OCA and we will try to contribute to OpenJDK. As I am newbie at openJDK, I'd \
like to discuss some of my ideas and ask for suggestions.
Recently I tried to use "jmap -histo" to collect histogram info on a large heap \
at ~200G bytes. it took about 160 seconds, which actually cause our online system \
suffering "non-responsive" issue. In addition, if we stop jmap with "kill", jmap \
exits directly (even the attached JVM still working on heap iteration), leave no help \
info for analysis.
To address the issue above, I proposed following approaches:
1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in \
parallel and save the intermediate results incrementally during heap scanning \
(controlled by threshold to decide when to dump).
2) Make Jmap to accept arguments that enable/disable parallel and partial heap \
iterating, also make parallel thread number configurable.
3) Make Jmap to dump results to specified file path given by argument, rather than \
output to stdout directly.
A prototype has been made internally with G1 upon JDK 11, and our preliminary \
measurement show that parallel iterate ~200GB heap with 4 threads can save ~30-40 \
seconds, and I am also analyzing the approach to make the time more scaled with \
parallel threads. Moreover, with the implementation of incremental dump, we can get \
usefully partial dump info for analyzing, even when jmap hang for a long time, or get \
killed for some reason.
Several problems were encountered during development. One problem is that it is \
not easy to dynamically allocate some data structures for thread local operations, \
such as KlassInfoTable,which is designed to be StackObj that not allowed to be \
allocated using "new", another problem is that some base classes definition in \
hotspot has to be modified to support parallel operations, such as ObjectClosure etc. \
I am not sure whether these modifications can be acceptable.
Any suggestions/guidance would be much appreciated!
BRs,
Lin
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} \
--></style> </head>
<body dir="ltr">
<div id="divtagdefaultwrapper" \
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" \
dir="ltr"> <p>Add serviceability-dev. </p>
<p>Thanks.</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" \
style="font-size:11pt"><b>·¢¼þÈË:</b> ê°ÁÕ<br> <b>·¢ËÍʱ¼ä:</b> 2018Äê11Ô 28ÈÕ \
10:30<br> <b>ÊÕ¼þÈË:</b> Martijn Verburg<br>
<b>³ËÍ:</b> discuss@openjdk.java.net<br>
<b>Ö÷Ìâ:</b> Re: Call for discussion: Extend JMap to support parallel and incremental \
heap scanning.</font> <div> </div>
</div>
<div>
<div style="">
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; color:#1F497D">Hi Martijn, \
</span></p> <p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: \
ËÎÌå;"><span lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; \
color:#1F497D"> Thanks a lot for you guidance! I \
will go to serviceability-dev and try to join the Atlantis discussion.</span></p> <p \
style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; \
color:#1F497D"> </span></p> <p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; \
font-family: ËÎÌå;"><span lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; \
color:#1F497D">Cheers, </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; color:#1F497D">Lin</span></p> \
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; \
color:#1F497D"> </span></p> <p style="text-align: justify; margin: 0cm 0cm \
0.0001pt; font-size: 12pt; font-family: ËÎÌå;"> <span lang="EN-US" \
style="font-size:14.0pt; font-family:"΢ÈíÑźÚ",sans-serif; \
color:#595757"> </span></p> <p style="text-align: justify; margin: 0cm 0cm \
0.0001pt; font-size: 12pt; font-family: ËÎÌå;"> <span lang="EN-US" \
style="font-size:14.0pt; font-family:"΢ÈíÑźÚ",sans-serif; \
color:#595757"> </span></p> <p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; \
font-family: ËÎÌå;"><span lang="EN-US" style="font-size:10.5pt; font-family:µÈÏß; \
color:#1F497D"> </span></p> <p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; \
font-family: ËÎÌå;"><b><span style="font-size:11.0pt; font-family:µÈÏß">·¢¼þÈË<span \
lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:11.0pt; \
font-family:µÈÏß"> Martijn Verburg [mailto:martijnverburg@gmail.com] <br>
</span><b><span style="font-size:11.0pt; font-family:µÈÏß">·¢ËÍʱ¼ä<span \
lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:11.0pt; \
font-family:µÈÏß"> 2018</span><span style="font-size:11.0pt; \
font-family:µÈÏß">Äê<span lang="EN-US">11</span>Ô <span lang="EN-US">27</span>ÈÕ<span \
lang="EN-US"> 23:18<br>
</span><b>ÊÕ¼þÈË<span lang="EN-US">:</span></b><span lang="EN-US"> </span>ê°ÁÕ<span \
lang="EN-US"> <zanglin5@jd.com><br> </span><b>³ËÍ<span \
lang="EN-US">:</span></b><span lang="EN-US"> discuss@openjdk.java.net<br> \
</span><b>Ö÷Ìâ<span lang="EN-US">:</span></b><span lang="EN-US"> Re: Call for \
discussion: Extend JMap to support parallel and incremental heap \
scanning.</span></span></p> <p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; \
font-family: ËÎÌå;"><span lang="EN-US"> </span></p> <div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US">Hi Lin and welcome to OpenJDK!</span></p> <div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US"> </span></p> </div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US">There's an ongoing thread discussing project Atlantis (<a \
href="http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html" \
style="color: blue; text-decoration: underline;" id="LPlnk621507" \
previewremoved="true">http://mail.openjdk.java.net/pipermail/discuss/2018-November/004904.html</a>)
which I think will be looking at potential enhancements like this alongside the \
serviceability group.</span></p> </div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US"> </span></p> </div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US">In the meantime, I think the serviceability-dev mailing list \
is the right place to start your discussion.</span></p> </div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US"><br clear="all"> </span></p>
<div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US">Cheers,<br> Martijn</span></p>
</div>
</div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US"> </span></p> </div>
</div>
</div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US"> </span></p> <div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: ËÎÌå;"><span \
lang="EN-US">On Tue, 27 Nov 2018 at 15:08, </span>ê°ÁÕ<span lang="EN-US"> <<a \
href="mailto:zanglin5@jd.com" style="color: blue; text-decoration: \
underline;">zanglin5@jd.com</a>> wrote:</span></p> </div>
<blockquote style="border:none; border-left:solid #CCCCCC 1.0pt; padding:0cm 0cm 0cm \
6.0pt; margin-left:4.8pt; margin-right:0cm"> <p style="margin: 0cm 0cm 0.0001pt; \
font-size: 12pt; font-family: ËÎÌå;"><span lang="EN-US"> Hello,<br> <br>
This is Zang Lin, JVM developer in <a href="http://JD.COM" \
target="_blank" style="color: blue; text-decoration: underline;"> JD.COM</a>. My \
department just signed OCA and we will try to contribute to OpenJDK. As I \
am newbie at openJDK, I'd like to discuss some of my ideas and ask for \
suggestions.<br> <br>
<br>
Recently I tried to use "jmap -histo" to collect histogram \
info on a large heap at ~200G bytes. it took about 160 seconds, which actually \
cause our online system suffering "non-responsive" issue. In \
addition, if we stop jmap with "kill", jmap exits directly (even the \
attached JVM still working on heap iteration), leave no help info for analysis.<br> \
<br> <br>
<br>
To address the issue above, I proposed following approaches:<br>
<br>
1) Extend Jmap to be parallel and incremental, which means Jmap iterates heap in \
parallel and save the intermediate results incrementally during heap scanning \
(controlled by threshold to decide when to dump).<br> <br>
2) Make Jmap to accept arguments that enable/disable parallel and partial heap \
iterating, also make parallel thread number configurable.<br> <br>
3) Make Jmap to dump results to specified file path given by argument, rather than \
output to stdout directly.<br> <br>
<br>
<br>
A prototype has been made internally with G1 upon JDK 11, and our \
preliminary measurement show that parallel iterate ~200GB heap with 4 threads can \
save ~30-40 seconds, and I am also analyzing the approach to make the time more \
scaled with parallel threads. Moreover, with the implementation of incremental \
dump, we can get usefully partial dump info for analyzing, even when jmap hang for a \
long time, or get killed for some reason.<br> <br>
<br>
<br>
Several problems were encountered during development. One problem is \
that it is not easy to dynamically allocate some data structures for thread local \
operations, such as KlassInfoTable,which is designed to be StackObj that not allowed \
to be allocated using "new", another problem is that some base classes \
definition in hotspot has to be modified to support parallel operations, such as \
ObjectClosure etc. I am not sure whether these modifications can be acceptable.<br> \
<br> <br>
<br>
Any suggestions/guidance would be much appreciated!<br>
<br>
<br>
<br>
BRs,<br>
<br>
Lin<br>
<br>
<br>
</span></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic