[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: nodemanager process crash for array index out of bound
From: yaoxiaohua <yaoxiaohua () outlook ! com>
Date: 2015-12-18 3:02:30
Message-ID: BAY176-DS204676E956A645DF38F350C8E10 () phx ! gbl
[Download RAW message or body]
Hi guys,
Nodemanager process error log:
Could you give me some tips for this , why happened?
2015-12-16 17:24:38,605 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl: Uncaught exception in ContainerMemoryManager while managing
memory of container_1448915696877_23390_01_000430
java.lang.ArrayIndexOutOfBoundsException
at
sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:848)
at
sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
at
java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
at java.util.Formatter.format(Formatter.java:2500)
at java.util.Formatter.format(Formatter.java:2435)
at java.lang.String.format(String.java:2148)
at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
at
org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(Strin
gUtils.java:758)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:4
87)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-16 17:24:39,663 ERROR org.apache.hadoop.mapred.ShuffleHandler:
Shuffle error:
java.lang.ArrayIndexOutOfBoundsException
at sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:652)
at java.lang.StringCoding.encode(StringCoding.java:899)
at java.lang.String.getBytes(String.java:2221)
at
org.apache.hadoop.mapreduce.security.SecureShuffleUtils.verifyReply(SecureSh
uffleUtils.java:100)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.verifyRequest(ShuffleHandler
.java:558)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandl
er.java:499)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWri
teHandler.java:142)
at
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpC
hunkAggregator.java:148)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceive
d(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDe
coder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Replay
ingDecoder.java:435)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:107)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:88)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
57)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
27)
at java.lang.Thread.run(Thread.java:809)
2015-12-16 17:24:39,663 ERROR org.apache.hadoop.mapred.ShuffleHandler:
Shuffle error [id: 0x34ed4c97, /172.19.206.148:34641 =>
/172.19.206.142:8080] EXCEPTION: java.lang.ArrayIndexOutOfBoundsException
2015-12-16 17:24:39,663 FATAL
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Container
Monitor,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java 堆空间(heapspace)
at java.util.HashMap.inflateTable(HashMap.java:328)
at java.util.HashMap.<init>(HashMap.java:308)
at
org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsB
asedProcessTree.java:154)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:390)
2015-12-16 17:24:39,666 INFO org.apache.hadoop.util.ExitUtil: Halt with
status -1 Message: HaltException
Best Regards,
Evan Yao
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type \
content="text/html; charset=gb2312"><meta name=Generator content="Microsoft Word 14 \
(filtered medium)"><style><!-- /* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div \
class=WordSection1><p class=MsoNormal>Hi guys,<o:p></o:p></p><p \
class=MsoNormal>Nodemanager process error log:<o:p></o:p></p><p class=MsoNormal>Could \
you give me some tips for this , why happened?<o:p></o:p></p><p \
class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2015-12-16 17:24:38,605 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: \
Uncaught exception in ContainerMemoryManager while managing memory of \
container_1448915696877_23390_01_000430<o:p></o:p></p><p \
class=MsoNormal>java.lang.ArrayIndexOutOfBoundsException<o:p></o:p></p><p \
class=MsoNormal> at \
sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:848)<o:p></o:p></p><p \
class=MsoNormal> at \
sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter.format(Formatter.java:2500)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.Formatter.format(Formatter.java:2435)<o:p></o:p></p><p \
class=MsoNormal> at \
java.lang.String.format(String.java:2148)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorIm \
pl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)<o:p></o:p></p><p \
class=MsoNormal>2015-12-16 17:24:39,663 ERROR \
org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: <o:p></o:p></p><p \
class=MsoNormal>java.lang.ArrayIndexOutOfBoundsException<o:p></o:p></p><p \
class=MsoNormal> at \
sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:652)<o:p></o:p></p><p \
class=MsoNormal> at \
java.lang.StringCoding.encode(StringCoding.java:899)<o:p></o:p></p><p \
class=MsoNormal> at \
java.lang.String.getBytes(String.java:2221)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.mapreduce.security.SecureShuffleUtils.verifyReply(SecureShuffleUtils.java:100)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.verifyRequest(ShuffleHandler.java:558)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:499)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)<o:p></o:p></p><p \
class=MsoNormal> at \
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)<o:p></o:p></p><p \
class=MsoNormal> at \
java.lang.Thread.run(Thread.java:809)<o:p></o:p></p><p class=MsoNormal>2015-12-16 \
17:24:39,663 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: \
0x34ed4c97, /172.19.206.148:34641 => /172.19.206.142:8080] EXCEPTION: \
java.lang.ArrayIndexOutOfBoundsException<o:p></o:p></p><p class=MsoNormal>2015-12-16 \
17:24:39,663 FATAL org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread \
Thread[Container Monitor,5,main] threw an Error. Shutting down \
now...<o:p></o:p></p><p class=MsoNormal>java.lang.OutOfMemoryError: Java <span \
lang=ZH-CN style='font-family:宋体'>堆空间</span>(heapspace)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.HashMap.inflateTable(HashMap.java:328)<o:p></o:p></p><p \
class=MsoNormal> at \
java.util.HashMap.<init>(HashMap.java:308)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsBasedProcessTree.java:154)<o:p></o:p></p><p \
class=MsoNormal> at \
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:390)<o:p></o:p></p><p \
class=MsoNormal>2015-12-16 17:24:39,666 INFO org.apache.hadoop.util.ExitUtil: Halt \
with status -1 Message: HaltException<o:p></o:p></p><p \
class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p \
class=MsoNormal>Best Regards,<o:p></o:p></p><p class=MsoNormal>Evan \
Yao<o:p></o:p></p></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic