[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    =?gb2312?B?16q3ojogREZTQ2xpZW50IGdvdCBkZWFkbG9jayB3aGVuIGNsb3NlIGZpbGUg?= =?gb2312?Q?and_failed_to_r
From:       <daniedeng () tencent ! com>
Date:       2015-10-19 1:43:08
Message-ID: DC1EB8E42E743A4DA247CDF9755B6B923619C817 () EXMBX-SZMAIL016 ! tencent ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

[Attachment #4 (text/plain)]



________________________________
[cid:_Foxmail.1@8432ab7f-0601-e295-1353-b37103af40d9]daniedeng(µΛ·Ι)

·ΆΌώΘΛ£Ί daniedeng(µΛ·Ι)<mailto:daniedeng@tencent.com>
·ΆΛΝΚ±Όδ£Ί 2015-10-16 15:44
ΚΥΌώΘΛ£Ί hdfs-issues<mailto:hdfs-issues@hadoop.apache.org>; \
user@hadoop.apache.org<mailto:user@hadoop.apache.org> ΦχΜβ£Ί DFSClient got deadlock \
when close file and failed to renew lease Hi,All
    We found a deadlock at our HBase(0.98) cluster(and the Hadoop Version is \
2.2.0),and it should be HDFS BUG,at the time our network is not stable.  below is the \
stack:

*************************************************************************************************************************************
 Found one Java-level deadlock:
=============================
"MemStoreFlusher.1":
  waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe0, a \
org.apache.hadoop.hdfs.LeaseRenewer),  which is held by \
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel" \
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":  waiting to lock monitor \
0x00007ff2e67e16a8 (object 0x0000000486ce6620, a \
org.apache.hadoop.hdfs.DFSOutputStream),  which is held by "MemStoreFlusher.0"
"MemStoreFlusher.0":
  waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe0, a \
org.apache.hadoop.hdfs.LeaseRenewer),  which is held by \
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"

Java stack information for the threads listed above:
===================================================
"MemStoreFlusher.1":
at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
- locked <0x000000055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
 at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
 at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
at org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
 at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
                
- locked <0x000000059869eed8> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
 at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)
 at java.lang.Thread.run(Thread.java:744)
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
at org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
- waiting to lock <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStream)
at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
at org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
- locked <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:744)
"MemStoreFlusher.0":
at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
- locked <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStream)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
 at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
 at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
at org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
 at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
                
- locked <0x00000004888f6848> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
 at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:435)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:66)
 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:253)
 at java.lang.Thread.run(Thread.java:744)

Found 1 deadlock.

**********************************************************************


the thread "MemStoreFlusher.0" is closing outputStream and remove it's lease ;
other side the daemon thread "LeaseRenewer" failed to connect active nn  for renewing \
lease,but  got SocketTimeoutException   cause of network is not good,so abort \
outputstream. then deadlock is made.

and it seems not solved at Hadoop 2.7.1 .If confirmed , we can fixed the issue.


[cid:_Foxmail.1@810316d7-f17c-e968-42de-ed0367db1624]
daniedeng(µΛ·Ι)


[Attachment #5 (text/html)]

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; \
margin-left: 0.5em; }div.foxdiv20151019094151204378 { }body { font-size: 10.5pt; \
font-family: ΞΆΘνΡΕΊΪ; color: rgb(0, 0, 0); line-height: 1.5; }</style> </head>
<body>
<div><span></span><br>
</div>
<div><br>
</div>
<hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span>
<div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">
<div><img src="cid:_Foxmail.1@8432ab7f-0601-e295-1353-b37103af40d9" border="0" \
style="font-size: 10pt; line-height: 1.5; background-color: \
window;">daniedeng(µΛ·Ι)</div> </div>
</span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;">
<div>&nbsp;</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: \
12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; \
PADDING-TOP: 8px"> <div><b>·ΆΌώΘΛ£Ί</b>&nbsp;<a \
href="mailto:daniedeng@tencent.com">daniedeng(µΛ·Ι)</a></div> \
<div><b>·ΆΛΝΚ±Όδ£Ί</b>&nbsp;2015-10-16&nbsp;15:44</div> <div><b>ΚΥΌώΘΛ£Ί</b>&nbsp;<a \
href="mailto:hdfs-issues@hadoop.apache.org">hdfs-issues</a>; <a \
href="mailto:user@hadoop.apache.org"> user@hadoop.apache.org</a></div>
<div><b>ΦχΜβ£Ί</b>&nbsp;DFSClient got deadlock when close file and failed to renew \
lease</div> </div>
</div>
<div>
<div class="FoxDiv20151019094151204378">
<div><span></span><span style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; \
line-height: normal; widows: 1;">Hi,All</span> <div style="font-family: ΞΆΘνΡΕΊΪ, \
sans-serif; font-size: 16px; line-height: normal; widows: 1;"> &nbsp; &nbsp; We found \
a deadlock at our HBase(0.98) cluster(and the Hadoop Version is 2.2.0),and it should \
be HDFS BUG,at the time our network is not stable.</div> <div style="font-family: \
ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; widows: 1;"> &nbsp;below \
is the stack:</div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; \
line-height: normal; widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
                widows: 1;">
*************************************************************************************************************************************</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> Found one Java-level deadlock:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> =============================</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &quot;MemStoreFlusher.1&quot;:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; waiting to lock monitor 0x00007ff27cfa5218 (object \
0x00000002fae5ebe0, a org.apache.hadoop.hdfs.LeaseRenewer),</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; which is held by \
&quot;LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel&quot;</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &quot;LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel&quot;:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; waiting to lock monitor 0x00007ff2e67e16a8 (object \
0x0000000486ce6620, a org.apache.hadoop.hdfs.DFSOutputStream),</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; which is held by &quot;MemStoreFlusher.0&quot;</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &quot;MemStoreFlusher.0&quot;:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; waiting to lock monitor 0x00007ff27cfa5218 (object \
0x00000002fae5ebe0, a org.apache.hadoop.hdfs.LeaseRenewer),</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &nbsp; which is held by \
&quot;LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel&quot;</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> Java stack information for the threads listed above:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> ===================================================</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> &quot;MemStoreFlusher.1&quot;:</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- waiting \
to lock &lt;0x00000002fae5ebe0&gt; (a org.apache.hadoop.hdfs.LeaseRenewer)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- locked \
&lt;0x000000055b606cb0&gt; (a org.apache.hadoop.hdfs.DFSOutputStream)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- locked \
&lt;0x000000059869eed8&gt; (a java.lang.Object)</div> <div style="font-family: \
ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; widows: 1;"> <span \
class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
java.lang.Thread.run(Thread.java:744)</div> <div style="font-family: ΞΆΘνΡΕΊΪ, \
sans-serif; font-size: 16px; line-height: normal; widows: 1;"> \
&quot;LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel&quot;:</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- waiting \
to lock &lt;0x0000000486ce6620&gt; (a org.apache.hadoop.hdfs.DFSOutputStream)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- locked \
&lt;0x00000002fae5ebe0&gt; (a org.apache.hadoop.hdfs.LeaseRenewer)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
java.lang.Thread.run(Thread.java:744)</div> <div style="font-family: ΞΆΘνΡΕΊΪ, \
sans-serif; font-size: 16px; line-height: normal; widows: 1;"> \
&quot;MemStoreFlusher.0&quot;:</div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; \
font-size: 16px; line-height: normal; widows: 1;"> <span class="Apple-tab-span" \
style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- waiting \
to lock &lt;0x00000002fae5ebe0&gt; (a org.apache.hadoop.hdfs.LeaseRenewer)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- locked \
&lt;0x0000000486ce6620&gt; (a org.apache.hadoop.hdfs.DFSOutputStream)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)</div> \
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>- locked \
&lt;0x00000004888f6848&gt; (a java.lang.Object)</div> <div style="font-family: \
ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; widows: 1;"> <span \
class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)</div> <div \
style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:435)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:66)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:253)</div>
 <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span class="Apple-tab-span" style="white-space: pre;"></span>at \
java.lang.Thread.run(Thread.java:744)</div> <div style="font-family: ΞΆΘνΡΕΊΪ, \
sans-serif; font-size: 16px; line-height: normal; widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> Found 1 deadlock.&nbsp;</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
                widows: 1;">
**********************************************************************</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <br>
</div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> the thread&nbsp;<span style="font-size: \
12pt;">&quot;MemStoreFlusher.0&quot; is closing outputStream and remove it's lease \
;</span></div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; \
line-height: normal; widows: 1;"> <span style="font-size: 12pt;">other side \
t</span><span style="font-size: 12pt; background-color: window;">he daemon thread \
&quot;</span><span style="font-size: 12pt; background-color: \
window;">LeaseRenewer</span><span style="font-size: 12pt; background-color: \
window;">&quot;  failed to connect active nn &nbsp;for renewing &nbsp;lease,but \
&nbsp;got SocketTimeoutException &nbsp; cause of network is not good,so abort \
outputstream.</span></div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: \
16px; line-height: normal; widows: 1;"> <span style="font-size: 12pt;">then deadlock \
is made.</span></div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; \
line-height: normal; widows: 1;"> <span style="font-size: 12pt;"><br>
</span></div>
<div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; font-size: 16px; line-height: normal; \
widows: 1;"> <span style="font-size: 12pt;">and it seems not solved \
at&nbsp;</span><span style="font-size: 12pt;">Hadoop 2.7.1 .If confirmed , we can \
fixed the issue.</span></div> <div style="font-family: ΞΆΘνΡΕΊΪ, sans-serif; \
font-size: 16px; line-height: normal; widows: 1;"> <span style="font-size: \
12pt;"><br> </span></div>
</div>
<div>
<div><br>
</div>
<div><span>
<div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">
<div><img src="cid:_Foxmail.1@810316d7-f17c-e968-42de-ed0367db1624" border="0" \
style="font-size: 10pt; line-height: 1.5; background-color: window;"></div> \
<div>daniedeng(µΛ·Ι)</div> </div>
</span></div>
<div><br>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</body>
</html>


["24050_signature (1).png" (image/png)]
["24050_signature (10-19-09-41-51).png" (image/png)]

[prev in list] [next in list] [prev in thread] [next in thread]

Configure | About | News | Add a list | Sponsored by KoreLogic