[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: flume use quite a lot of direct momery and blast
From: Brock Noland <brock () cloudera ! com>
Date: 2014-01-22 14:10:36
Message-ID: CAFukC=5y=K_CFZk=3RwGmuP+CdUgDfii0F8aJPWYD_WL2=M1Fg () mail ! gmail ! com
[Download RAW message or body]
Are you using hdfs sink? If so please share your configuration.
I have seen this when the sink keeps many open files...each file holding on
to a native compression buffer.
On Jan 22, 2014 1:37 AM, "Shangan Chen" <chenshangan521@gmail.com> wrote:
> Thanks a lot for your reply. I deployed flume on ubuntu and what confused
> me is not about the virtual address space but NIO Direct Memory increasing
> boundlessly. It will finally cause the machine to report swap is low.
>
>
> On Wed, Jan 22, 2014 at 1:08 AM, Brock Noland <brock@cloudera.com> wrote:
>
> > Looks like it's arena allocation. Basically it's nothing to worry about
> > since virtual address space on 64bit machines is not in short supply. You
> > could limit it with a parameter like so:
> >
> >
> > https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
> >
> >
> > On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <chenshangan521@gmail.com>wrote:
> >
> > > I have quite a lot of flume-agents streaming logs to several
> > > flume-collectors. The problem I face is the memory consumed by
> > > flume-collector is increasing all the time but slowly even I limit the max
> > > heap. I know flume use NIO, but I don't know why it cause memory increase
> > > boundlessly. I prints some information I can get, I really appreciate if
> > > anyone can give any help.
> > >
> > > *my jvm conf:*
> > >
> > > JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
> > > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
> > >
> > > *top prints of flume instance:*
> > >
> > > top - 17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86,
> > > 0.79
> > > Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
> > > Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si,
> > > 0.4%st
> > > Mem: 16435540k total, 16194572k used, 240968k free, 25896k buffers
> > > Swap: 8385892k total, 205592k used, 8180300k free, 165968k cached
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 java
> > >
> > >
> > > *jmap prints :*
> > >
> > > jmap -heap 16580
> > > Attaching to process ID 16580, please wait...
> > > Debugger attached successfully.
> > > Server compiler detected.
> > > JVM version is 23.21-b01
> > >
> > > using thread-local object allocation.
> > > Parallel GC with 8 thread(s)
> > >
> > > Heap Configuration:
> > > MinHeapFreeRatio = 40
> > > MaxHeapFreeRatio = 70
> > > MaxHeapSize = 8589934592 (8192.0MB)
> > > NewSize = 1310720 (1.25MB)
> > > MaxNewSize = 17592186044415 MB
> > > OldSize = 5439488 (5.1875MB)
> > > NewRatio = 2
> > > SurvivorRatio = 8
> > > PermSize = 21757952 (20.75MB)
> > > MaxPermSize = 85983232 (82.0MB)
> > > G1HeapRegionSize = 0 (0.0MB)
> > >
> > > Heap Usage:
> > > PS Young Generation
> > > Eden Space:
> > > capacity = 2105147392 (2007.625MB)
> > > used = 1048958560 (1000.3648376464844MB)
> > > free = 1056188832 (1007.2601623535156MB)
> > > 49.828271596861185% used
> > > From Space:
> > > capacity = 369229824 (352.125MB)
> > > used = 369168400 (352.06642150878906MB)
> > > free = 61424 (0.0585784912109375MB)
> > > 99.98336429074591% used
> > > To Space:
> > > capacity = 388890624 (370.875MB)
> > > used = 0 (0.0MB)
> > > free = 388890624 (370.875MB)
> > > 0.0% used
> > > PS Old Generation
> > > capacity = 5726666752 (5461.375MB)
> > > used = 4282458544 (4084.0707244873047MB)
> > > free = 1444208208 (1377.3042755126953MB)
> > > 74.78099790780352% used
> > > PS Perm Generation
> > > capacity = 37879808 (36.125MB)
> > > used = 37833736 (36.08106231689453MB)
> > > free = 46072 (0.04393768310546875MB)
> > > 99.8783731955558% used
> > >
> > > 11441 interned Strings occupying 1025920 bytes.
> > >
> > > *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
> > > <https://gist.github.com/rednaxelafx/1593521>) :*
> > >
> > > java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
> > > Attaching to process ID 16580, please wait...
> > > Debugger attached successfully.
> > > Server compiler detected.
> > > JVM version is 23.21-b01
> > > NIO direct memory: (in bytes)
> > > reserved size = 5864.191787 MB (6149050767 bytes)
> > > max size = 7850.687500 MB (8232042496 bytes)
> > >
> > >
> > >
> > > --
> > > have a good day!
> > > chenshang'an
> > >
> > >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> >
>
>
>
> --
> have a good day!
> chenshang'an
>
>
[Attachment #3 (text/html)]
<p dir="ltr">Are you using hdfs sink? If so please share your configuration. </p>
<p dir="ltr">I have seen this when the sink keeps many open files...each file holding \
on to a native compression buffer. </p> <div class="gmail_quote">On Jan 22, 2014 1:37 \
AM, "Shangan Chen" <<a \
href="mailto:chenshangan521@gmail.com">chenshangan521@gmail.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr">Thanks a lot for \
your reply. I deployed flume on ubuntu and what confused me is not about the virtual \
address space but NIO Direct Memory increasing boundlessly. It will finally cause the \
machine to report swap is low.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 22, 2014 at \
1:08 AM, Brock Noland <span dir="ltr"><<a href="mailto:brock@cloudera.com" \
target="_blank">brock@cloudera.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <div dir="ltr">Looks like it's arena allocation. \
Basically it's nothing to worry about since virtual address space on 64bit \
machines is not in short supply. You could limit it with a parameter like \
so:<div><br></div>
<div><a href="https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d \
0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128" \
target="_blank">https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c0 \
0d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128</a><br>
</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On \
Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <span dir="ltr"><<a \
href="mailto:chenshangan521@gmail.com" \
target="_blank">chenshangan521@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div>I have quite a lot of \
flume-agents streaming logs to several flume-collectors. The problem I face is the \
memory consumed by flume-collector is increasing all the time but slowly even I limit \
the max heap. I know flume use NIO, but I don't know why it cause memory increase \
boundlessly. I prints some information I can get, I really appreciate if anyone can \
give any help.</div>
<div><br></div><div><b>my jvm \
conf:</b></div><div><br></div><div>JAVA_OPTS="-Xms8192m -Xmx8192m \
-Dcom.sun.management.jmxremote -XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/var/sankuai/logs"<br></div><div> <br>
</div><div><b>top prints of flume instance:</b></div><div><br></div><div><div>top - \
17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86, \
0.79</div><div>Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 \
zombie</div>
<div>Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si, \
0.4%st</div><div>Mem: 16435540k total, 16194572k used, 240968k free, 25896k \
buffers</div><div>Swap: 8385892k total, 205592k used, 8180300k free, 165968k \
cached</div>
<div><br></div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ \
COMMAND</div><div>16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 \
java</div></div></div><div><br></div><div><br></div><div><b>jmap prints :</b></div>
<div><br></div><div><div>jmap -heap 16580</div><div>Attaching to process ID 16580, \
please wait...</div><div>Debugger attached successfully.</div><div>Server compiler \
detected.</div><div>JVM version is 23.21-b01</div><div>
<br></div><div>using thread-local object allocation.</div><div>Parallel GC with 8 \
thread(s)</div><div><br></div><div>Heap Configuration:</div><div> MinHeapFreeRatio \
= 40</div><div> MaxHeapFreeRatio = 70</div><div> MaxHeapSize = <a \
href="tel:8589934592" value="+18589934592" target="_blank">8589934592</a> \
(8192.0MB)</div>
<div> NewSize = 1310720 (1.25MB)</div><div> MaxNewSize = \
17592186044415 MB</div><div> OldSize = 5439488 (5.1875MB)</div><div> \
NewRatio = 2</div><div> SurvivorRatio = 8</div><div>
PermSize = 21757952 (20.75MB)</div><div> MaxPermSize = 85983232 \
(82.0MB)</div><div> G1HeapRegionSize = 0 (0.0MB)</div><div><br></div><div>Heap \
Usage:</div><div>PS Young Generation</div><div>Eden Space:</div>
<div> capacity = <a href="tel:2105147392" value="+12105147392" \
target="_blank">2105147392</a> (2007.625MB)</div><div> used = 1048958560 \
(1000.3648376464844MB)</div><div> free = 1056188832 \
(1007.2601623535156MB)</div>
<div> 49.828271596861185% used</div><div>From Space:</div>
<div> capacity = 369229824 (352.125MB)</div><div> used = 369168400 \
(352.06642150878906MB)</div><div> free = 61424 \
(0.0585784912109375MB)</div><div> 99.98336429074591% used</div><div>To \
Space:</div><div> capacity = 388890624 (370.875MB)</div>
<div> used = 0 (0.0MB)</div><div> free = 388890624 (370.875MB)</div><div> \
0.0% used</div><div>PS Old Generation</div><div> capacity = 5726666752 \
(5461.375MB)</div><div> used = 4282458544 (4084.0707244873047MB)</div>
<div> free = 1444208208 (1377.3042755126953MB)</div><div> 74.78099790780352% \
used</div><div>PS Perm Generation</div><div> capacity = 37879808 \
(36.125MB)</div><div> used = 37833736 (36.08106231689453MB)</div>
<div> free = 46072 (0.04393768310546875MB)</div><div> 99.8783731955558% \
used</div><div><br></div><div>11441 interned Strings occupying 1025920 \
bytes.</div></div><div><br></div><div><b>DirectMemory prints(refer <a \
href="https://gist.github.com/rednaxelafx/1593521" \
target="_blank">https://gist.github.com/rednaxelafx/1593521</a>) :</b></div>
<div><br></div><div>java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize \
16580</div><div>Attaching to process ID 16580, please wait...</div><div>Debugger \
attached successfully.</div><div>Server compiler detected.</div><div>
JVM version is 23.21-b01</div><div>NIO direct memory: (in bytes)</div><div> reserved \
size = <a href="tel:5864.191787" value="+15864191787" target="_blank">5864.191787</a> \
MB <a href="tel:%286149050767" value="+16149050767" target="_blank">(6149050767</a> \
bytes)</div>
<div> max size = 7850.687500 MB (8232042496 bytes)</div></div><span><font \
color="#888888"><div><br></div><br clear="all"> <div><br></div>-- <br>have a good \
day! <div>chenshang'an</div><div><br></div> </font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font \
color="#888888">-- <br><div dir="ltr">Apache MRUnit - Unit testing MapReduce - <a \
href="http://mrunit.apache.org" target="_blank">http://mrunit.apache.org</a></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>have a good day! \
<div>chenshang'an</div><div><br></div> </div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic