[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: RE: recovery after memory transaction capacity is exceeded
From: Balthasar Schopman <b.schopman () tech ! leaseweb ! com>
Date: 2015-10-16 9:01:52
Message-ID: F5EFD3F659C0404BA8709452526F1871EA6AE4 () nlhlmexdb05 ! ocom ! lan
[Download RAW message or body]
[Attachment #2 (text/plain)]
I re-ran the test with the source & sink batchSizes set to 100, the memory channel \
transactionCapacity set to 100 and its capacity on 300. With those values, the proof \
of concept works exactly as expected, so it indeed that bug that caused my problem. \
Thank you!
To specify the statement "the proof of concept works exactly as expected": if Flume \
isn't able to write events to the sink, it will read files from the spoolDir until \
the memory capacity is maxed out (i.e. 300 events). Any files that are added to the \
spoolDir are _not_ processed, until Flume is able to write to the sink. As soon as \
Flume is able to write to the sink again, all the stalled events are processed (the \
events in memory and the unread files in the spoolDir).
--
Kind regards,
Balthasar Schopman
LeaseWeb CDN Innovation Engineer
Kind regards,
Balthasar Schopman
Software Developer
LeaseWeb Technologies B.V.
T: +31 20 316 0232
E: b.schopman@tech.leaseweb.com
W: www.leaseweb.com<http://www.leaseweb.com>
Luttenbergweg 8, 1101 EC Amsterdam, Netherlands
________________________________
From: Gonzalo Herreros [gherreros@gmail.com]
Sent: Tuesday, October 13, 2015 10:04
To: user@flume.apache.org
Subject: Re: recovery after memory transaction capacity is exceeded
I believe you are suffering from this bug: \
https://issues.apache.org/jira/browse/FLUME-2778 So when it's running is able to keep \
up but when the channel has more than 4 events queued, the Sink tried to extract 100 \
(default batch size) and you get that error.
Regards,
Gonzalo
On 13 October 2015 at 08:12, Balthasar Schopman \
<b.schopman@tech.leaseweb.com<mailto:b.schopman@tech.leaseweb.com>> wrote: Hi,
I'm creating a proof-of-concept of a Flume agent that'll buffer events and stops \
consuming events from the source when the sink is unavailable. Only when the sink is \
available again, the buffered events should be processed and then the source restarts \
consumption.
For this I've created a simple agent, which reads from a SpoolDir and writes to a \
file. To simulate that the sink service is down, I change file permissions so Flume \
can't write to it. Then I start Flume some events are buffered in the memory channel \
and it stops consuming events when the channel capacity is full, as expected. As soon \
as the file becomes writeable, the sink is able to process the events and Flume \
recovers. However, that only works when the transaction capacity is not exceeded. As \
soon as the transaction capacity is exceeded, Flume never recovers and keeps writing \
the following error:
2015-10-02 14:52:51,940 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR -
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to
deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to process transaction
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction,
capacity 4 full, consider committing more frequently, increasing capacity, or
increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:191)
... 3 more
As soon as the number of events buffered in memory exceed the transaction capacity \
(4) this error occurs. I don't understand why, because the batchSize of the fileout \
is 1, so it should take out the events one by one.
This is the config I'm using:
agent.sources = spool-src
agent.channels = mem-channel
agent.sinks = fileout
agent.sources.spool-src.channels = mem-channel
agent.sources.spool-src.type = spooldir
agent.sources.spool-src.spoolDir = /tmp/flume-spool
agent.sources.spool-src.batchSize = 1
agent.channels.mem-channel.type = memory
agent.channels.mem-channel.capacity = 10
agent.channels.mem-channel.transactionCapacity = 4
agent.sinks.fileout.channel = mem-channel
agent.sinks.fileout.type = file_roll
agent.sinks.fileout.sink.directory = /tmp/flume-output
agent.sinks.fileout.sink.rollInterval = 0
agent.sinks.fileout.batchSize = 1
I've tested this config with different values for the channel capacity & transaction \
capacity (e.g., 3 and 3), but haven't found a config where Flume is able to recover \
after the channel capacity is full. Any ideas on how to achieve this?
--
Kind regards,
Balthasar Schopman
LeaseWeb CDN Innovation Engineer
Kind regards,
Balthasar Schopman
Software Developer
LeaseWeb Technologies B.V.
T: +31 20 316 0232
M:
E: b.schopman@tech.leaseweb.com<mailto:b.schopman@tech.leaseweb.com>
W: http://www.leaseweb.com
Luttenbergweg 8, 1101 EC Amsterdam, Netherlands
[Attachment #3 (text/html)]
<html dir="ltr">
<head>
<!-- Template generated by Exclaimer Signature Manager Exchange Edition on 11:01:53 \
vrijdag, 16 oktober 2015 --> <meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"> <style type="text/css">P.ImprintUniqueID {
MARGIN: 0cm 0cm 0pt
}
LI.ImprintUniqueID {
MARGIN: 0cm 0cm 0pt
}
DIV.ImprintUniqueID {
MARGIN: 0cm 0cm 0pt
}
TABLE.ImprintUniqueIDTable {
MARGIN: 0cm 0cm 0pt
}
DIV.Section1 {
page: Section1
}
</style><style id="owaParaStyle" type="text/css">P \
{margin-top:0;margin-bottom:0;}</style> </head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">I \
re-ran the test with the source & sink batchSizes set to 100, the memory channel \
transactionCapacity set to 100 and its capacity on 300. With those values, the proof \
of concept works exactly as expected, so it indeed that bug that caused my problem. \
Thank you!<br> <div><br>
To specify the statement "the proof of concept works exactly as expected": \
if Flume isn't able to write events to the sink, it will read files from the spoolDir \
until the memory capacity is maxed out (i.e. 300 events). Any files that are added to \
the spoolDir are _not_ processed, until Flume is able to write to the sink. As soon \
as Flume is able to write to the sink again, all the stalled events are processed \
(the events in memory and the unread files in the spoolDir).<br> <br>
<div style="font-family:Tahoma; font-size:13px">--<br>
Kind regards,<br>
Balthasar Schopman<br>
LeaseWeb CDN Innovation Engineer<br>
<br>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px"></div>
</div>
<p class="ImprintUniqueID"><font face="Verdana"><font size="2">Kind regards,<br>
<br>
</font></font>
<table class="ImprintUniqueIDTable" style="HEIGHT: 135px; BORDER-COLLAPSE: collapse; \
WIDTH: 706px" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td align="left"><font style="font-family:Verdana;font-size:10pt;">Balthasar \
Schopman</font><br> <font \
style="font-family:Verdana;font-size:10pt;font-weight:bold;">Software \
Developer</font><br> <font style="font-family:Verdana;font-size:10pt;">LeaseWeb \
Technologies B.V.</font><br> <br>
<font style="font-family:Verdana;font-size:10pt;"><font \
style="font-family:Verdana;font-size:10pt;">T: </font>+31 20 316 0232</font><br>
<font style="font-family:Verdana;font-size:10pt;"></font><br>
<font style="font-family:Verdana;font-size:10pt;"><font \
style="font-family:Verdana;font-size:10pt;">E: \
</font>b.schopman@tech.leaseweb.com</font><br> <font \
style="font-family:Verdana;font-size:10pt;"><font \
style="font-family:Verdana;font-size:10pt;">W: </font><span \
style="font-family:Verdana;font-size:10pt;"><a href="http://www.leaseweb.com" \
title="" target="" style="font-family:Verdana;font-size:10pt;"><span \
style="font-family:Verdana; \
font-size:10pt;">www.leaseweb.com</span></a></span></font></td> </tr>
</tbody>
</table>
<br>
<table class="ImprintUniqueIDTable" style="BORDER-COLLAPSE: collapse" cellspacing="0" \
cellpadding="0" border="0"> <tbody>
<tr>
<td align="left"><font size="2" face="Verdana"><font \
style="font-family:Verdana;font-size:10pt;">Luttenbergweg 8</font>, </font></td> \
<td align="left"><font size="2" face="Verdana"><font \
style="font-family:Verdana;font-size:10pt;">1101 EC</font> <font \
style="font-family:Verdana;font-size:10pt;">Amsterdam</font>, </font></td> <td \
align="left"><font size="2" face="Verdana"><font \
style="font-family:Verdana;font-size:10pt;">Netherlands</font></font></td> </tr>
</tbody>
</table>
</p>
<br>
<br>
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF189267"><font face="Tahoma" size="2" \
color="#000000"><b>From:</b> Gonzalo Herreros [gherreros@gmail.com]<br> <b>Sent:</b> \
Tuesday, October 13, 2015 10:04<br> <b>To:</b> user@flume.apache.org<br>
<b>Subject:</b> Re: recovery after memory transaction capacity is exceeded<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">I believe you are suffering from this bug: <a \
href="https://issues.apache.org/jira/browse/FLUME-2778" \
target="_blank">https://issues.apache.org/jira/browse/FLUME-2778</a> <div>So when \
it's running is able to keep up but when the channel has more than 4 events queued, \
the Sink tried to extract 100 (default batch size) and you get that error.</div> \
<div><br> </div>
<div>Regards,</div>
<div>Gonzalo</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 13 October 2015 at 08:12, Balthasar Schopman <span \
dir="ltr"> <<a href="mailto:b.schopman@tech.leaseweb.com" \
target="_blank">b.schopman@tech.leaseweb.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; \
padding-left:1ex"> Hi,<br>
<br>
I'm creating a proof-of-concept of a Flume agent that'll buffer events and stops \
consuming events from the source when the sink is unavailable. Only when the sink is \
available again, the buffered events should be processed and then the source restarts \
consumption.<br> <br>
For this I've created a simple agent, which reads from a SpoolDir and writes to a \
file. To simulate that the sink service is down, I change file permissions so Flume \
can't write to it. Then I start Flume some events are buffered in the memory channel \
and it stops consuming events when the channel capacity is full, as expected. As \
soon as the file becomes writeable, the sink is able to process the events and Flume \
recovers. However, that only works when the transaction capacity is not exceeded. As \
soon as the transaction capacity is exceeded, Flume never recovers and keeps writing \
the following error:<br> <br>
2015-10-02 14:52:51,940 (SinkRunner-PollingRunner-DefaultSinkProcessor) \
[ERROR -<br> \
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to<br> \
deliver event. Exception follows.<br> \
org.apache.flume.EventDeliveryException: Failed to process transaction<br> \
at \
org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)<br> \
at \
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)<br> \
at \
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)<br> \
at java.lang.Thread.run(Thread.java:745)<br> Caused by: \
org.apache.flume.ChannelException: Take list for MemoryTransaction,<br> \
capacity 4 full, consider committing more frequently, increasing capacity, or<br> \
increasing thread count<br> at \
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)<br>
at \
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)<br>
at \
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)<br>
at \
org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:191)<br> \
... 3 more<br> <br>
As soon as the number of events buffered in memory exceed the transaction capacity \
(4) this error occurs. I don't understand why, because the batchSize of the fileout \
is 1, so it should take out the events one by one.<br> <br>
This is the config I'm using:<br>
<br>
agent.sources = spool-src<br>
agent.channels = mem-channel<br>
agent.sinks = fileout<br>
<br>
agent.sources.spool-src.channels = mem-channel<br>
agent.sources.spool-src.type = spooldir<br>
agent.sources.spool-src.spoolDir = /tmp/flume-spool<br>
agent.sources.spool-src.batchSize = 1<br>
<br>
agent.channels.mem-channel.type = memory<br>
agent.channels.mem-channel.capacity = 10<br>
agent.channels.mem-channel.transactionCapacity = 4<br>
<br>
agent.sinks.fileout.channel = mem-channel<br>
agent.sinks.fileout.type = file_roll<br>
agent.sinks.fileout.sink.directory = /tmp/flume-output<br>
agent.sinks.fileout.sink.rollInterval = 0<br>
agent.sinks.fileout.batchSize = 1<br>
<br>
I've tested this config with different values for the channel capacity & \
transaction capacity (e.g., 3 and 3), but haven't found a config where Flume is able \
to recover after the channel capacity is full. Any ideas on how to achieve this?<br> \
<br>
--<br>
Kind regards,<br>
Balthasar Schopman<br>
LeaseWeb CDN Innovation Engineer<br>
<br>
Kind regards,<br>
<br>
Balthasar Schopman<br>
Software Developer<br>
LeaseWeb Technologies B.V.<br>
<br>
T: +31 20 316 0232<br>
M:<br>
E: <a href="mailto:b.schopman@tech.leaseweb.com" \
target="_blank">b.schopman@tech.leaseweb.com</a><br>
W: <a href="http://www.leaseweb.com" rel="noreferrer" \
target="_blank">http://www.leaseweb.com</a><br> <br>
Luttenbergweg 8, 1101 EC Amsterdam, Netherlands<br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic