[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-user
Subject:    Re: Sinks are likely not keeping up with sources, or the buffer size is too tight error
From:       Johny Rufus <jrufus () cloudera ! com>
Date:       2015-07-11 23:28:19
Message-ID: CALmgQN7jF7YZGEjiNGGvq_vJPRdQO_yHRnpOsAXP+joU1UYbSw () mail ! gmail ! com
[Download RAW message or body]

You are running out of space in your memory channel queue.
Can  you try reducing your transaction capacity to an experimental value
like 1000, and batchSize to 100 and see how that works for you and take it
from there.

Also make sure that the size of your memory channel can always account for
the difference in the rate at which your source produces data and your sink
consumes data.

Thanks,
Rufus

On Sat, Jul 11, 2015 at 12:39 AM, Buntu Dev <buntudev@gmail.com> wrote:

> I'm using Memory channel along with Kite dataset sink and keep running
> into this error:
>
> ~~~~
>
> ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: tracksChannel}
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)
> at
> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.ChannelFullException: Space for commit to
> queue couldn't be acquired. Sinks are likely not keeping up with sources,
> or the buffer size is too tight
> at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:130)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
>
> ~~~~
>
> I tried different combinations of channel capacity and transactionCapacity
> for the channel and various sink's batchSize < channel's
> transactionCapacity.
>
> Here is one of those combinations of configs:
>
> ~~~~~~
> myAgent.sinks.mySink.kite.batchSize = 10000
> myAgent.sinks.myHdfsSink.kite.rollInterval = 300
>
> myAgent.channels.myChannel.type = memory
> myAgent.channels.myChannel.capacity = 200000
> myAgent.channels.myChannel.transactionCapacity = 20000
> ~~~~~~
>
> Its currently a single flume agent and I look for recommendations on
> different tiered architectures involving Flume, Kafka and ultimately to the
> Kite dataset sink to help overcome this issue.
>
>
> Thanks!
>

[Attachment #3 (text/html)]

<div dir="ltr">You are running out of space in your memory channel queue.<div>Can   \
you try reducing your transaction capacity to an experimental value like 1000, and \
batchSize to 100 and see how that works for you and take it from \
there.</div><div><br></div><div>Also make sure that the size of your memory channel \
can always account for the difference in the rate at which your source produces data \
and your sink consumes data.</div><div class="gmail_extra"><br clear="all"><div><div \
class="gmail_signature"><div dir="ltr"><div \
style="font-family:arial;font-size:small">Thanks,</div><div \
style="font-family:arial;font-size:small">Rufus</div></div></div></div> <br><div \
class="gmail_quote">On Sat, Jul 11, 2015 at 12:39 AM, Buntu Dev <span \
dir="ltr">&lt;<a href="mailto:buntudev@gmail.com" \
target="_blank">buntudev@gmail.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">I&#39;m using Memory channel along with Kite \
dataset sink and keep running into this error:<br><br>~~~~<br><br>ERROR \
kafka.KafkaSource: KafkaSource EXCEPTION, {}<br>org.apache.flume.ChannelException: \
Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: \
tracksChannel}<br>	at \
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)<br>	at \
org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)<br>	at \
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)<br>	at \
java.lang.Thread.run(Thread.java:745)<br>Caused by: \
org.apache.flume.ChannelFullException: Space for commit to queue couldn&#39;t be \
acquired. Sinks are likely not keeping up with sources, or the buffer size is too \
tight<br>	at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:130)<br>	at \
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)<br>	at \
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)<br><br>~~~~<br><br>I \
tried different combinations of channel capacity and transactionCapacity for the \
channel and various sink&#39;s batchSize &lt; channel&#39;s \
transactionCapacity.<div><br></div><div>Here is one of those combinations of \
configs:</div><div><br></div><div>~~~~~~</div><div><div>myAgent.sinks.mySink.kite.batchSize \
= 10000</div><div>myAgent.sinks.myHdfsSink.kite.rollInterval = \
300</div></div><div><br></div><div><div>myAgent.channels.myChannel.type = \
memory<br></div><div>myAgent.channels.myChannel.capacity = \
200000</div><div>myAgent.channels.myChannel.transactionCapacity = \
20000</div></div><div>~~~~~~</div><div><br></div><div>Its currently a single flume \
agent and I look for recommendations on different tiered architectures involving \
Flume, Kafka and ultimately to the Kite dataset sink to help overcome this \
issue.</div><div><br></div><div><br></div><div>Thanks!</div></div> \
</blockquote></div><br></div></div>



[prev in list] [next in list] [prev in thread] [next in thread]