[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-user
Subject:    Re: Estimating the event loss
From:       Hari Shreedharan <hshreedharan () cloudera ! com>
Date:       2015-07-17 18:20:29
Message-ID: CAHbPYVZu9qtLqkzyDx70gj-C84K5_=YJLG7KE00bYw4j9w4_BQ () mail ! gmail ! com
[Download RAW message or body]

Note that if you simply reconfigure (rather than kill the JVM and restart)
you would not lose data in the memory channel


Thanks,
Hari

On Fri, Jul 17, 2015 at 7:08 AM, Johny Rufus <jrufus@cloudera.com> wrote:

> When you restart the agent, all the events in the memory channel will be
> lost (if the memory channel was full and contained 100k events, they will
> be lost)
> File channel will not result in loss of events, on restart the channel's
> state will be restored (File channel always maintains the last two data
> files the events were written to, even if the events in them were
> completely drained, this may give the impression that it still has data
> left)
>
> Thanks,
> Rufus
>
> On Thu, Jul 16, 2015 at 11:10 PM, Buntu Dev <buntudev@gmail.com> wrote:
>
>> I'm using a Memory channel with capacity set to 100000. Does this mean
>> when the flume agent restarts, its possible that I loose about 100k events?
>>
>> For any other durable channel say File channel I noticed .tmp files
>> created and written to but when I restart the agent these .tmp files are
>> left as-is. Since some events seem to be written to the .tmp files, should
>> I expect loss of events in this scenario?
>>
>>
>> Thanks!
>>
>
>

[Attachment #3 (text/html)]

<div dir="ltr">Note that if you simply reconfigure (rather than kill the JVM and \
restart) you would not lose data in the memory channel</div><div \
class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div \
dir="ltr"><br><div>Thanks,</div><div>Hari</div></div></div></div> <br><div \
class="gmail_quote">On Fri, Jul 17, 2015 at 7:08 AM, Johny Rufus <span \
dir="ltr">&lt;<a href="mailto:jrufus@cloudera.com" \
target="_blank">jrufus@cloudera.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">When you restart the agent, all the events in \
the memory channel will be lost (if the memory channel was full and contained 100k \
events, they will be lost)<div>File channel will not result in loss of events, on \
restart the channel&#39;s state will be restored (File channel always maintains the \
last two data files the events were written to, even if the events in them were \
completely drained, this may give the impression that it still has data \
left)</div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div \
style="font-family:arial;font-size:small">Thanks,</div><div \
style="font-family:arial;font-size:small">Rufus</div></div></div></div><div><div \
class="h5"> <br><div class="gmail_quote">On Thu, Jul 16, 2015 at 11:10 PM, Buntu Dev \
<span dir="ltr">&lt;<a href="mailto:buntudev@gmail.com" \
target="_blank">buntudev@gmail.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">I&#39;m using a Memory channel with capacity \
set to 100000. Does this mean when the flume agent restarts, its possible that I \
loose about 100k events?  <div><br></div><div>For any other durable channel say File \
channel I noticed .tmp files created and written to but when I restart the agent \
these .tmp files are left as-is. Since some events seem to be written to the .tmp \
files, should I expect loss of events in this \
scenario?</div><div><br></div><div><br></div><div>Thanks!</div></div> \
</blockquote></div><br></div></div></div></div> </blockquote></div><br></div>



[prev in list] [next in list] [prev in thread] [next in thread]