[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: Netcat source stops processing data
From: Mike Percy <mpercy () apache ! org>
Date: 2012-11-20 23:48:24
Message-ID: CAJLbxRan29W+Jwe9b0rDeRR8Y6mapnMQFv7aqSmemsCsSLW=7w () mail ! gmail ! com
[Download RAW message or body]
Rahul,
A patch and a unit test to add this as an option would be greatly
appreciated!
There is already a JIRA open for this:
https://issues.apache.org/jira/browse/FLUME-1713
Regards,
Mike
On Tue, Nov 20, 2012 at 3:20 PM, Rahul Ravindran <rahulrv@yahoo.com> wrote:
> Pinging on this slightly old thread.
>
> I want to avoid the overhead of HTTP. Is it possible to get the NetCat
> source be fixed to not send the OK? I did a local build of flume without
> the "OK" being written, and it works fine.
> If there are tests which depend on the OK being sent back, could an
> additional parameter which skips it be added?
>
>
>
> ------------------------------
> *From:* Rahul Ravindran <rahulrv@yahoo.com>
> *To:* "user@flume.apache.org" <user@flume.apache.org>
> *Cc:* "user@flume.apache.org" <user@flume.apache.org>
> *Sent:* Thursday, November 8, 2012 6:15 PM
> *Subject:* Re: Netcat source stops processing data
>
> Thanks! Will try removing the ok.
>
> Sent from my phone.Excuse the terseness.
>
> On Nov 8, 2012, at 3:15 PM, Hari Shreedharan <hshreedharan@cloudera.com>
> wrote:
>
> Rahul,
>
> Are you reading the responses sent by the net cat source? If you don't
> read the OK sent by net cat source on your application side, your
> application's buffer gets full causing the net cat source to queue up stuff
> and eventually die. It is something we need to fix, but I don't know if
> anyone is using net cat in production - you should probably test using Avro
> source or the new HTTP source(for this you would need to build trunk/1.3
> branch or wait for 1.3 release).
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran wrote:
>
> Hello,
> I wanted to perform a load test to get an idea of how we would look to
> scale flume for our deployment. I have pasted the config file at the source
> below. I have a netcat source which is listening on a port and have 2
> channels, 2 avro sinks consuming the events from the netcat source.
>
> My load generator is a simple C program which is continually sending 20
> characters in each message using a socket, and send(). I notice that ,
> initially, a lot of traffic makes it through and then the flume agent
> appears to stop consuming data(after about 80k messages). This results in
> the tcp receive and send buffer being full. I understand that the rate at
> which I am generating traffic may overwhelm flume, but I would expect it to
> gradually consume data. It does not consume any more messages. I looked
> through the flume logs and did not see anything there (no stack trace). I
> ran tcpdump and see that the receive window initially is non-zero but
> begins to decrease and then goes down to zero, and very slowly opens up to
> a size of 1 (once in 10 seconds)
>
> Could you help on what may be going on or if there is something wrong with
> my config?
>
> agent1.channels.ch1.type = MEMORY
> agent1.channels.ch1.capacity = 50000
> agent1.channels.ch1.transactionCapacity = 5000
>
> agent1.sources.netcat.channels = ch1
> agent1.sources.netcat.type= netcat
> agent1.sources.netcat.bind = 127.0.0.1
> agent1.sources.netcat.port = 44444
>
> agent1.sinks.avroSink1.type = avro
> agent1.sinks.avroSink1.channel = ch1
> agent1.sinks.avroSink1.hostname = <remote hostname>
> agent1.sinks.avroSink1.port = 4545
> agent1.sinks.avroSink1.connect-timeout = 300000
>
>
> agent1.sinks.avroSink2.type = avro
> agent1.sinks.avroSink2.channel = ch1
> agent1.sinks.avroSink2.hostname = <remote hostname>
> agent1.sinks.avroSink2.port = 4546
> agent1.sinks.avroSink2.connect-timeout = 300000
>
> agent1.channels = ch1
> agent1.sources = netcat
> agent1.sinks = avroSink1 avroSink2 avroSink2
>
>
>
>
>
[Attachment #3 (text/html)]
Rahul,<div>A patch and a unit test to add this as an option would be greatly \
appreciated!</div><div><br></div><div>There is already a JIRA open for this: <a \
href="https://issues.apache.org/jira/browse/FLUME-1713">https://issues.apache.org/jira/browse/FLUME-1713</a></div>
<div><br></div><div>Regards,</div><div>Mike</div><div><br><br><div \
class="gmail_quote">On Tue, Nov 20, 2012 at 3:20 PM, Rahul Ravindran <span \
dir="ltr"><<a href="mailto:rahulrv@yahoo.com" \
target="_blank">rahulrv@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new \
roman,new york,times,serif"><div>Pinging on this slightly old thread.</div>
<div><br></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">I want to avoid the overhead of HTTP. \
Is it possible to get the NetCat source be fixed to not send the OK? I did a local \
build of flume without the "OK" being written, and it works fine.</div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">If there are tests which depend on the \
OK being sent back, could an additional parameter which skips it be added?</div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif"><br></div><div class="hm HOEnZb"><div \
style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">
<br></div><div><br></div> </div><div style="font-family:'times new \
roman','new york',times,serif;font-size:12pt"><div class="hm HOEnZb"> \
</div><div style="font-family:'times new roman','new \
york',times,serif;font-size:12pt">
<div class="hm HOEnZb"> <div dir="ltr"> <font face="Arial"> <hr size="1"> <b><span \
style="font-weight:bold">From:</span></b> Rahul Ravindran <<a \
href="mailto:rahulrv@yahoo.com" target="_blank">rahulrv@yahoo.com</a>><br>
<b><span style="font-weight:bold">To:</span></b> "<a \
href="mailto:user@flume.apache.org" target="_blank">user@flume.apache.org</a>" \
<<a href="mailto:user@flume.apache.org" \
target="_blank">user@flume.apache.org</a>> <br>
<b><span style="font-weight:bold">Cc:</span></b> "<a \
href="mailto:user@flume.apache.org" target="_blank">user@flume.apache.org</a>" \
<<a href="mailto:user@flume.apache.org" \
target="_blank">user@flume.apache.org</a>> <br>
<b><span style="font-weight:bold">Sent:</span></b> Thursday, November 8, 2012 6:15 \
PM<br> <b><span style="font-weight:bold">Subject:</span></b> Re: Netcat source stops \
processing data<br> </font> </div></div><div><div class="h5"> <br>
<div><div><div>Thanks! Will try removing the ok. <br><br>Sent from my phone.Excuse \
the terseness.</div><div><br>On Nov 8, 2012, at 3:15 PM, Hari Shreedharan <<a \
rel="nofollow" href="mailto:hshreedharan@cloudera.com" \
target="_blank">hshreedharan@cloudera.com</a>> wrote:<br>
<br></div><blockquote type="cite"><div>
<div>
<span style="font-size:12px">Rahul,</span>
</div><div><span style="font-size:12px"><br></span></div><div><span \
style="font-size:12px">Are you reading the responses sent by the net cat source? If \
you don't read the OK sent by net cat source on your application side, your \
application's buffer gets full causing the net cat source to queue up stuff and \
eventually die. It is something we need to fix, but I don't know if anyone is \
using net cat in production - you should probably test using Avro source or the new \
HTTP source(for this you would need to build trunk/1.3 branch or wait for 1.3 \
release).</span></div>
<div><span style="font-size:12px"><br></span></div><div><span \
style="font-size:12px"><br></span></div><div><span \
style="font-size:12px">Thanks</span></div><div><span \
style="font-size:12px">Hari</span></div>
<div><div><br></div><div>-- </div><div>Hari \
Shreedharan</div><div><br></div></div>
<div style="color:#a0a0a8">On Thursday, November 8, 2012 at 3:05 PM, \
Rahul Ravindran wrote:</div>
<blockquote type="cite" \
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px"> \
<span><div><div><div style="font-size:12pt;font-family:'times new \
roman','new york',times,serif"><div style="font-family:'times new \
roman','new york',times,serif;font-size:12pt">
Hello,</div><div style="font-family:'times new roman','new \
york',times,serif;font-size:12pt"> I wanted to perform a load test to get an \
idea of how we would look to scale flume for our deployment. I have pasted the config \
file at the source below. I have a netcat source which is listening on a port and \
have 2 channels, 2 avro sinks consuming the events from the netcat source.</div>
<div style="font-family:'times new roman','new \
york',times,serif;font-size:12pt"><br></div><div \
style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">
My load generator is a simple C program
which is continually sending 20 characters in each message
using a socket, and send(). I notice that , initially, a lot of traffic makes it \
through and then the flume agent appears to stop consuming data(after about 80k \
messages). This results in the tcp receive and send buffer being full. I understand \
that the rate at which I am generating traffic may overwhelm flume, but I would \
expect it to gradually consume data. It does not consume any more messages. I looked \
through the flume logs and did not see anything there (no stack trace). I ran tcpdump \
and see that the receive window initially is non-zero but begins to decrease and then \
goes down to zero, and very slowly opens up to a size of 1 (once in 10 seconds)</div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif"><br></div><div \
style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">
Could you help on what may be going on or if there is something wrong with my \
config?</div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times \
new roman','new york',times,serif">
<br></div><div style="background-color:transparent"><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.channels.ch1.type = MEMORY</font></div><div \
style="background-color:transparent">
<font face="times new roman, new york, times, serif">agent1.channels.ch1.capacity = \
50000</font></div><div style="background-color:transparent"><font face="times new \
roman, new york, times, serif">agent1.channels.ch1.transactionCapacity = \
5000</font></div>
<div style="background-color:transparent"><br></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sources.netcat.channels = ch1</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sources.netcat.type= netcat</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sources.netcat.bind = 127.0.0.1</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sources.netcat.port = 44444</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif"><br>
</font></div><div style="background-color:transparent"><font face="times new roman, \
new york, times, serif">agent1.sinks.avroSink1.type = avro</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sinks.avroSink1.channel = ch1</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sinks.avroSink1.hostname = <remote \
hostname></font></div><div style="background-color:transparent"><font face="times \
new roman, new york, times, serif">agent1.sinks.avroSink1.port = 4545</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sinks.avroSink1.connect-timeout = 300000</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif"><br>
</font></div><div style="background-color:transparent"><font face="times new roman, \
new york, times, serif"><br></font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sinks.avroSink2.type = avro</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sinks.avroSink2.channel = ch1</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sinks.avroSink2.hostname = </font><span style="font-family:'times \
new roman','new \
york',times,serif;background-color:transparent"><remote \
hostname></span></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sinks.avroSink2.port = 4546</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sinks.avroSink2.connect-timeout = 300000</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif"><br></font></div><div style="background-color:transparent"><font \
face="times new roman, new york, times, serif">agent1.channels = ch1</font></div>
<div style="background-color:transparent"><font face="times new roman, new york, \
times, serif">agent1.sources = netcat</font></div><div \
style="background-color:transparent"><font face="times new roman, new york, times, \
serif">agent1.sinks = avroSink1 avroSink2 \
avroSink2</font></div></div></div></div></div></span>
</blockquote>
<div>
<br>
</div>
</div></blockquote></div></div><br><br> </div></div></div> </div> \
</div></div></blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic