[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: Flume custom source get BLOCKED automatically after some time
From: Venkatesh B <venkateshkavayah () gmail ! com>
Date: 2014-07-16 10:37:41
Message-ID: CAGKBPaHid5frT2hyikHKM+5OoJc1tZa-U=PXZ_rEROzpC6JaTA () mail ! gmail ! com
[Download RAW message or body]
Hi Natty,
Thank you so much for letting me to know about jstack command.
Now the problem got solved.
In flume custom source thread execution is stopping at line
"System.out.println() or e.printStackTrace()" the statements which prints
on console.
After knowing this thing I removed all the console print statements it
worked then.
I don't know what really happening behind print statements.
Here is the dump of the thread where it stops execution.
============================================================================Thread \
11911: (state = IN_NATIVE)
- java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0
(Interpreted frame)
- java.io.FileOutputStream.write(byte[], int, int) @bci=4, line(2
(Interpreted frame)
- java.io.BufferedOutputStream.flushBuffer() @bci , linee (Interpreted
frame)
- java.io.BufferedOutputStream.flush() @bci=1, line3 (Interpreted frame)
- java.io.PrintStream.write(byte[], int, int) @bci0, lineC2 (Compiled
frame)
- sun.nio.cs.StreamEncoder.writeBytes() @bci0, line 2 (Compiled frame)
- sun.nio.cs.StreamEncoder.implFlushBuffer() @bci, line'2 (Compiled
frame)
- sun.nio.cs.StreamEncoder.flushBuffer() @bci, line… (Interpreted
frame)
- java.io.OutputStreamWriter.flushBuffer() @bci=4, line8 (Interpreted
frame)
- java.io.PrintStream.write(java.lang.String) @bci', lineG7
(Interpreted frame)
- java.io.PrintStream.print(java.lang.String) @bci=9, linea9
(Interpreted frame)
- java.io.PrintStream.println(java.lang.String) @bci=6, lineu6
(Interpreted frame)
-
org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2.writeComments(java.lang.String)
@bci11, lineu0 (Interpreted frame)
- org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2$3.run()
@bci‡1, line"7 (Interpreted frame)
- java.lang.Thread.run() @bci, linef2 (Interpreted frame)
==================================================================
Upto some time there is no problem with print statements but after sometime
getting error at these print statements.
Once again thanks a lot Natty.
Thanks,
Venkatesh.
On Tue, Jul 15, 2014 at 10:57 PM, Jonathan Natkins <natty@streamsets.com>
wrote:
> Hi Venkatesh,
>
> Does it reliably stop processing events after about 7 minutes, or does it
> happen randomly, and just quickly? Does the program immediately start up
> the Flume agent?
>
> Have you looked at a thread dump from the program, at all? You can use
> `jstack -F <pid>` to produce a stacktrace of all the threads in the
> program, which might have some useful information. Can you post a thread
> dump of the agent process that is spawned from your program?
>
> Thanks,
> Natty
>
>
> On Tue, Jul 15, 2014 at 8:44 AM, Venkatesh B <venkateshkavayah@gmail.com>
> wrote:
>
> > Hello,
> >
> > I have written flume custom source and in custom source in start() method
> > I'm initiating a thread. This thread "data_thread" will collect data from
> > web and writes to channel processor.
> >
> > My data collection thread is some thing like thread starts collecting
> > data from web and when data is over then sleeps for some time and then
> > again fetches data from web, this process is continuous. Here the control
> > will be inside the thread "data_thread" all the time, since I have used
> > while(true) inside thread logic.
> >
> > Here thread "data_thread" is collecting data upto 7mins and then it stops
> > collecting data.
> > I implemented logging and there by I came to know that thread
> > "data_thread" state is "BLOCKED". And not returning for ever.
> > In my custom source I tried in both ways by synchronizing all methods and
> > also by non synchronizing all methods but no use.
> >
> > This blocking occurs when I start flume agent from java program, but when
> > I start flume agent from shell it is working fine data is collected
> > continuously for days without any error.
> >
> > Any idea on why thread is going to BLOCKED state??
> >
> > And also I want to know about flume custom source environment
> >
> > Thanking you
> > Venkatesh
> >
> >
> >
>
[Attachment #3 (text/html)]
<div dir="ltr">Hi Natty,<div><br></div><div>Thank you so much for letting me to know \
about jstack command.</div><div>Now the problem got solved.</div><div>In flume custom \
source thread execution is stopping at line "System.out.println() or \
e.printStackTrace()" the statements which prints on console.</div> <div>After \
knowing this thing I removed all the console print statements it worked \
then.</div><div> I don't know what really happening behind print \
statements.</div><div>Here is the dump of the thread where it stops execution.</div> \
<div><br></div><div>=============================================================================</div><div><div>Thread \
11911: (state = IN_NATIVE)</div><div> - java.io.FileOutputStream.writeBytes(byte[], \
int, int) @bci=0 (Interpreted frame)</div> <div> - \
java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=282 (Interpreted \
frame)</div><div> - java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 \
(Interpreted frame)</div><div> - java.io.BufferedOutputStream.flush() @bci=1, \
line=123 (Interpreted frame)</div> <div> - java.io.PrintStream.write(byte[], int, \
int) @bci=30, line=432 (Compiled frame)</div><div> - \
sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Compiled frame)</div><div> \
- sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Compiled frame)</div> \
<div> - sun.nio.cs.StreamEncoder.flushBuffer() @bci=15, line=85 (Interpreted \
frame)</div><div> - java.io.OutputStreamWriter.flushBuffer() @bci=4, line=168 \
(Interpreted frame)</div><div> - java.io.PrintStream.write(java.lang.String) \
@bci=27, line=477 (Interpreted frame)</div> <div> - \
java.io.PrintStream.print(java.lang.String) @bci=9, line=619 (Interpreted \
frame)</div><div> - java.io.PrintStream.println(java.lang.String) @bci=6, line=756 \
(Interpreted frame)</div><div> - \
org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2.writeComments(java.lang.String) \
@bci=1311, line=750 (Interpreted frame)</div> <div> - \
org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2$3.run() @bci=871, \
line=227 (Interpreted frame)</div><div> - java.lang.Thread.run() @bci=11, line=662 \
(Interpreted frame)</div></div><div><br></div><div>===================================================================</div>
<div><br></div><div><br></div><div>Upto some time there is no problem with print \
statements but after sometime getting error at these print \
statements.</div><div><br></div><div>Once again thanks a lot Natty.</div><div> <br> \
</div><div><br></div><div>Thanks,</div><div>Venkatesh.</div></div><div \
class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 15, 2014 at 10:57 \
PM, Jonathan Natkins <span dir="ltr"><<a href="mailto:natty@streamsets.com" \
target="_blank">natty@streamsets.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Venkatesh,<div><br></div><div>Does it \
reliably stop processing events after about 7 minutes, or does it happen randomly, \
and just quickly? Does the program immediately start up the Flume agent?</div> <div>
<br></div><div>Have you looked at a thread dump from the program, at all? You can use \
`jstack -F <pid>` to produce a stacktrace of all the threads in the program, \
which might have some useful information. Can you post a thread dump of the agent \
process that is spawned from your program?</div>
<div><br></div><div>Thanks,</div><div>Natty </div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div \
class="gmail_quote">On Tue, Jul 15, 2014 at 8:44 AM, Venkatesh B <span \
dir="ltr"><<a href="mailto:venkateshkavayah@gmail.com" \
target="_blank">venkateshkavayah@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I have written flume \
custom source and in custom source in start() method I'm initiating a thread. \
This thread "data_thread" will collect data from web and writes to channel \
processor.</div>
<div><br></div><div>My data collection thread is some thing like thread starts \
collecting data from web and when data is over then sleeps for some time and then \
again fetches data from web, this process is continuous. Here the control will be \
inside the thread "data_thread" all the time, since I have used while(true) \
inside thread logic. </div>
<div><br></div><div>Here thread "data_thread" is collecting data upto \
7mins and then it stops collecting data.</div><div>I implemented logging and there by \
I came to know that thread "data_thread" state is "BLOCKED". \
And not returning for ever.</div>
<div>In my custom source I tried in both ways by synchronizing all methods and also \
by non synchronizing all methods but no use.</div><div><br></div><div>This blocking \
occurs when I start flume agent from java program, but when I start flume agent from \
shell it is working fine data is collected continuously for days without any \
error.</div>
<div><br></div><div>Any idea on why thread is going to BLOCKED \
state??</div><div><br></div><div>And also I want to know about flume custom source \
environment</div><div><br></div><div>Thanking you</div><span><font \
color="#888888"><div>
Venkatesh</div>
<div><br></div><div><br></div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic