[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Client JVM hangs with Flume1.6 log4j appender
From: Hedi BOUFAIED <hedi.boufaied () amadeus ! com>
Date: 2016-01-08 16:48:54
Message-ID: a119895705854d3d98b20bf991b1c06b () MUCEXMBXCASP08 ! iis ! amadeus ! net
[Download RAW message or body]
Hello everyone,
I am facing an issue with Flume 1.6 using the log4j flume appender and I am looking \
for some help.
The picture is:
[CLIENT JVM : LOG4J -> FLUME-LOG4J-APPENDER ]---> [FLUME JVM : AVRO-SRC -> MEM \
CHANNEL -> LOGGER SINK]
After all the logs are sent by the client's log4j appender - and received and \
processed correctly by the flume agent (using avro source) - the client JVM (which is \
a simple test) hangs forever and never exits. When the client is recompiled and \
executed with the jars from Flume 1.5, it executes and terminates correctly...
An extract of the client execution with jdb seems to indicate that, with Flume 1.6 \
jars, after the main thread terminates, some threads related to Avro / \
NettyTransceiver remain alive hung in some deadlock. Has anyone hit this kind of \
issue ? I am copying below the jdb session output, my client's java code and \
log4j.properties and flume.conf files.
Thanks in advance for any help you might provide !
Hedi
JDB SESSION:
---
$ jdb Main
Initializing jdb ...
> run
run Main
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
>
VM Started: Using default maxIOWorkers
Start of main
This is a 1st log message
This is a 2nd log message
This is the 3rd and last message
End of main
suspend
All threads suspended.
> threads
Group system:
(java.lang.ref.Reference$ReferenceHandler)0x128 Reference Handler \
cond. waiting (java.lang.ref.Finalizer$FinalizerThread)0x127 Finalizer \
cond. waiting (java.lang.Thread)0x126 Signal Dispatcher \
running Group main:
(java.lang.Thread)0x293 New I/O worker #1 \
running (java.lang.Thread)0x29b New I/O worker #2 \
running (java.lang.Thread)0x29c New I/O worker #3 \
running (java.lang.Thread)0x29d New I/O worker #4 \
running (java.lang.Thread)0x29e New I/O worker #5 \
running (java.lang.Thread)0x29f New I/O worker #6 \
running (java.lang.Thread)0x2a0 New I/O worker #7 \
running (java.lang.Thread)0x2a1 New I/O worker #8 \
running (java.lang.Thread)0x31f Avro NettyTransceiver Boss 1 \
cond. waiting (java.lang.Thread)0x320 Hashed wheel timer #1 \
sleeping (java.lang.Thread)0x4bd Flume Avro RPC Client Call \
Invoker 1 cond. waiting (java.lang.Thread)0x535 Flume Avro \
RPC Client Call Invoker 2 cond. waiting (java.lang.Thread)0x536 \
DestroyJavaVM running
---
Client source code and log4j.properties:
---
import org.apache.log4j.Logger;
class Main {
final static Logger logger = Logger.getLogger(Main.class);
public static void main(String[] args) {
System.out.println("Start of main");
logger.warn("This is a 1st log message");
logger.warn("This is a 2nd log message");
logger.warn("This is the 3rd and last log message");
System.out.println("End of main");
}
}
---
log4j.rootLogger=INFO, stdout, flume
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%m%n
log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Port=5554
log4j.appender.flume.Hostname=localhost
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%m/%n
---
Flume.conf file:
---
agent.sources = avroSource
agent.channels = logChannel
agent.sinks = loggerSink
agent.sources.avroSource.type = avro
agent.sources.avroSource.bind = 0.0.0.0
agent.sources.avroSource.port = 5554
agent.sources.avroSource.channels = logChannel
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = logChannel
agent.channels.logChannel.type = memory
agent.channels.logChannel.capacity = 100
---
END
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello everyone, <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am facing an issue with Flume 1.6 using the \
log4j flume appender and I am looking for some help.<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">The picture is: <o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">[CLIENT JVM : LOG4J -> \
FLUME-LOG4J-APPENDER ]---> [FLUME JVM : AVRO-SRC -> MEM CHANNEL -> \
LOGGER SINK]<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">After all the logs are sent by the client’s log4j appender - and \
received and processed correctly by the flume agent (using avro source) - the client \
JVM (which is a simple test) hangs forever and never exits.<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">When the client is recompiled and executed with \
the jars from Flume 1.5, it executes and terminates \
correctly…<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span lang="EN-US">An \
extract of the client execution with jdb seems to indicate that, with Flume 1.6 jars, \
after the main thread terminates, some threads related to Avro / NettyTransceiver \
remain alive hung in some deadlock.<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">Has anyone hit this kind of issue \
?<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">I am copying below \
the jdb session output, my client’s java code and log4j.properties and \
flume.conf files.<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">Thanks in advance for any help you might provide !<o:p></o:p></span></p> \
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">Hedi<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">JDB SESSION:<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">---<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">$ jdb Main <o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">Initializing jdb ...<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">> run<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">run Main<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">Set uncaught \
java.lang.Throwable<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">Set \
deferred uncaught java.lang.Throwable<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">> <o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">VM Started: Using default \
maxIOWorkers<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">Start of \
main<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">This is a 1st log \
message<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">This is a 2nd \
log message<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">This is the \
3rd and last message<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">End of main<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">suspend<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">All threads suspended.<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">> threads<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">Group system:<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"> (java.lang.ref.Reference$ReferenceHandler)0x128 Reference \
Handler \
cond. waiting<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.ref.Finalizer$FinalizerThread)0x127 \
Finalizer   \
; \
cond. waiting<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x126 &nb \
sp; \
Signal Dispatcher \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US">Group \
main:<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x293 \
New I/O worker \
#1 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x29b &nb \
sp; New \
I/O worker #2 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x29c &nb \
sp; New \
I/O worker #3 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x29d &nb \
sp; New \
I/O worker #4 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x29e &nb \
sp; New \
I/O worker #5 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x29f &nb \
sp; New \
I/O worker #6 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"> (java.lang.Thread)0x2a0 & \
nbsp; \
New I/O worker \
#7 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x2a1 &nb \
sp; New \
I/O worker #8 \
running<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x31f &nb \
sp; \
Avro NettyTransceiver Boss 1 cond. \
waiting<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x320 &nb \
sp; \
Hashed wheel timer #1 \
sleeping<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x4bd &nb \
sp; \
Flume Avro RPC Client Call Invoker 1 cond. waiting<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x535 &nb \
sp; \
Flume Avro RPC Client Call Invoker 2 cond. waiting<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> \
(java.lang.Thread)0x536 &nb \
sp; \
DestroyJavaVM \
running<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">---<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">Client source code and log4j.properties:<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">---<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">import \
org.apache.log4j.Logger;<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">class Main {<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"> final static Logger logger = \
Logger.getLogger(Main.class);<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"> public static void main(String[] args) \
{<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"> \
System.out.println("Start of main");<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> \
logger.warn("This is a 1st log message");<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> \
logger.warn("This is a 2nd log message");<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> \
logger.warn("This is the 3rd and last log message");<o:p></o:p></span></p> \
<p class="MsoNormal"><span lang="EN-US"> \
System.out.println("End of main");<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US"> }<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">---<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">log4j.rootLogger=INFO, stdout, \
flume<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">log4j.appender.stdout=org.apache.log4j.ConsoleAppender<o:p></o:p></span></p>
<p class="MsoNormal"><span \
lang="EN-US">log4j.appender.stdout.Target=System.out<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
lang="EN-US">log4j.appender.stdout.layout=org.apache.log4j.PatternLayout<o:p></o:p></span></p>
<p class="MsoNormal"><span \
lang="EN-US">log4j.appender.stdout.layout.ConversionPattern=%m%n<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
lang="EN-US">log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender<o:p></o:p></span></p>
<p class="MsoNormal"><span \
lang="EN-US">log4j.appender.flume.Port=5554<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
lang="EN-US">log4j.appender.flume.Hostname=localhost<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
lang="EN-US">log4j.appender.flume.layout=org.apache.log4j.PatternLayout<o:p></o:p></span></p>
<p class="MsoNormal"><span \
lang="EN-US">log4j.appender.flume.layout.ConversionPattern=%m/%n<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">---<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Flume.conf file:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">---<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR">agent.sources = avroSource<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR">agent.channels = logChannel \
<o:p></o:p></span></p> <p class="MsoNormal"><span lang="FR">agent.sinks = loggerSink \
<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="FR"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="FR">agent.sources.avroSource.type = avro<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="FR">agent.sources.avroSource.bind = \
0.0.0.0<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="FR">agent.sources.avroSource.port = 5554<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="FR">agent.sources.avroSource.channels = logChannel \
<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="FR"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="FR">agent.sinks.loggerSink.type = logger<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="FR">agent.sinks.loggerSink.channel = \
logChannel<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="FR"><o:p> </o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">agent.channels.logChannel.type = memory<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-US">agent.channels.logChannel.capacity = \
100<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">---<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-US">END<o:p></o:p></span></p> </div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic