[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Losing Events | SyslogTCP and Syslog Multi Port
From: "Romero, Miguel" <miguel.romero-remirez () hp ! com>
Date: 2015-03-05 9:42:49
Message-ID: 6AFB897E5F9D9944A4D94E030264681440E845D5 () G4W3213 ! americas ! hpqcorp ! net
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hi all,
I am testing with SyslogSourceTCP and MultiSyslogSourceTCP.
My problem is that the Source loses syslog-events but depends on the tools which sent \
us syslog.
• If I use a tool like QRadar the Source loses events (No exceptions, \
ni trace in log about the lost) • If I use a adhoc software (log4j \
appender o cat | nc command or …), there aren't lost events.
I have configured flume of a lot of ways, but I always have the same result. Have \
to be the syslog client of a special way?
Thanks.
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Measurements
From: Smaine Kahlouch [mailto:smaine.kahlouch@smartjog.com]
Sent: jueves, 05 de marzo de 2015 10:11
To: user@flume.apache.org
Subject: Re: Syslog TCP performances issue with filechannel
Actually the batchSize is configured on sink level.
I didn't find this option on file channel.
Furthermore, the source batchSize can't be configured because it is a syslog-ng tool \
which doesn't have this capability. I tried with "netcat" source and i face the same \
behaviour.
I guess you're right, for each event there's a fsync which causes the heavy load on \
diks. However i've read this topic : \
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Measurements
And they didn't have the same problem obviously.
Regards,
--
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com
On 03/04/15 20:08, Hari Shreedharan wrote:
You should probably increase the batch size, since each batch causes an fsync which \
slows things down.
Thanks,
Hari
On Wed, Mar 4, 2015 at 6:28 AM, Smaine Kahlouch \
<smaine.kahlouch@smartjog.com<mailto:smaine.kahlouch@smartjog.com>> wrote: Hi all,
I'm currently doing benchmarks on flume.
We're planning to use flume with syslogtcp as source and filechannel in order to have \
avoid data loss.
The performances are quiet good when a memorychannel is used :
~100 000events/sec (event size = 600bytes)
But as soon as i switch to filechannel the performances drop drammatically:
~300events/sec
Despite this poor result, the behaviour is really strange because i have a heavy disk \
usage (all the disks), near 100%.
I use a tool provided by syslog-ng in order to generate syslog logs : \
loggen<http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/loggen.1.html>
ex : loggen -i -I 3000000 --size 600 --active-connections 200 myflumehost 20515
Flume version : 1.5.2
Operating System : Centos 6
Please find my flume configuration enclosed. The filechannel is spread over 5 disks \
in order to improve performance.
Could you please help me to configure properly syslogtcp source with filechannel ?
Regards,
--
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com
<flume.conf>
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1616332618;
mso-list-type:hybrid;
mso-list-template-ids:903799028 67698689 67698691 67698693 67698689 67698691 \
67698693 67698689 67698691 67698693;} @list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi \
all,<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I \
am testing with SyslogSourceTCP and MultiSyslogSourceTCP.<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">My \
problem is that the Source loses syslog-events but depends on the tools which sent us \
syslog.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">• \
If I use a tool like QRadar the Source loses events (No exceptions, ni trace in \
log about the lost)<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">• \
If I use a adhoc software (log4j appender o cat | nc command or …), there aren't \
lost events.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I \
have configured flume of a lot of ways, but I always have the same \
result. Have to be the syslog client of a special way?<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks.</span><o:p></o:p></p>
<p class="MsoNormal"><a \
href="https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance&# \
43;Measurements">https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Measurements</a><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"> \
Smaine Kahlouch [mailto:smaine.kahlouch@smartjog.com] <br>
<b>Sent:</b> jueves, 05 de marzo de 2015 10:11<br>
<b>To:</b> user@flume.apache.org<br>
<b>Subject:</b> Re: Syslog TCP performances issue with \
filechannel<o:p></o:p></span></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Actually the batchSize is configured on sink level.<br>
I didn't find this option on file channel.<br>
<br>
Furthermore, the source batchSize can't be configured because it is a syslog-ng tool \
which doesn't have this capability.<br> I tried with "netcat" source and i \
face the same behaviour.<br> <br>
I guess you're right, for each event there's a fsync which causes the heavy load on \
diks.<br> However i've read this topic : <a \
href="https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Measurements">
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Measurements</a><br>
<br>
And they didn't have the same problem obviously.<br>
<br>
Regards,<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Smaine Kahlouch - Engineer, Research & Engineering<o:p></o:p></pre>
<pre>Arkena | T: +33 1 5868 6196<o:p></o:p></pre>
<pre>27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France<o:p></o:p></pre>
<pre>arkena.com<o:p></o:p></pre>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 03/04/15 20:08, Hari Shreedharan wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">You should probably increase the batch size, since each batch \
causes an fsync which slows things down.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><br>
Thanks, <o:p></o:p></p>
<div>
<p class="MsoNormal">Hari<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p>On Wed, Mar 4, 2015 at 6:28 AM, Smaine Kahlouch <<a \
href="mailto:smaine.kahlouch@smartjog.com" \
target="_blank">smaine.kahlouch@smartjog.com</a>> wrote:<o:p></o:p></p> \
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-right:0cm"> <div>
<p class="MsoNormal">Hi all,<br>
<br>
I'm currently doing benchmarks on flume.<br>
We're planning to use flume with syslogtcp as source and filechannel in order to have \
avoid data loss.<br> <br>
The performances are quiet good when a memorychannel is used :<br>
~<b>100 000events/sec</b> (event size = 600bytes)<br>
<br>
But as soon as i switch to filechannel the performances drop drammatically:<br>
~<b>300events/sec</b><br>
<br>
Despite this poor result, the behaviour is really strange because i have a heavy disk \
usage (all the disks), near 100%.<br> <br>
I use a tool provided by syslog-ng in order to generate syslog logs : <a \
href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/loggen.1.html">
loggen</a><br>
<br>
ex : loggen -i -I 3000000 --size 600 --active-connections 200 myflumehost 20515<br>
<br>
<br>
Flume version : 1.5.2<br>
Operating System : Centos 6<br>
<br>
Please find my flume configuration enclosed. The filechannel is spread over 5 disks \
in order to improve performance.<br> <br>
<br>
Could you please help me to configure properly syslogtcp source with filechannel \
?<br> <br>
Regards,<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Smaine Kahlouch - Engineer, Research & Engineering<o:p></o:p></pre>
<pre>Arkena | T: +33 1 5868 6196<o:p></o:p></pre>
<pre>27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France<o:p></o:p></pre>
<pre>arkena.com<o:p></o:p></pre>
</div>
<p class="MsoNormal"><flume.conf><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre><o:p> </o:p></pre>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic