[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] RE syslog-ng OSE parsing
From: Scheidler, Balázs <balazs.scheidler () balabit ! com>
Date: 2018-06-22 13:18:15
Message-ID: CANWQT2NqMfPceBO4xPkM2kQxAZnYN25zt8LOj+JexZ8XQKUZLA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
I have added handling of this timestamp into our cisco-parser():
https://github.com/balabit/syslog-ng/pull/2134
--
Bazsi
On Wed, Jun 20, 2018 at 11:42 AM, Scheidler, Balázs <
balazs.scheidler@balabit.com> wrote:
> Hi,
>
> This format seems quite out of the ordinary, which the cisco parser
> wouldn't cope with at least today. There would be two level of support for
> this format:
>
> 1. at the very least, process date/host and message properly
> 2. parse the name-value pairs in brackets (e.g. Login Date/Time)
>
> The first one should be doable by extending the cisco parser. But before
> going there, can you pls reproduce the log message character-by-character
> how syslog-ng receives it? In the original example, I think an extra space
> is inserted between the timestamp and the following colon. Cisco logs tend
> to be ": " separater (e.g. a colon and a space). If that assumption is
> true, then support for this can be added by changing
> cisco-timestamp-parser().
>
>
>
> --
> Bazsi
>
> On Wed, Jun 20, 2018 at 7:45 AM, Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au
> > wrote:
>
> > Hi Bazsi and Gabor,
> >
> >
> >
> > Bazsi, so maybe I should update syslog-ng and then try the cisco parser?
> >
> >
> >
> > The logs are coming from Cisco Unified Call Manager logs, running v10.5.2.
> >
> >
> >
> > Kind Regards,
> >
> >
> >
> > *Daniel Ehrlich*
> >
> > ICT Security Officer
> > Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
> >
> >
> >
> > *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf
> > Of *Nagy, Gábor
> > *Sent:* Tuesday, 19 June 2018 9:05 PM
> >
> > *To:* Syslog-ng users' and developers' mailing list <
> > syslog-ng@lists.balabit.hu>
> > *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
> >
> >
> >
> > Hi Daniel!
> >
> > > ... how to filter these logs before applying that parser to them?
> > It really depends on the format of your log messages, especially on the
> > non-compliant ones.
> >
> > Of course if they have a well-defined structure you can use a specific
> > filter on these messages and then you can use the parser.
> >
> > To process log messages differently in a log path you can use the new
> > if-elif branching in your config (or the previous junction-channel setup),
> > which is released in version 3.15.
> >
> > So you can parse the non-complaint messages only, and the rest can go
> > through the usual path.
> >
> > The if-else branching will be documented in the upcoming documentation,
> > until then you can read the PR about the feature which has a description.
> >
> > https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863
> >
> >
> >
> > An example would be:
> >
> > source{ ... flags(no-parse); };
> >
> > if(filter(f_non_compliant)) { parser { kv-parser(); }; }
> >
> > else {};
> >
> > destination {....};
> >
> >
> >
> > I know I have missed the definition of "f_non_compliant", but I don't
> > know what would be a good filter, hence not answering your original
> > question. :)
> >
> > I would like to improve our cisco parser if I prove that this is a Cisco
> > format:
> >
> > > Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed:
> > %[Login Date/Time=06/15/2018 at 09:57:27][Login IP
> > Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login
> > UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]:
> > Login Authentication failed.
> >
> > What kind of device is sending this log message?
> >
> >
> >
> > Regards,
> >
> > Gabor
> >
> >
> >
> > On Tue, Jun 19, 2018 at 8:13 AM, Daniel Ehrlich <
> > Daniel.Ehrlich@usq.edu.au> wrote:
> >
> > Thanks Gabor,
> >
> >
> >
> > I'm wondering if you have any suggestion on how to filter these logs
> > before applying that parser to them?
> >
> >
> >
> > I have numerous sources coming in on the udp 514 listener. Then logging
> > them to files which Splunk reads.
> >
> >
> >
> > Below is my current config….:
> >
> > source s_network {
> >
> > udp(port(514));
> >
> > };
> >
> >
> >
> > # Parser
> >
> >
> >
> > # Rewrite
> >
> >
> >
> > #Destinations
> >
> > destination d_files_splunk {
> >
> > file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log"
> > create_dirs(yes));
> >
> > };
> >
> > destination d_files_seamail {
> >
> > file("/opt/splunk/var/lib/splunk/syslog-ng/seamail/$MONTH$DAY.log"
> > create_dirs(yes));
> >
> > };
> >
> >
> >
> > # Filters
> >
> > filter seamail {
> >
> > host("q=*" type(glob));
> >
> > };
> >
> > filter splunk {
> >
> > not (filter(seamail));
> >
> > };
> >
> >
> >
> > # Log
> >
> > log {
> >
> > source(s_network);
> >
> > filter(splunk);
> >
> > destination(d_files_splunk);
> >
> > };
> >
> > log {
> >
> > source(s_network);
> >
> > filter(seamail);
> >
> > destination(d_files_seamail);
> >
> > };
> >
> >
> >
> > Kind Regards,
> >
> >
> >
> > *Daniel Ehrlich*
> >
> > ICT Security Officer
> > Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
> >
> >
> >
> > *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf
> > Of *Nagy, Gábor
> > *Sent:* Friday, 15 June 2018 6:20 PM
> > *To:* Syslog-ng users' and developers' mailing list <
> > syslog-ng@lists.balabit.hu>
> > *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
> >
> >
> >
> > Hi Daniel!
> >
> >
> >
> > Your log does not conform to either RFC-3164 or RFC-5424, it looks like
> > kind-of cisco format to me.
> >
> > I've tried one or two ideas, using our `default-network-drivers()` which
> > includes the cisco-parser() but it is not dealing with it properly.
> >
> >
> >
> > On second try, I've experimented with `kv-parser()` so the key-value
> > parts in the message will be parsed in a structured format and after that
> > you can use a rewrite rule to modify HOST field of the message.
> >
> > I had to adjust the pair-separator option to get an optimal result.
> >
> > `parser { kv-parser( prefix(".kv.") pair-separator("]")); };`
> >
> >
> >
> > After a successful parsing, you can use a rewrite rule to set the
> > HOSTNAME.
> >
> > rewrite { set("${.kv.ID}" value("HOST")); };
> >
> > There is one catch: you need to know the name of the key: "Node ID" or
> > "Hostname", which comes from the log message.
> >
> > So, if this expression is not fix, then your config will not work.
> >
> >
> >
> > You can also see that I've added the key "ID" rather than "Node ID",
> > since your log message does not quote the keys in it and
> >
> > in this case kv-parser will only use the "ID" from "Node ID" part which
> > leads to that similar key-values are overwritten: "App ID", "Cluster ID"
> >
> >
> >
> > I will think about it, but hopefully others will come up with a better
> > idea. :)
> >
> >
> >
> > Regards,
> >
> > Gabor
> >
> >
> >
> > On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <
> > Daniel.Ehrlich@usq.edu.au> wrote:
> >
> > Hi,
> >
> >
> >
> > Hoping you can assist me, I haven't really come across anything that
> > makes full sense to me in my searching of various sites/forums.
> >
> >
> >
> > This is an example log.
> >
> > Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login
> > Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
> > Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
> > ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
> >
> >
> >
> > Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
> > cucmsub-prd-t2 ? i.e. Node ID=
> >
> >
> >
> > Thanks you
> >
> >
> >
> > Kind Regards,
> >
> >
> >
> > *Daniel Ehrlich*
> >
> > *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
> >
> > ICT Security Officer
> > ICT Client Services|Infrastructure Services
> > Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
> >
> > Toowoomba | Queensland | 4350 | Australia
> >
> >
> >
> > _____________________________________________________________
> >
> > This email (including any attached files) is confidential and is for the intended \
> > recipient(s) only. If you received this email by mistake, please, as a courtesy, \
> > tell the sender, then delete this email.
> >
> >
> > The views and opinions are the originator's and do not necessarily reflect those \
> > of the University of Southern Queensland. Although all reasonable precautions \
> > were taken to ensure that this email contained no viruses at the time it was sent \
> > we accept no liability for any losses arising from its receipt.
> >
> >
> > The University of Southern Queensland is a registered provider of education with \
> > the Australian Government.
> > (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
> >
> >
> > ____________________________________________________________
> > __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support
> > /documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> >
> >
> > On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <
> > Daniel.Ehrlich@usq.edu.au> wrote:
> >
> > Hi,
> >
> >
> >
> > Hoping you can assist me, I haven't really come across anything that
> > makes full sense to me in my searching of various sites/forums.
> >
> >
> >
> > This is an example log.
> >
> > Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login
> > Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
> > Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
> > ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
> >
> >
> >
> > Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
> > cucmsub-prd-t2 ? i.e. Node ID=
> >
> >
> >
> > Thanks you
> >
> >
> >
> > Kind Regards,
> >
> >
> >
> > *Daniel Ehrlich*
> >
> > *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
> >
> > ICT Security Officer
> > ICT Client Services|Infrastructure Services
> > Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
> >
> > Toowoomba | Queensland | 4350 | Australia
> >
> >
> >
> > _____________________________________________________________
> >
> > This email (including any attached files) is confidential and is for the intended \
> > recipient(s) only. If you received this email by mistake, please, as a courtesy, \
> > tell the sender, then delete this email.
> >
> >
> > The views and opinions are the originator's and do not necessarily reflect those \
> > of the University of Southern Queensland. Although all reasonable precautions \
> > were taken to ensure that this email contained no viruses at the time it was sent \
> > we accept no liability for any losses arising from its receipt.
> >
> >
> > The University of Southern Queensland is a registered provider of education with \
> > the Australian Government.
> > (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
> >
> >
> > ____________________________________________________________
> > __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support
> > /documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> > _____________________________________________________________
> >
> > This email (including any attached files) is confidential and is for the intended \
> > recipient(s) only. If you received this email by mistake, please, as a courtesy, \
> > tell the sender, then delete this email.
> >
> >
> > The views and opinions are the originator's and do not necessarily reflect those \
> > of the University of Southern Queensland. Although all reasonable precautions \
> > were taken to ensure that this email contained no viruses at the time it was sent \
> > we accept no liability for any losses arising from its receipt.
> >
> >
> > The University of Southern Queensland is a registered provider of education with \
> > the Australian Government.
> > (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
> >
> >
> > ____________________________________________________________
> > __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support
> > /documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> > _____________________________________________________________
> > This email (including any attached files) is confidential and is for the intended \
> > recipient(s) only. If you received this email by mistake, please, as a courtesy, \
> > tell the sender, then delete this email.
> > The views and opinions are the originator's and do not necessarily reflect those \
> > of the University of Southern Queensland. Although all reasonable precautions \
> > were taken to ensure that this email contained no viruses at the time it was sent \
> > we accept no liability for any losses arising from its receipt.
> > The University of Southern Queensland is a registered provider of education with \
> > the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, \
> > TEQSA PRV12081 )
> >
> > ____________________________________________________________
> > __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support
> > /documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
>
[Attachment #5 (text/html)]
<div dir="ltr"><div> I have added handling of this timestamp into our \
cisco-parser(): <br></div><div><br></div><div><a \
href="https://github.com/balabit/syslog-ng/pull/2134">https://github.com/balabit/syslog-ng/pull/2134</a></div><div><br></div><div><br></div></div><div \
class="gmail_extra"><br clear="all"><div><div class="gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div> \
<br><div class="gmail_quote">On Wed, Jun 20, 2018 at 11:42 AM, Scheidler, Balázs \
<span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" \
target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="auto"><div \
dir="ltr"><div>Hi,</div><div><br></div><div>This format seems quite out of the \
ordinary, which the cisco parser wouldn't cope with at least today. There would \
be two level of support for this format:</div><div><ol><li>at the very least, process \
date/host and message properly<br></li><li>parse the name-value pairs in brackets \
(e.g. Login Date/Time)</li></ol><div>The first one should be doable by extending the \
cisco parser. But before going there, can you pls reproduce the log message \
character-by-character how syslog-ng receives it? In the original example, I think an \
extra space is inserted between the timestamp and the following colon. Cisco logs \
tend to be ": " separater (e.g. a colon and a space). If that assumption is \
true, then support for this can be added by changing \
cisco-timestamp-parser().</div><span class="HOEnZb"><font \
color="#888888"><div><br></div></font></span></div><span class="HOEnZb"><font \
color="#888888"><br></font></span></div></div><div class="gmail_extra"><span \
class="HOEnZb"><font color="#888888"><br clear="all"><div><div \
class="m_-5229866869232812868m_-6635595554444740672gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr">-- \
<br>Bazsi<br></div></div></div></font></span><div><div class="h5"> <br><div \
class="gmail_quote">On Wed, Jun 20, 2018 at 7:45 AM, Daniel Ehrlich <span \
dir="ltr"><<a href="mailto:Daniel.Ehrlich@usq.edu.au" \
target="_blank">Daniel.Ehrlich@usq.edu.au</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-AU"><div \
class="m_-5229866869232812868m_-6635595554444740672m_5360552797081291590WordSection1"><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Hi \
Bazsi and Gabor,<u></u><u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Bazsi, \
so maybe I should update syslog-ng and then try the cisco \
parser?<u></u><u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The \
logs are coming from Cisco Unified Call Manager logs, running \
v10.5.2.<u></u><u></u></span></p><span><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> \
<u></u></span></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Kind \
Regards,</span><span style="font-size:11.0pt;color:black"><u></u><u></u></span></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><u></u> \
<u></u></span></p><p class="MsoNormal"><b><span \
style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#7f7f7f">Daniel \
Ehrlich<u></u><u></u></span></b></p><p class="MsoNormal"><span \
style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#7f7f7f">ICT \
Security Officer</span><span \
style="font-size:11.0pt;font-family:"Verdana",sans-serif;color:#7f7f7f"><br></span><span \
style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#7f7f7f">Phone: \
+61 7 4687 5600 Email: </span><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a \
href="mailto:Daniel.Ehrlich@usq.edu.au" \
target="_blank">Daniel.Ehrlich@usq.edu.<wbr>au</a></span><span \
style="font-family:"Calibri",sans-serif;color:black"><u></u><u></u></span></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> \
<u></u></span></p></span><p class="MsoNormal"><b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif" \
lang="EN-US">From:</span></b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US"> \
syslog-ng [mailto:<a href="mailto:syslog-ng-bounces@lists.balabit.hu" \
target="_blank">syslog-ng-bounces@list<wbr>s.balabit.hu</a>] <b>On Behalf Of \
</b>Nagy, Gábor<br><b>Sent:</b> Tuesday, 19 June 2018 9:05 PM</span></p><div><div \
class="m_-5229866869232812868m_-6635595554444740672h5"><br><b>To:</b> Syslog-ng \
users' and developers' mailing list <<a \
href="mailto:syslog-ng@lists.balabit.hu" \
target="_blank">syslog-ng@lists.balabit.hu</a>><br><b>Subject:</b> Re: [syslog-ng] \
RE syslog-ng OSE parsing<u></u><u></u></div></div><p></p><div><div \
class="m_-5229866869232812868m_-6635595554444740672h5"><p class="MsoNormal"><u></u> \
<u></u></p><div><p class="MsoNormal">Hi Daniel!<br><br>> ... how to filter these \
logs before applying that parser to them?<br>It really depends on the format of your \
log messages, especially on the non-compliant ones.<br><br>Of course if they have a \
well-defined structure you can use a specific filter on these messages and then you \
can use the parser.<u></u><u></u></p><div><p class="MsoNormal">To process log \
messages differently in a log path you can use the new if-elif branching in your \
config (or the previous junction-channel setup), which is released in version \
3.15.<u></u><u></u></p></div><div><p class="MsoNormal">So you can parse the \
non-complaint messages only, and the rest can go through the usual \
path.<u></u><u></u></p></div><div><p class="MsoNormal">The if-else branching will be \
documented in the upcoming documentation, until then you can read the PR about the \
feature which has a description.<u></u><u></u></p></div><div><p class="MsoNormal"><a \
href="https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863" \
target="_blank">https://github.com/balabit/sys<wbr>log-ng/pull/1856#issuecomment-<wbr>369049863</a><u></u><u></u></p></div><div><p \
class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">An example \
would be:<u></u><u></u></p></div><div><p class="MsoNormal">source{ ... \
flags(no-parse); };<u></u><u></u></p></div><div><p \
class="MsoNormal">if(filter(f_non_compliant)) { parser { kv-parser(); }; \
}<u></u><u></u></p></div><div><p class="MsoNormal">else \
{};<u></u><u></u></p></div><div><p class="MsoNormal">destination \
{....};<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> \
<u></u></p></div><div><p class="MsoNormal">I know I have missed the definition of \
"f_non_compliant", but I don't know what would be a good filter, hence \
not answering your original question. :)<u></u><u></u></p></div><div><p \
class="MsoNormal">I would like to improve our cisco parser if I prove that this is a \
Cisco format:<u></u><u></u></p></div><p class="MsoNormal">> Jun 14 11:57:27 PM.685 \
UTC : %UC_LOGIN-4-AuthenticationFai<wbr>led: %[Login Date/Time=06/15/2018 at \
09:57:27][Login IP Address/Hostname=10.25.1.16][L<wbr>ogin Interface=cucm-uds][Login \
UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login \
Authentication failed.<u></u><u></u></p><div><p class="MsoNormal">What kind of device \
is sending this log message?<u></u><u></u></p></div><div><div><p \
class="MsoNormal"><u></u> <u></u></p></div><div><p \
class="MsoNormal">Regards,<u></u><u></u></p></div></div><div><p \
class="MsoNormal">Gabor<u></u><u></u></p></div></div><div><p \
class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">On Tue, Jun 19, 2018 \
at 8:13 AM, Daniel Ehrlich <<a href="mailto:Daniel.Ehrlich@usq.edu.au" \
target="_blank">Daniel.Ehrlich@usq.edu.au</a>> wrote:<u></u><u></u></p><blockquote \
style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-right:0cm"><div><div><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thanks \
Gabor,</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I'm \
wondering if you have any suggestion on how to filter these logs before applying that \
parser to them?</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I \
have numerous sources coming in on the udp 514 listener. Then logging them to files \
which Splunk reads.</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Below \
is my current config….:</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">source \
s_network {</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
udp(port(514));</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"># \
Parser</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"># \
Rewrite</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">#Destinations</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">destination \
d_files_splunk {</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
file("/opt/splunk/var/lib/splu<wbr>nk/syslog-ng/$HOST/$MONTH$DAY.<wbr>log" \
create_dirs(yes));</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">destination \
d_files_seamail {</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
file("/opt/splunk/var/lib/splu<wbr>nk/syslog-ng/seamail/$MONTH$DA<wbr>Y.log" \
create_dirs(yes));</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"># \
Filters</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">filter \
seamail {</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
host("q=*" type(glob));</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">filter \
splunk {</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
not (filter(seamail));</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"># \
Log</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">log \
{</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
source(s_network);</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
filter(splunk);</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
destination(d_files_splunk);</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">};</span><u></u><u></u></p><p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">log \
{</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
source(s_network);</span><u></u><u></u></p><p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> \
filter(seamail);</span><u></u><u></u></p><p class="MsoNormal"><span \
This email (including any attached files) is confidential and is for the intended \
recipient(s) only. If you received this email by mistake, please, as a courtesy, tell \
the sender, then delete this email.
The views and opinions are the originator's and do not necessarily reflect those \
of the University of Southern Queensland. Although all reasonable precautions were \
taken to ensure that this email contained no viruses at the time it was sent we \
accept no liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider of education with the \
Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA \
PRV12081 ) </pre></div></div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br> <br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
[Attachment #6 (text/plain)]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic