[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] using VARARGS correctly
From: Daniel Ehrlich <Daniel.Ehrlich () usq ! edu ! au>
Date: 2021-05-19 23:35:04
Message-ID: SYBPR01MB6448A4FDE982720C88781D05B92B9 () SYBPR01MB6448 ! ausprd01 ! prod ! outlook ! com
[Download RAW message or body]
Hi Everyone,
I am having an issue when the Zulu timestamp is between 10 and 23:59. i.e. =
the logs format differently before 10AM and after 10AM.
I have captured in a tcpdump the syslogs coming in and they both seem the s=
ame.
We're at GMT+10 so this event was as 11:14:14 on 19th May.
Msg: 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 1621386854,23.85,31.48,5.91,n/a=
,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d
This event was 09:07:47 the next day the 20th May:
Msg: 2021-05-19T23:07:46Z 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/=
a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d
In the output files, both events go to the 0519.log file, until 10AM or 00:=
00:00Z the next day.
First event logs as:
May 19 11:14:14 10.18.0.14 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO 162138685=
4,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O
Second event logs as:
May 19 23:07:46 10.18.0.14 E-MICRO 1621465666,24.96,32.54,7.36,n/a,n/a,n/a,=
n/a,n/a,n/a,O,O
I assume some built-in filtering is changing the way these are parsed in sy=
slog-ng?
I have tried to play with raw message filtering but it doesn't take the con=
f file:
@version:3.5
@include "scl.conf"
# syslog-ng configuration file.
#
options {
chain_hostnames(no);
create_dirs (yes);
dir_perm(0755);
dns_cache(yes);
keep_hostname(yes);
log_fifo_size(2048);
log_msg_size(8192);
perm(0644);
time_reopen (10);
use_dns(yes);
use_fqdn(yes);
};
source s_network {
udp(port(514));
};
source attivo {
tcp(port(514));
};
### DESTINATIONS
destination d_files_splunk {
file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_=
dirs(yes));
};
destination d_files_nti {
file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_=
dirs(yes) template(t_nti));
};
### FILTERS
filter nti {
host("10.18.0.14" type(glob));
};
filter splunk {
not (filter(nti));
};
### LOG
log {
source(s_network);
#filter(splunk);
destination(d_files_splunk);
};
log {
source(s_network);
filter(nti);
destination(d_files_nti);
};
log {
source(attivo);
# filter(splunk);
destination(d_files_splunk);
};
### TEMPLATES
template t_nti {
template("${RAWMSG}\n")
};
Any help is appreciated.
Thanks
Daniel Ehrlich
__________________________________________________________________
This email (including any attached files) is confidential and is =
for the intended recipient(s) only. If you received this email by =
mistake, please, as a courtesy, tell the sender, then delete this =
email.
The views and opinions are the originator's and do not necessarily =
reflect those of the University of Southern Queensland. Although =
all reasonable precautions were taken to ensure that this email =
contained no viruses at the time it was sent we accept no =
liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider =
of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} \
</style> </head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> Hi Everyone,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> I am having an issue when the Zulu timestamp is between 10 and \
23:59. i.e. the logs format differently before 10AM and after 10AM.</div> <div \
style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: \
rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> I have captured in a tcpdump the syslogs coming in and they \
both seem the same. </div> <div style="font-family: Calibri, Arial, Helvetica, \
sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span style="color: rgb(0, 0, 0); \
font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">We're at GMT+10 \
so this event was as </span><span style="color: rgb(0, 0, 0); font-family: \
Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">11:14:14 on 19th \
May.</span></div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; \
font-size: 12pt; color: rgb(0, 0, 0);"> <span style="color: rgb(0, 0, 0); \
font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Msg: \
2021-05-19T1:14:14Z 10.18.0.14 E-MICRO \
1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d<br> </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> <div style="font-style: normal; font-variant-ligatures: \
normal; font-variant-caps: normal; font-weight: 400; font-size: 12pt; font-family: \
Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);"> <span style="font-size: \
12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);"><br> \
</span></div> <div style="font-style: normal; font-variant-ligatures: normal; \
font-variant-caps: normal; font-weight: 400; font-size: 12pt; font-family: Calibri, \
Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);"> <span style="font-size: 12pt; \
font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">This event \
was 09:07:47 the next day the 20th May:</span></div> <div style="font-style: normal; \
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; \
font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, \
0);"> <span style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, \
sans-serif; color: rgb(0, 0, 0);">Msg: 2021-05-19T23:07:46Z 10.18.0.14 E-MICRO \
1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O\0x0a\0x0d</span></div> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> In the output files, both events go to the 0519.log file, \
until 10AM or 00:00:00Z the next day.</div> <div style="font-family: Calibri, Arial, \
Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> First event logs \
as:</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: \
12pt; color: rgb(0, 0, 0);">
May 19 11:14:14 10.18.0.14 2021-05-19T1:14:14Z 10.18.0.14 E-MICRO \
1621386854,23.85,31.48,5.91,n/a,n/a,n/a,n/a,n/a,n/a,O,O<br> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> Second event logs as:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);">
May 19 23:07:46 10.18.0.14 E-MICRO \
1621465666,24.96,32.54,7.36,n/a,n/a,n/a,n/a,n/a,n/a,O,O<br> </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; \
color: rgb(0, 0, 0);"> I assume some built-in filtering is changing the way these are \
parsed in syslog-ng?</div> <div style="font-family: Calibri, Arial, Helvetica, \
sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </div>
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> I have tried to play with raw \
message filtering but it doesn't take the conf file:</div> <div \
style="color:rgb(212,212,212); background-color:rgb(30,30,30); \
font-family:"Droid Sans Mono",monospace,monospace,"Droid Sans \
Fallback"; font-weight:normal; font-size:14px; line-height:19px"> <span \
style="color: rgb(106, 153, 85);">@version:3.5</span><br> </div>
<div style="color:rgb(212,212,212); background-color:rgb(30,30,30); \
font-family:"Droid Sans Mono",monospace,monospace,"Droid Sans \
Fallback"; font-weight:normal; font-size:14px; line-height:19px"> <span><span \
style="color:rgb(106,153,85)"> <div>@include "scl.conf"</div>
<div><br>
</div>
<div># syslog-ng configuration file.</div>
<div>#</div>
<div><br>
</div>
<div>options {</div>
<div> chain_hostnames(no);</div>
<div> create_dirs (yes);</div>
<div> dir_perm(0755);</div>
<div> dns_cache(yes);</div>
<div> keep_hostname(yes);</div>
<div> log_fifo_size(2048);</div>
<div> log_msg_size(8192);</div>
<div> perm(0644);</div>
<div> time_reopen (10);</div>
<div> use_dns(yes);</div>
<div> use_fqdn(yes);</div>
<div>};</div>
<div><br>
</div>
<div>source s_network {</div>
<div> udp(port(514));</div>
<div>};</div>
<div><br>
</div>
<div>source attivo {</div>
<div> tcp(port(514));</div>
<div>};</div>
<div><br>
</div>
<div>### DESTINATIONS</div>
<div>destination d_files_splunk {</div>
<div> file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" \
create_dirs(yes));</div> <div>};</div>
<div>destination d_files_nti {</div>
<div> file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" \
create_dirs(yes) template(t_nti));</div> <div>};</div>
<div><br>
</div>
<div>### FILTERS</div>
<div>filter nti {</div>
<div> host("10.18.0.14" type(glob));</div>
<div>};</div>
<div>filter splunk {</div>
<div> not (filter(nti));</div>
<div>};</div>
<div><br>
</div>
<div>### LOG</div>
<div>log {</div>
<div> source(s_network);</div>
<div> #filter(splunk);</div>
<div> destination(d_files_splunk);</div>
<div>};</div>
<div>log {</div>
<div> source(s_network);</div>
<div> filter(nti);</div>
<div> destination(d_files_nti);</div>
<div>};</div>
<div>log {</div>
<div> source(attivo);</div>
<div># filter(splunk);</div>
<div> destination(d_files_splunk);</div>
<div>};</div>
<div><br>
</div>
<div>### TEMPLATES</div>
<div>template t_nti {</div>
<div> template("${RAWMSG}\n")</div>
<span>};</span><br>
</span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> <span \
style="background-color:rgb(255, 255, 255);display:inline !important">Any help is \
appreciated.</span><br> </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> <span \
style="background-color:rgb(255, 255, 255);display:inline \
!important">Thanks</span></div> <div \
style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; \
color:rgb(0,0,0); background-color:rgb(255,255,255)"> <span \
style="background-color:rgb(255, 255, 255);display:inline !important">Daniel \
Ehrlich</span></div> </div>
</div>
<div>__________________________________________________________________<div>This \
email (including any attached files) is confidential and is <div>for the \
intended recipient(s) only. If you received this email by <div>mistake, please, \
as a courtesy, tell the sender, then delete this <div>email.<br><div>The views \
and opinions are the originator's and do not necessarily <div>reflect those of \
the University of Southern Queensland. Although <div>all reasonable precautions \
were taken to ensure that this email <div>contained no viruses at the time it \
was sent we accept no <div>liability for any losses arising from its \
receipt.<br><div>The University of Southern Queensland is a registered \
provider <div>of education with the Australian Government.<div>(CRICOS \
Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)</body> </html>
[Attachment #4 (unknown)]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic