[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] setting sequenceId in forwarded log messages read from journald reader
From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere () oneidentity ! com>
Date: 2020-07-29 7:54:38
Message-ID: CY4PR19MB109561ABD576BCE9B325655B9D700 () CY4PR19MB1095 ! namprd19 ! prod ! outlook ! com
[Download RAW message or body]
Hello Peter,
I do not know a way to read sequence id without modifying the journal module. \
(Syslog-ng already do it to persist it's current position in journal. So it is \
definitely feasible to read it.)
I found this documentation about cursors: \
https://www.freedesktop.org/software/systemd/man/sd_journal_get_cursor.html, and I am \
worried about this part of the documentation: "The cursor string should be \
considered opaque and not be parsed by clients."
So parsing and using the sequence id as a core functionality will be unwise. However \
I can imagine a feature behind a configuration option (turned off by default), where \
syslog-ng will put the whole __CURSOR string into the NVTable, so it can be accessed \
via a custom parser.
Please tell me your opinion about this solution. (Especially: is there a particular \
reason why you don't want to modify the journal module?)
Best regards,
Laci
________________________________
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Peter Vollmer \
<peter.vollmer@gmail.com>
Sent: Monday, July 27, 2020 09:23
To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu>
Subject: [syslog-ng] setting sequenceId in forwarded log messages read from journald \
reader
CAUTION: This email originated from outside of the organization. Do not follow \
guidance, click links, or open attachments unless you recognize the sender and know \
the content is safe.
Hi,
I am currently trying to find a way to set meta.sequenceId of log messages that have \
been read from the locally running systemd-journal to forward them to a remote syslog \
server that expects the logs to contain a sequenceId according to RFC 5424 section \
7.3.1.
I found that a sequence number could be taken from the __CURSOR field "i=..." of the \
journald log:
# journalctl -o json-pretty -f
...
"__CURSOR" : "s=02a7b30ba17b4a43846f265706bd3a70;i=f01;b=ba633698f20848e480bca4e72476e4d3;m=1a355c1d5;t=5ab670340c8ea;x=33389988ef680e7e",
...
My problem is that the journal reader does not seem to parse the __CURSOR string when \
reading from journald logs. Is there a way to get this information into \
meta.sequenceId of the forwarded log without modifying the systemd-journal module in \
syslog-ng ?
Thank you for any ideas and best regards
Peter Vollmer
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} \
</style> </head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> Hello Peter,</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> I do not know a way to read sequence id without modifying the journal \
module. <span style="font-family: Arial, Helvetica, sans-serif; \
background-color: rgb(255, 255, 255); display: inline !important">(Syslog-ng already \
do it to persist it's current position in journal. So it is definitely feasible to \
read it.)</span></div> <div style="font-family: Arial, Helvetica, sans-serif; \
font-size: 12pt; color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> I found this documentation about cursors: <a \
href="https://www.freedesktop.org/software/systemd/man/sd_journal_get_cursor.html">https://www.freedesktop.org/software/systemd/man/sd_journal_get_cursor.html</a>, \
and <span style="color: rgb(0, 0, 0); font-family: Arial, Helvetica, sans-serif; \
font-size: 12pt;">I am worried about this part of the documentation:</span></div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> "<span style="font-family: "Times New Roman"; \
font-size: medium; background-color: rgb(255, 255, 255); display: inline \
!important">The cursor string should be considered opaque and not be parsed by \
clients.<b>"</b></span></div> <div style="font-family: Arial, Helvetica, \
sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> So parsing and using the sequence id as a core functionality will be \
unwise. However I can imagine a feature behind a configuration option (turned off by \
default), where syslog-ng will put the whole __CURSOR string into the NVTable, so it \
can be accessed via a custom parser.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> <br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> Please tell me your opinion about this solution. (Especially: is there \
a particular reason why you don't want to modify the journal module?)</div> <div \
style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, \
0);"> <br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> Best regards,</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> Laci</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> <br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, \
0, 0);"> <br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" \
style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng \
<syslog-ng-bounces@lists.balabit.hu> on behalf of Peter Vollmer \
<peter.vollmer@gmail.com><br> <b>Sent:</b> Monday, July 27, 2020 09:23<br>
<b>To:</b> syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] setting sequenceId in forwarded log messages read from \
journald reader</font> <div> </div>
</div>
<div>
<div style="background-color:#FFEB9C; width:100%; border-style:solid; \
border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; \
line-height:12pt; font-family:'Calibri'; color:Black; text-align:left"> <span \
style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from \
outside of the organization. Do not follow guidance, click links, or open attachments \
unless you recognize the sender and know the content is safe.</div> <br>
<div>
<div dir="ltr">Hi,
<div>I am currently trying to find a way to set meta.sequenceId of log messages that \
have been read from the locally running systemd-journal to forward them to a remote \
syslog server that expects the logs to contain a sequenceId according to RFC \
5424 section 7.3.1.</div>
<div><br>
</div>
<div>I found that a sequence number could be taken from the __CURSOR field \
"i=..." of the journald log:</div> <div><br>
</div>
<div># journalctl -o json-pretty -f<br>
</div>
<div>...</div>
<div> "__CURSOR" : \
"s=02a7b30ba17b4a43846f265706bd3a70;i=f01;b=ba633698f20848e480bca4e72476e4d3;m=1a355c1d5;t=5ab670340c8ea;x=33389988ef680e7e",<br>
</div>
<div>...</div>
<div>My problem is that the journal reader does not seem to parse the __CURSOR \
string when reading from journald logs. Is there a way to get this information into \
meta.sequenceId of the forwarded log without modifying the systemd-journal module in \
syslog-ng ?</div>
<div><br>
</div>
<div>Thank you for any ideas and best regards</div>
<div><br>
</div>
<div>Peter Vollmer</div>
</div>
</div>
</div>
</body>
</html>
[Attachment #4 (unknown)]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic