[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] Regarding GSoC16 Project: syslog-ng as a command line tool
From: Scheidler, Balázs <balazs.scheidler () balabit ! com>
Date: 2016-03-07 7:44:55
Message-ID: CANWQT2P8sxd4aSGc1LOeu_6R9kAkLQEHLzA705nnAbh_8y0FxQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi,
I've Cced the syslog-ng list, so others can see this description as well.
The goal here is to change syslog-ng to be able to process input from stdin
and produce output to stdout and exit when the EOF is reached, while
processing a "potentially" simplified syslog-ng configuration block. A bit
similar to how awk works, but instead of the awk language, use syslog-ng's
constructs.
E.g.
$ syslog-ng -P 'parser { kv-parser(); }; destination {
stdout(template("$(format-json *"))); };' < log-file-with-keywords.log >
log-file-in-json
Where -P would be equivalent to --pipe, the next argument is a
configuration snippet. This snippet would get all logs as read from stdin()
and that would be default functionality provided by the --pipe construct.
Probably --pipe would use a configuration file something like:
log {
source { stdin(); };
# program supplied by --pipe would be included here
pipe-program();
};
This is the basic idea, some customization would be useful, like:
- being able to specify the source as well (so the above wouldn't be
hardwired, only used as a default)
- maybe use a default for output as well, so I wouldn't have to include
it in the actual language
- examples: a few examples how to process existing log files from
postfix, iptables, snort/suricata, etc to convert their logs into json.
This would be both a set of examples and could also drive the functionality
further, so we don't miss important options.
Code wise, this would need something like:
- stdin source: would probably only be an SCL wrapper around the
existing file() driver; e.g. file("/dev/stdin"); check for portability on
various UNIXes (freebsd, solaris, macosx)
- stdout destination: would probably only be an SCL wrapper around the
existing file driver, e.g. file("/dev/stdout");
- stderr destinatiion: would probably only be an SCL wrapper around the
existing file driver, e.g. file("/dev/stderr");
- a means to construct the complete configuration as passed in by
command line arguments; would probably need to register confgen blocks, and
a larger configuration frame where they are inserted.
- a means to terminate syslog-ng when we read the entire input and all
outputs are finished. (would probably be the most difficult)
- tests
Hope this helps.
Bazsi
--
Bazsi
On Mon, Mar 7, 2016 at 6:37 AM, Viraj Madhawa <emadhawa23@gmail.com> wrote:
> Dear Sir,
>
> I'm a computer engineering undergraduate student of University of
> Peradeniya Sri Lanka. During my internship period I done a log managment
> system project to a bank (using syslog protocol) . I'm quite interested
> about this project idea and I like to get more details about it (ask some
> questions).
>
> --
> Thank you,
> Viraj Premaratne
>
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div><div><div><div><div><div><div><div>Hi,<br><br></div><div>I've \
Cced the syslog-ng list, so others can see this description as \
well.<br></div><div><br></div>The goal here is to change syslog-ng to be able to \
process input from stdin and produce output to stdout and exit when the EOF is \
reached, while processing a "potentially" simplified syslog-ng \
configuration block. A bit similar to how awk works, but instead of the awk \
language, use syslog-ng's constructs.<br><br></div>E.g. <br><br></div>$ syslog-ng \
-P 'parser { kv-parser(); }; destination { stdout(template("$(format-json \
*"))); };' < log-file-with-keywords.log > \
log-file-in-json<br><br></div>Where -P would be equivalent to --pipe, the next \
argument is a configuration snippet. This snippet would get all logs as read from \
stdin() and that would be default functionality provided by the --pipe construct. \
Probably --pipe would use a configuration file something like:<br><br></div>log \
{<br></div> source { stdin(); };<br></div> # program supplied by --pipe would be \
included here<br></div> \
pipe-program();<br>};<br><div><div><div><div><div><div><br></div><div>This is the \
basic idea, some customization would be useful, like:<br><ul><li>being able to \
specify the source as well (so the above wouldn't be hardwired, only used as a \
default)</li><li>maybe use a default for output as well, so I wouldn't have to \
include it in the actual language</li><li>examples: a few examples how to process \
existing log files from postfix, iptables, snort/suricata, etc to convert their logs \
into json. This would be both a set of examples and could also drive the \
functionality further, so we don't miss important options.</li></ul><p>Code wise, \
this would need something like:</p><ul><li>stdin source: would probably only be an \
SCL wrapper around the existing file() driver; e.g. file("/dev/stdin"); \
check for portability on various UNIXes (freebsd, solaris, macosx)<br></li><li>stdout \
destination: would probably only be an SCL wrapper around the existing file driver, \
e.g. file("/dev/stdout");</li><li>stderr destinatiion: would probably only \
be an SCL wrapper around the existing file driver, e.g. \
file("/dev/stderr");</li><li>a means to construct the complete \
configuration as passed in by command line arguments; would probably need to register \
confgen blocks, and a larger configuration frame where they are \
inserted.<br></li><li>a means to terminate syslog-ng when we read the entire input \
and all outputs are finished. (would probably be the most \
difficult)</li><li>tests</li></ul><p>Hope this \
helps.</p><p>Bazsi<br></p></div></div></div></div></div></div></div><div \
class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div \
dir="ltr">-- <br>Bazsi<br></div></div></div> <br><div class="gmail_quote">On Mon, Mar \
7, 2016 at 6:37 AM, Viraj Madhawa <span dir="ltr"><<a \
href="mailto:emadhawa23@gmail.com" \
target="_blank">emadhawa23@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Dear Sir, <div><br></div><div>I'm a \
computer engineering undergraduate student of University of Peradeniya Sri Lanka. \
During my internship period I done a log managment system project to a bank (using \
syslog protocol) . I'm quite interested about this project idea and I like to \
get more details about it (ask some questions). <span class="HOEnZb"><font \
color="#888888"><br clear="all"><div><br></div>-- </font></span></div><span \
class="HOEnZb"><font color="#888888"><div>Thank you, <br><div><div dir="ltr">Viraj \
Premaratne<div><br></div><div><br></div></div></div> </div></font></span></div>
</blockquote></div><br></div>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic