[prev in list] [next in list] [prev in thread] [next in thread]
List: syslog-ng
Subject: Re: [syslog-ng] Problem using Python Parser
From: Ronald Fenner <rfenner () gamecircus ! com>
Date: 2017-12-30 15:51:05
Message-ID: 34F4D568-72E1-495C-B2BB-F96D2F532B7D () gamecircus ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Including the file worked. I thought scl.conf was supposed to be including anything \
in the conf.d directory but looking into the file again it's actually the scl \
directory.
Ronald Fenner
Programmer
Game Circus LLC.
rfenner@gamecircus.com
> On Dec 30, 2017, at 1:12 AM, Antal Nemes <antal.nemes@balabit.com> wrote:
>
> Hi,
>
> The problem is syslog-ng cannot import the user code:
> [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', \
> class='PhpFpmParser', exception='None'
> Please try either to put the python user code
> python {
> class PhpFpmParser(object):
> ...
> };
> directly into syslog-ng.conf, or you can use @include to include the file that \
> contains the parser code.
> The _syslogng module is created by syslog-ng config parse: when syslog-ng processes \
> the python keyword with the user code. So the _syslogng import problem should be \
> resolved automatically with the above.
> Br,
> Antal
>
> On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <rfenner@gamecircus.com \
> <mailto:rfenner@gamecircus.com>> wrote: Here's the config I've redacted the Kafka \
> servers. I've tried adding a @module "mod-python" but it doesn't help.
> #############################################################################
> # Default syslog-ng.conf file which collects all local logs into a
> # single file called /var/log/messages.
> #
>
> @version: 3.11
> @module "mod-java"
> @include "scl.conf"
>
> source s_internal {internal();};
>
> source s_rtl_stream {
> unix-stream("/var/log/rtl-stream.sock" flags(no-parse));
> };
>
> source s_php_fpm {
> file("/var/log/php-fpm.www.log", flags(no-parse));
> };
>
> destination d_kafka_unstructured {
> kafka (
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("******")
> topic("syslog-ng-{{DEPLOYMENT}}")
> );
> };
>
> destination d_kafka_structured {
> kafka (
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("*****")
> topic("${topic}.{{DEPLOYMENT}}")
> template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
> );
> };
>
> destination d_syslog_ng {
> file("/var/log/syslog-ng");
> };
>
> destination d_test_log {
> file("/var/log/test.log");
> };
>
> parser p_json { json-parser(); };
>
> parser p_apache { apache-accesslog-parser(prefix("")); };
>
> parser p_php_fpm { python(class("PhpFpmParser")); };
>
> rewrite r_add_access_topic {
> set("access.log", value("topic"));
> };
>
> log {
> source(s_internal);
> destination(d_syslog_ng);
> };
>
> log {
> source(s_rtl_stream);
> parser(p_json);
> destination(d_kafka_structured);
> };
>
> log {
> source(s_php_fpm);
> parser(p_php_fpm);
> destination(d_test_log);
> };
>
>
> Here's the actual python parser:
> python {
> class PhpFpmParser(object):
> def parse(self, log_msg):
> msg = log_msg['MESSAGE']
> str_pos = msg.find('] ')
> if str_pos == -1:
> return True
> log_date = msg[1:str_pos]
> msg = msg[str_pos+2:]
> str_pos = msg.find(':')
> if str_pos == -1:
> return True
> level = msg[:str_pos]
> if "Parse" in level:
> level = "parse"
> elif "Compile" in level:
> level = 'compile'
> elif "Fatal" in level:
> level = 'fatal'
> elif "Core" in level:
> level = 'core'
> elif "Notice" in level:
> level = 'notice'
> elif "Warning" in level:
> level = 'warning'
> msg = msg[str_pos+2:].strip()
> log_msg['err_msg'] = msg
> log_msg['log_level'] = level
> log_msg['timestamp_utc'] = log_date
> return True
> };
>
> It's stored the the etc/conf.d directory within the syslog path..
>
> Ronald Fenner
> Programmer
> Game Circus LLC.
>
> rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>
>
> > On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler@balabit.com \
> > <mailto:balazs.scheidler@balabit.com>> wrote:
> > The _syslogng module is automatically created from the top level python block in \
> > syslog-ng and behaves similarly to the python __main__ module.
> > Do you explicitly import that module using the imports() option?
> >
> > Can you please post your config?
> >
> > On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com \
> > <mailto:rfenner@gamecircus.com>> wrote: When I try to load my config with a \
> > python parser in it I'm getting this error message: Starting \
> > /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python \
> > module; module='_syslogng', exception='exceptions.ImportError: No module named \
> > _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; \
> > parser='p_php_fpm', class='PhpFpmParser', exception='None' \
> > [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin \
> > name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
> > I build syslog-ng from source with the python options. Here is the -V output
> > syslog-ng 3 (3.11.1)
> > Installer-Version: 3.11.1
> > Revision:
> > Compile-Date: Dec 29 2017 21:24:13
> > Module-Directory: /opt/syslog-ng/lib/syslog-ng
> > Module-Path: /opt/syslog-ng/lib/syslog-ng
> > Available-Modules: \
> > snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-py \
> > thon,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongo \
> > db,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat
> >
> > Enable-Debug: off
> > Enable-GProf: off
> > Enable-Memtrace: off
> > Enable-IPv6: on
> > Enable-Spoof-Source: off
> > Enable-TCP-Wrapper: off
> > Enable-Linux-Caps: off
> > Enable-Systemd: off
> >
> > Not sure how to fix this as from what I can tell this module is supposed to be \
> > compiled in and automatically imported.
> >
> > Ronald Fenner
> > Programmer
> > Game Circus LLC.
> >
> > rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng \
> > <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng \
> > <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq \
> > <http://www.balabit.com/wiki/syslog-ng-faq>
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng \
> > <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng \
> > <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq \
> > <http://www.balabit.com/wiki/syslog-ng-faq>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng \
> <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng \
> <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq \
> <http://www.balabit.com/wiki/syslog-ng-faq>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
[Attachment #5 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space;" class="">Including the file worked. I thought \
scl.conf was supposed to be including anything in the conf.d directory but looking \
into the file again it's actually the scl directory.<div class=""><br \
class=""></div><div class=""><br class=""><div class=""> <div style="color: rgb(0, 0, \
0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: \
normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: \
-webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; \
word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; \
word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; " class=""><div style="color: rgb(0, 0, 0); font-family: \
Helvetica; font-style: normal; font-variant: normal; font-weight: normal; \
letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; \
text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: \
0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " \
class=""><div class=""><div class="">Ronald Fenner</div><div \
class="">Programmer</div><div class="">Game Circus LLC.</div></div><div class=""><br \
class=""></div><div class=""><a href="mailto:rfenner@gamecircus.com" \
class="">rfenner@gamecircus.com</a></div></div></div> </div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Dec 30, 2017, at \
1:12 AM, Antal Nemes <<a href="mailto:antal.nemes@balabit.com" \
class="">antal.nemes@balabit.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div \
class=""><div class=""><div class=""> Hi,<br class=""><br \
class=""></div>The problem is syslog-ng cannot import the user code: <br \
class=""></div><div class=""><div class="">[2017-12-29T23:00:05.814066] Error looking \
Python parser class; parser='p_php_fpm', class='PhpFpmParser', \
exception='None'</div><div class=""><br class=""></div><div class="">Please try \
either to put the python user code <br class=""></div><div class="">python \
{</div><div class=""> class PhpFpmParser(object):</div><div \
class=""> ...<br class=""></div><div class="">};<br \
class=""></div><div class="">directly into syslog-ng.conf, or you can use @include \
to include the file that contains the parser code.</div><div class=""><br \
class=""></div><div class="">The _syslogng module is created by syslog-ng config \
parse: when syslog-ng processes the python keyword with the user code. So the \
_syslogng import problem should be resolved automatically with the above.<br \
class=""></div><div class=""><br class=""></div></div><div \
class="">Br,</div></div><div class=""> Antal<br class=""></div></div><br \
class=""><div class="gmail_quote"><div dir="ltr" class="">On Sat, Dec 30, 2017 at \
7:09 AM Ronald Fenner <<a href="mailto:rfenner@gamecircus.com" \
class="">rfenner@gamecircus.com</a>> wrote:<br class=""></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Here's the config I've redacted the Kafka servers. I've tried \
adding a @module "mod-python" but it doesn't help.<br class=""> <br class="">
#############################################################################<br \
class=""> # Default syslog-ng.conf file which collects all local logs into a<br \
class=""> # single file called /var/log/messages.<br class="">
#<br class="">
<br class="">
@version: 3.11<br class="">
@module "mod-java"<br class="">
@include "scl.conf"<br class="">
<br class="">
source s_internal {internal();};<br class="">
<br class="">
source s_rtl_stream {<br class="">
unix-stream("/var/log/rtl-stream.sock" flags(no-parse));<br class="">
};<br class="">
<br class="">
source s_php_fpm {<br class="">
file("/var/log/php-fpm.www.log", flags(no-parse));<br class="">
};<br class="">
<br class="">
destination d_kafka_unstructured {<br class="">
kafka (<br class="">
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br \
class=""> kafka-bootstrap-servers("******")<br class="">
topic("syslog-ng-{{DEPLOYMENT}}")<br class="">
);<br class="">
};<br class="">
<br class="">
destination d_kafka_structured {<br class="">
kafka (<br class="">
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br \
class=""> kafka-bootstrap-servers("*****")<br class="">
topic("${topic}.{{DEPLOYMENT}}")<br class="">
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")<br class="">
);<br class="">
};<br class="">
<br class="">
destination d_syslog_ng {<br class="">
file("/var/log/syslog-ng");<br class="">
};<br class="">
<br class="">
destination d_test_log {<br class="">
file("/var/log/test.log");<br class="">
};<br class="">
<br class="">
parser p_json { json-parser(); };<br class="">
<br class="">
parser p_apache { apache-accesslog-parser(prefix("")); };<br class="">
<br class="">
parser p_php_fpm { python(class("PhpFpmParser")); };<br class="">
<br class="">
rewrite r_add_access_topic {<br class="">
set("access.log", value("topic"));<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_internal);<br class="">
destination(d_syslog_ng);<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_rtl_stream);<br class="">
parser(p_json);<br class="">
destination(d_kafka_structured);<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_php_fpm);<br class="">
parser(p_php_fpm);<br class="">
destination(d_test_log);<br class="">
};<br class="">
<br class="">
<br class="">
Here's the actual python parser:<br class="">
python {<br class="">
class PhpFpmParser(object):<br class="">
def parse(self, log_msg):<br class="">
msg = log_msg['MESSAGE']<br class="">
str_pos = msg.find('] ')<br class="">
if str_pos == -1:<br class="">
return True<br class="">
log_date = msg[1:str_pos]<br class="">
msg = msg[str_pos+2:]<br class="">
str_pos = msg.find(':')<br class="">
if str_pos == -1:<br class="">
return True<br class="">
level = msg[:str_pos]<br class="">
if "Parse" in level:<br class="">
level = "parse"<br class="">
elif "Compile" in level:<br class="">
level = 'compile'<br class="">
elif "Fatal" in level:<br class="">
level = 'fatal'<br class="">
elif "Core" in level:<br class="">
level = 'core'<br class="">
elif "Notice" in level:<br class="">
level = 'notice'<br class="">
elif "Warning" in level:<br class="">
level = 'warning'<br class="">
msg = msg[str_pos+2:].strip()<br class="">
log_msg['err_msg'] = msg<br class="">
log_msg['log_level'] = level<br class="">
log_msg['timestamp_utc'] = log_date<br class="">
return True<br class="">
};<br class="">
<br class="">
It's stored the the etc/conf.d directory within the syslog path..<br class="">
<br class="">
Ronald Fenner<br class="">
Programmer<br class="">
Game Circus LLC.<br class="">
<br class="">
<a href="mailto:rfenner@gamecircus.com" target="_blank" \
class="">rfenner@gamecircus.com</a><br class=""> <br class="">
> On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <<a \
href="mailto:balazs.scheidler@balabit.com" target="_blank" \
class="">balazs.scheidler@balabit.com</a>> wrote:<br class=""> ><br class="">
> The _syslogng module is automatically created from the top level python block in \
syslog-ng and behaves similarly to the python __main__ module.<br class=""> ><br \
class=""> > Do you explicitly import that module using the imports() option?<br \
class=""> ><br class="">
> Can you please post your config?<br class="">
><br class="">
> On Dec 30, 2017 00:27, "Ronald Fenner" <<a \
href="mailto:rfenner@gamecircus.com" target="_blank" \
class="">rfenner@gamecircus.com</a>> wrote:<br class=""> > When I try to load \
my config with a python parser in it I'm getting this error message:<br class=""> \
> Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error \
loading Python module; module='_syslogng', exception='exceptions.ImportError: No \
module named _syslogng'<br class=""> > [2017-12-29T23:00:05.814066] Error looking \
Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'<br \
class=""> > [2017-12-29T23:00:05.814116] Error initializing message pipeline; \
plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'<br class=""> \
><br class=""> > I build syslog-ng from source with the python options. Here is \
the -V output<br class=""> > syslog-ng 3 (3.11.1)<br class="">
> Installer-Version: 3.11.1<br class="">
> Revision:<br class="">
> Compile-Date: Dec 29 2017 21:24:13<br class="">
> Module-Directory: /opt/syslog-ng/lib/syslog-ng<br class="">
> Module-Path: /opt/syslog-ng/lib/syslog-ng<br class="">
> Available-Modules: \
snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python \
,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,syste \
m-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat<br \
class=""> > Enable-Debug: off<br class="">
> Enable-GProf: off<br class="">
> Enable-Memtrace: off<br class="">
> Enable-IPv6: on<br class="">
> Enable-Spoof-Source: off<br class="">
> Enable-TCP-Wrapper: off<br class="">
> Enable-Linux-Caps: off<br class="">
> Enable-Systemd: off<br class="">
><br class="">
> Not sure how to fix this as from what I can tell this module is supposed to be \
compiled in and automatically imported.<br class=""> ><br class="">
><br class="">
> Ronald Fenner<br class="">
> Programmer<br class="">
> Game Circus LLC.<br class="">
><br class="">
> <a href="mailto:rfenner@gamecircus.com" target="_blank" \
class="">rfenner@gamecircus.com</a><br class=""> ><br class="">
><br class="">
> ______________________________________________________________________________<br \
class=""> > Member info: <a \
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" \
target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br \
class=""> > Documentation: <a \
href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank" \
class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br \
class=""> > FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" \
rel="noreferrer" target="_blank" \
class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""> ><br class="">
><br class="">
> ______________________________________________________________________________<br \
class=""> > Member info: <a \
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" \
target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br \
class=""> > Documentation: <a \
href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank" \
class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br \
class=""> > FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" \
rel="noreferrer" target="_blank" \
class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""> ><br class="">
<br class="">
______________________________________________________________________________<br \
class=""> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank" \
class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br \
class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank" \
class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br \
class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""> \
<br class=""> </blockquote></div>
______________________________________________________________________________<br \
class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br \
class="">Documentation: <a \
href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br \
class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" \
class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br \
class=""></div></blockquote></div><br class=""></div></body></html>
[Attachment #6 (text/plain)]
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic