'Re: [syslog-ng] Rewrite regex not working entirely'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       syslog-ng
Subject:    Re: [syslog-ng] Rewrite regex not working entirely
From:       Nagy, Gábor <gabor.nagy () oneidentity ! com>
Date:       2019-01-17 8:20:25
Message-ID: CAETAYnAtPEpqdBPLAJemgFhngdSoEc0S-F929Pt5K_rCe8Rk2w () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

Hi Max,

We have some guidelines for regexes, how to optimize them, syntax and
others:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.19/administration-guide/64#TOPIC-1094710
 https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.19/administration-guide/65

Regards,
Gabor

On Wed, Jan 16, 2019 at 5:33 PM N. Max Pierson <nmaxpierson@gmail.com>
wrote:

> Hi Atilla,
> 
> I switched the double quotes to single quotes and that fixed the issue. I
> do not believe the docs stated to use single quotes for full regular
> expressions to work which is why I used double quotes but in either case
> this resolved the issue.
> 
> Thanks for the feedback!
> 
> Regards,
> Max
> 
> On Wed, Jan 16, 2019 at 5:41 AM Szakacs, Attila <
> attila.szakacs@balabit.com> wrote:
> 
> > Hi Max,
> > 
> > I tried "\w\d" , etc... in "pcre" type subst rewrite rule on 3.19.
> > My config:
> > 
> > @version: 3.19
> > @include "scl.conf"
> > 
> > source s_udp5001 {
> > udp(
> > port(5001)
> > keep-hostname(yes)
> > flags(no-parse)
> > );
> > };
> > 
> > destination d_test {
> > file(
> > "/tmp/test.log"
> > );
> > };
> > 
> > rewrite r_chars {
> > subst(
> > "^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ",
> > "",
> > value("$MESSAGE"),
> > type("pcre"),
> > flags("ignore-case")
> > );
> > };
> > 
> > rewrite r_pcre {
> > subst(
> > '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s',
> > "",
> > value("$MESSAGE"),
> > type("pcre"),
> > flags("ignore-case")
> > );
> > };
> > 
> > log {
> > source(s_udp5001);
> > #rewrite(r_chars);
> > rewrite(r_pcre);
> > destination(d_test);
> > };
> > 
> > I think you need to make sure, that the regular expression is set between
> > single quotes (e.g.: '^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s')
> > 
> > Best regards,
> > Attila
> > 
> > On Tue, Jan 15, 2019 at 11:12 PM N. Max Pierson <nmaxpierson@gmail.com>
> > wrote:
> > 
> > > Hi Evan,
> > > 
> > > I have tried both pcre and posix and neither seem to work.
> > > 
> > > On Tue, Jan 15, 2019 at 4:08 PM Evan Rempel <erempel@uvic.ca> wrote:
> > > 
> > > > You have defined your regular expresion as "posix" which does not have
> > > > the \d \s etc.
> > > > If you change the type to "pcre" then it should work for you.
> > > > 
> > > > 
> > > > On 1/15/19 2:01 PM, N. Max Pierson wrote:
> > > > 
> > > > Hi List,
> > > > 
> > > > I am using version 3.5 and it seems as though regex (posix or pcre)
> > > > doesn't work completely. Take the example string below (which is the
> > > > message part of the syslog).
> > > > 
> > > > Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping
> > > > message: Received NTP control mode packet. Drop count:147972  - ntpd[15029]
> > > > 
> > > > I am trying to match the date at the beginning of the message and
> > > > remove it. When I use \w, \s, \d, etc, they do not match anything. If I
> > > > match on a character classes it works fine (ex [a-z]+ or [0-9]+).
> > > > 
> > > > Here is my statement for the rewrite rule.
> > > > 
> > > > rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ",
> > > > "", value("MESSAGE"), type("posix"), flags("ignore-case"),
> > > > condition(filter(f_nexus))); };
> > > > 
> > > > The above seems to get me what I want but are the character matches not
> > > > supposed to work in syslog-ng version 3.5??
> > > > 
> > > > Regards,
> > > > Max
> > > > 
> > > > 
> > > > 
> > > > ______________________________________________________________________________
> > > >  Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > > Documentation:
> > > > http://www.balabit.com/support/documentation/?product=syslog-ng
> > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > > > 
> > > > 
> > > ______________________________________________________________________________
> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Documentation:
> > > http://www.balabit.com/support/documentation/?product=syslog-ng
> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > > 
> > > 
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > 
> > 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
> 

[Attachment #5 (text/html)]

<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi \
Max,<div><br></div><div>We have some guidelines for regexes, how to optimize them, \
syntax and others:</div><div><a \
href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/ \
3.19/administration-guide/64#TOPIC-1094710">https://www.syslog-ng.com/technical-docume \
nts/doc/syslog-ng-open-source-edition/3.19/administration-guide/64#TOPIC-1094710</a><br></div><div><a \
href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/ \
3.19/administration-guide/65">https://www.syslog-ng.com/technical-documents/doc/syslog \
-ng-open-source-edition/3.19/administration-guide/65</a><br></div><div><br></div><div>Regards,</div><div>Gabor</div></div></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Wed, Jan 16, 2019 at 5:33 PM N. Max Pierson \
&lt;<a href="mailto:nmaxpierson@gmail.com">nmaxpierson@gmail.com</a>&gt; \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi \
Atilla,<div><br></div><div>I switched the double quotes to single quotes and that \
fixed the issue. I do not believe the docs stated to use single quotes for full \
regular expressions to work which is why I used double quotes but in either case this \
resolved the issue.  </div><div><br></div><div>Thanks for the \
feedback!</div><div><br></div><div>Regards,</div><div>Max</div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail-m_-1777389148130162255gmail_attr">On \
Wed, Jan 16, 2019 at 5:41 AM Szakacs, Attila &lt;<a \
href="mailto:attila.szakacs@balabit.com" \
target="_blank">attila.szakacs@balabit.com</a>&gt; wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi \
Max,<div><br></div><div>I tried &quot;\w\d&quot; , etc... in &quot;pcre&quot; type \
subst rewrite rule on 3.19.</div><div>My config:</div><div><br></div><div><div><font \
face="monospace, monospace">@version: 3.19</font></div><div><font face="monospace, \
monospace">@include &quot;scl.conf&quot;</font></div><div><font face="monospace, \
monospace"><br></font></div><div><font face="monospace, monospace">source s_udp5001 \
{</font></div><div><font face="monospace, monospace">   udp(</font></div><div><font \
face="monospace, monospace">      port(5001)</font></div><div><font face="monospace, \
monospace">      keep-hostname(yes)</font></div><div><font face="monospace, \
monospace">      flags(no-parse)</font></div><div><font face="monospace, monospace">  \
);</font></div><div><font face="monospace, monospace">};</font></div><div><font \
face="monospace, monospace"><br></font></div><div><font face="monospace, \
monospace">destination d_test {</font></div><div><font face="monospace, monospace">   \
file(</font></div><div><font face="monospace, monospace">      \
&quot;/tmp/test.log&quot;</font></div><div><font face="monospace, monospace">   \
);</font></div><div><font face="monospace, monospace">};</font></div><div><font \
face="monospace, monospace"><br></font></div><div><font face="monospace, \
monospace">rewrite r_chars {</font></div><div><font face="monospace, monospace">   \
subst(</font></div><div><font face="monospace, monospace">      &quot;^[a-z]+ [0-9]+ \
[0-9]+:[0-9]+:[0-9]+ [a-z]+: &quot;,</font></div><div><font face="monospace, \
monospace">      &quot;&quot;,</font></div><div><font face="monospace, monospace">    \
value(&quot;$MESSAGE&quot;),</font></div><div><font face="monospace, monospace">      \
type(&quot;pcre&quot;),</font></div><div><font face="monospace, monospace">      \
flags(&quot;ignore-case&quot;)</font></div><div><font face="monospace, monospace">   \
);</font></div><div><font face="monospace, monospace">};</font></div><div><font \
face="monospace, monospace"><br></font></div><div><font face="monospace, monospace" \
color="#000000">rewrite r_pcre {</font></div><div><font face="monospace, monospace" \
color="#000000">   subst(</font></div><div><font face="monospace, monospace" \
color="#000000">      </font><font face="monospace, monospace" \
color="#ff0000">&#39;^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s&#39;</font><font face="monospace, \
monospace" color="#000000">,</font></div><div><font face="monospace, monospace" \
color="#000000">      &quot;&quot;,</font></div><div><font face="monospace, \
monospace" color="#000000">      value(&quot;$MESSAGE&quot;),</font></div><div><font \
face="monospace, monospace" color="#000000">      \
type(&quot;pcre&quot;),</font></div><div><font face="monospace, monospace" \
color="#000000">      flags(&quot;ignore-case&quot;)</font></div><div><font \
face="monospace, monospace" color="#000000">   );</font></div><div><font \
face="monospace, monospace" color="#000000">};</font></div><div><font \
face="monospace, monospace"><br></font></div><div><font face="monospace, \
monospace">log {</font></div><div><font face="monospace, monospace">   \
source(s_udp5001);</font></div><div><font face="monospace, monospace">   \
#rewrite(r_chars);</font></div><div><font face="monospace, monospace">   \
rewrite(r_pcre);</font></div><div><font face="monospace, monospace">   \
destination(d_test);</font></div><div><font face="monospace, \
monospace">};</font></div></div><div><br></div><div>I think you need to make sure, \
that the regular expression is set between single quotes (e.g.:  <span \
style="font-family:monospace,monospace">&#39;^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s&#39;</span>)</div><div><br></div><div>Best \
regards,</div><div>Attila</div></div></div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail_attr">On \
Tue, Jan 15, 2019 at 11:12 PM N. Max Pierson &lt;<a \
href="mailto:nmaxpierson@gmail.com" target="_blank">nmaxpierson@gmail.com</a>&gt; \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi \
Evan,<div><br></div><div>I have tried both pcre and posix and neither seem to \
work.</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 15, 2019 at \
4:08 PM Evan Rempel &lt;<a href="mailto:erempel@uvic.ca" \
target="_blank">erempel@uvic.ca</a>&gt; wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">  

  <div bgcolor="#FFFFFF">
    <div class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225moz-cite-prefix">You \
have defined your regular expresion  as &quot;posix&quot; which does not have the \d \
\s etc.</div>  <div class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225moz-cite-prefix">If \
you change the type to &quot;pcre&quot; then  it should work for you.</div>
    <div class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225moz-cite-prefix"><br>
  </div>
    <div class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225moz-cite-prefix"><br>
  </div>
    <div class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225moz-cite-prefix">On \
1/15/19 2:01 PM, N. Max Pierson  wrote:<br>
    </div>
    <blockquote type="cite">

      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">Hi List,
            <div><br>
            </div>
            <div>I am using version 3.5 and it seems as though regex
              (posix or pcre) doesn&#39;t work completely. Take the example
              string below (which is the message part of the syslog).  </div>
            <div><br>
            </div>
            <div>Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive
              dropping message: Received NTP control mode packet. Drop
              count:147972   - ntpd[15029]<br>
            </div>
            <div><br>
            </div>
            <div>I am trying to match the date at the beginning of the
              message and remove it. When I use \w, \s, \d, etc, they do
              not match anything. If I match on a character classes it
              works fine (ex [a-z]+ or [0-9]+).</div>
            <div><br>
            </div>
            <div>Here is my statement for the rewrite rule.</div>
            <div><br>
            </div>
            <div>rewrite r_nexus{ subst(&quot;^[a-z]+ [0-9]+
              [0-9]+:[0-9]+:[0-9]+ [a-z]+: &quot;, &quot;&quot;, \
value(&quot;MESSAGE&quot;),  type(&quot;posix&quot;), flags(&quot;ignore-case&quot;),
              condition(filter(f_nexus))); };<br>
            </div>
            <div><br>
            </div>
            <div>The above seems to get me what I want but are the
              character matches not supposed to work in syslog-ng
              version 3.5??</div>
            <div><br>
            </div>
            <div>Regards,</div>
            <div>Max</div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="gmail-m_-1777389148130162255gmail-m_-235784799549823427gmail-m_3687774629939134283gmail-m_5419114083870337225mimeAttachmentHeader"></fieldset>
  </blockquote>
    <br>
  </div>

______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" \
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>

Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" \
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>

FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" \
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br> <br>
</blockquote></div></div>

[Attachment #6 (text/plain)]

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[prev in list] [next in list] [prev in thread] [next in thread] 