[prev in list] [next in list] [prev in thread] [next in thread]
List: fail2ban-users
Subject: Re: [Fail2ban-users] fail2ban refuses to match even most basic acceptable regex
From: Alan Liddell <alan.c.liddell () gmail ! com>
Date: 2016-07-08 17:29:05
Message-ID: 577FE2E1.3000205 () gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/signed)]
[Attachment #4 (multipart/mixed)]
[Attachment #6 (multipart/alternative)]
...or not. I didn't realize the wiki was closed to contributors. If any
of you watching this thread is a contributor, please make this change.
On 07/08/2016 06:25 PM, Alan Liddell wrote:
>
> Thanks Zurd for taking the time to answer this. Turns out it was a
> really simple mistake. After reading this closed bug report
> <https://github.com/fail2ban/fail2ban/issues/1453> (specifically the
> comment here
> <https://github.com/fail2ban/fail2ban/issues/1453#issuecomment-222469723>)
> I realized that whitespace after the beginning-of-line anchor (^) was
> actually necessary.
>
> The entry in the wiki
> <http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters> has this
> to say, though:
>
>> If the failregex is anchored with a leading |^|, then the anchor
>> refers to the start of the remainder of the line, /after/ the
>> timestamp and intervening whitespace.
>
> So if I'm reading this correctly, this is actually inaccurate, since
> according to the bug report and that specific comment the whitespace
> after the timestamp needs to be explicitly included. Unless someone
> gets back to me telling me why I'm wrong, I'm going to change the
> entry in the wiki. Thanks for your time everyone.
>> This was the logical end of a sequence of more general regexes to make
>> sure I wasn't losing my mind. Here's the one I started out with:
>>
>> ^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No
>> such file or directory\), client\: <HOST>\, server\: \S*\, request\:
>> \"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$
>>
>> When you expand <HOST> out to |(?:::f{4,6}:)?(?P<host>\S+) |as the wiki
>> suggests (http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters),
>> you get
>>
>> ^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No
>> such file or directory\), client\: (?:::f{4,6}:)?(?P<host>\S+)\,
>> server\: \S*\, request\: \"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$
>>
>> (I tried this one as well), which debuggex.com says should match.
>> Compiling this last regex in both Python 2 and Python 3, I get a match
>> when I run it against this line. It's not the over-general regex. The
>> timestamp, from fail2ban-regex output in my first, should also match. So
>> what gives?
>>
>> > Date: Wed, 6 Jul 2016 21:43:33 -0400
>> > From: Zurd <zurd33@...>
>> > Subject: Re: [Fail2ban-users] fail2ban refuses to match even most
>> > basic acceptable regex
>> > To: fail2ban-users@...
>> > Message-ID:
>> > <CAFPUJG7_4_LjJCH0HWZJEbKngjaOU_hPizLuuVo_K-LSY2X9wQ@...>
>> > Content-Type: text/plain; charset="utf-8"
>> >
>> > I am by no means expert but I can see that having a filter so simple and so
>> > small doesn't work. I don't think it's really a bug though as fail2ban
>> > could be compromised about 1 or 2 years ago and the regex had to be remade.
>> > I think having as little as possible of .* helps. Maybe an expert can
>> > comment on this? Isn't there a wiki page on how to create regex on the
>> > website of fail2ban? I can't seem to find anything.
>> >
>> > As for your regex, I can make it work like so below by just adding a few
>> > characters/words:
>> >
>> > $line=
>> > '2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> > "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> > failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> > request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657
>> > HTTP/1.1", host: "www.appleipadwallpapers.com"'
>> >
>> > $regex=
>> > '^.*, client: <HOST>, server: _, request:.*$'
>> >
>> > Or just copy and paste this:
>> > fail2ban-regex '2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> > "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> > failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> > request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657
>> > HTTP/1.1", host: "www.appleipadwallpapers.com"' '^.*, client: <HOST>,
>> > server: _, request:.*$'
>> >
>> >
>> >
>> >
>> > On Wed, Jul 6, 2016 at 4:15 PM, Alan Liddell <alan.c.liddell@...>
>> > wrote:
>> >
>> >> Hi all,
>> >>
>> >> I checked the GitHub and asked on IRC (nobody around at the time) and
>> >> couldn't find anything like this. I'm running fail2ban 0.9.3 on Fedora
>> >> 24, Python 2.7.11/3.5.1, trying to check Nginx error logs for bots.
>> >> Here's the line:
>> >>
>> >> $ line='2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> >>
>> >> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> >> failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> >> request: "GET
>> >> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1",
>> >> host: "www.appleipadwallpapers.com"'
>> >>
>> >> Here's the regex:
>> >>
>> >> $ regex='^.*<HOST>.*$'
>> >>
>> >> This should be the most permissive possible regex on fail2ban, right?
>> >> But here's the output of fail2ban-regex:
>> >>
>> >> $ fail2ban-regex "$line" "$regex"
>> >>
>> >> Running tests
>> >> =============
>> >>
>> >> Use failregex line : ^.*<HOST>.*$
>> >> Use single line : 2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> >> "/u...
>> >>
>> >>
>> >> Results
>> >> =======
>> >>
>> >> Failregex: 0 total
>> >>
>> >> Ignoreregex: 0 total
>> >>
>> >> Date template hits:
>> >> |- [# of hits] date format
>> >> | [1] Year(?P<_sep>[-/.])Month(?P=_sep)Day
>> >> 24hour:Minute:Second(?:,Microseconds)?
>> >> `-
>> >>
>> >> Lines: 1 lines, 0 ignored, 0 matched, 1 missed [processed in 0.02 sec]
>> >> |- Missed line(s):
>> >> | 2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> >>
>> >> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> >> failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> >> request: "GET
>> >> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1",
>> >> host: "www.appleipadwallpapers.com"
>> >> `-
>> >>
>> >>
>> >> (I'm new to fail2ban and I was worried my timestamp might have been
>> >> nonstandard, but does the bit under "Date template hits" mean that I'm
>> >> in the clear there?) By the way, fail2ban-testcases fails a few tests
>> >> related to this:
>> >>
>> >> Regex for filter 'nginx-botsearch' has no samples: 2: '^\\[error\\]
>> >> \\d+#\\d+: \\*\\d+ \\S+\\(\\) \\"\\S+\\" (failed|is not found) \\(2\\:
>> >> No such file or directory\\), client\\:
>> >> (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w)\\, server\\: \\S*\\, request:
>> >> \\"(GET|POST|HEAD) \\/\\S+ \\S+\\"\\, .*?$'
>> >>
>> >> Regex for filter 'nginx-http-auth' has no samples: 1: '^ \\[error\\]
>> >> \\d+#\\d+: \\*\\d+ no user/password was provided for basic
>> >> authentication, client: (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w), server:
>> >> \\S+, request: "\\S+ \\S+ HTTP/\\d+\\.\\d+", host: "\\S+"\\s*$'
>> >>
>> >> and so forth. Don't know if this specifically is relevant, but thought
>> >> I'd mention it. Thanks all.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> >> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> >> present their vision of the future. This family event has something for
>> >> everyone, including kids. Get more information and register today.
>> >> http://sdm.link/attshape
>> >> _______________________________________________
>> >> Fail2ban-users mailing list
>> >> Fail2ban-users@...
>> >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>> >>
>> >>
>
[Attachment #9 (text/html)]
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
...or not. I didn't realize the wiki was closed to contributors. If
any of you watching this thread is a contributor, please make this
change.<br>
<br>
<div class="moz-cite-prefix">On 07/08/2016 06:25 PM, Alan Liddell
wrote:<br>
</div>
<blockquote cite="mid:577FE1FB.2010206@gmail.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<p>Thanks Zurd for taking the time to answer this. Turns out it
was a really simple mistake. After reading <a
moz-do-not-send="true"
href="https://github.com/fail2ban/fail2ban/issues/1453">this
closed bug report</a> (specifically the comment <a
moz-do-not-send="true"
href="https://github.com/fail2ban/fail2ban/issues/1453#issuecomment-222469723">here</a>)
I realized that whitespace after the beginning-of-line anchor
(^) was actually necessary.<br>
</p>
<p>The entry in the <a moz-do-not-send="true"
href="http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters">wiki</a>
has this to say, though:<br>
</p>
<blockquote type="cite"> If the failregex is anchored with a
leading <code>^</code>, then the anchor refers to the start of
the remainder of the line, <i>after</i> the timestamp and
intervening whitespace. </blockquote>
<br>
So if I'm reading this correctly, this is actually inaccurate,
since according to the bug report and that specific comment the
whitespace after the timestamp needs to be explicitly included.
Unless someone gets back to me telling me why I'm wrong, I'm going
to change the entry in the wiki. Thanks for your time everyone.<br>
<pre><blockquote type="cite"><pre>This was the logical end of a sequence of \
more general regexes to make sure I wasn't losing my mind. Here's the one I started \
out with:
^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No
such file or directory\), client\: <HOST>\, server\: \S*\, request\:
\"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$
When you expand <HOST> out to |(?:::f{4,6}:)?(?P<host>\S+) |as the wiki
suggests (<a moz-do-not-send="true" \
href="http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters" \
rel="nofollow">http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters</a>), you \
get
^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No
such file or directory\), client\: (?:::f{4,6}:)?(?P<host>\S+)\,
server\: \S*\, request\: \"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$
(I tried this one as well), which debuggex.com says should match.
Compiling this last regex in both Python 2 and Python 3, I get a match
when I run it against this line. It's not the over-general regex. The
timestamp, from fail2ban-regex output in my first, should also match. So
what gives?
> Date: Wed, 6 Jul 2016 21:43:33 -0400
> From: Zurd <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" \
href="mailto:zurd33@..."><zurd33@...></a> > Subject: Re: [Fail2ban-users] \
fail2ban refuses to match even most > basic acceptable regex
> To: fail2ban-users@...
> Message-ID:
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" \
href="mailto:CAFPUJG7_4_LjJCH0HWZJEbKngjaOU_hPizLuuVo_K-LSY2X9wQ@..."><CAFPUJG7_4_LjJCH0HWZJEbKngjaOU_hPizLuuVo_K-LSY2X9wQ@...></a>
> Content-Type: text/plain; charset="utf-8"
>
> I am by no means expert but I can see that having a filter so simple and so
> small doesn't work. I don't think it's really a bug though as fail2ban
> could be compromised about 1 or 2 years ago and the regex had to be remade.
> I think having as little as possible of .* helps. Maybe an expert can
> comment on this? Isn't there a wiki page on how to create regex on the
> website of fail2ban? I can't seem to find anything.
>
> As for your regex, I can make it work like so below by just adding a few
> characters/words:
>
> $line=
> '2016/07/05 23:10:26 [error] 2359#0: *21 open()
> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
> failed (2: No such file or directory), client: 198.143.46.17, server: _,
> request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657
> HTTP/1.1", host: "<a moz-do-not-send="true" class="moz-txt-link-abbreviated" \
href="http://www.appleipadwallpapers.com">www.appleipadwallpapers.com</a>"' >
> $regex=
> '^.*, client: <HOST>, server: _, request:.*$'
>
> Or just copy and paste this:
> fail2ban-regex '2016/07/05 23:10:26 [error] 2359#0: *21 open()
> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
> failed (2: No such file or directory), client: 198.143.46.17, server: _,
> request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657
> HTTP/1.1", host: "<a moz-do-not-send="true" class="moz-txt-link-abbreviated" \
href="http://www.appleipadwallpapers.com">www.appleipadwallpapers.com</a>"' '^.*, \
client: <HOST>, > server: _, request:.*$'
>
>
>
>
> On Wed, Jul 6, 2016 at 4:15 PM, Alan Liddell <a moz-do-not-send="true" \
class="moz-txt-link-rfc2396E" \
href="mailto:alan.c.liddell@..."><alan.c.liddell@...></a> > wrote:
>
>> Hi all,
>>
>> I checked the GitHub and asked on IRC (nobody around at the time) and
>> couldn't find anything like this. I'm running fail2ban 0.9.3 on Fedora
>> 24, Python 2.7.11/3.5.1, trying to check Nginx error logs for bots.
>> Here's the line:
>>
>> $ line='2016/07/05 23:10:26 [error] 2359#0: *21 open()
>>
>> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> request: "GET
>> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1",
>> host: "<a moz-do-not-send="true" class="moz-txt-link-abbreviated" \
href="http://www.appleipadwallpapers.com">www.appleipadwallpapers.com</a>"' >>
>> Here's the regex:
>>
>> $ regex='^.*<HOST>.*$'
>>
>> This should be the most permissive possible regex on fail2ban, right?
>> But here's the output of fail2ban-regex:
>>
>> $ fail2ban-regex "$line" "$regex"
>>
>> Running tests
>> =============
>>
>> Use failregex line : ^.*<HOST>.*$
>> Use single line : 2016/07/05 23:10:26 [error] 2359#0: *21 open()
>> "/u...
>>
>>
>> Results
>> =======
>>
>> Failregex: 0 total
>>
>> Ignoreregex: 0 total
>>
>> Date template hits:
>> |- [# of hits] date format
>> | [1] Year(?P<_sep>[-/.])Month(?P=_sep)Day
>> 24hour:Minute:Second(?:,Microseconds)?
>> `-
>>
>> Lines: 1 lines, 0 ignored, 0 matched, 1 missed [processed in 0.02 sec]
>> |- Missed line(s):
>> | 2016/07/05 23:10:26 [error] 2359#0: *21 open()
>>
>> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657"
>> failed (2: No such file or directory), client: 198.143.46.17, server: _,
>> request: "GET
>> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1",
>> host: "<a moz-do-not-send="true" class="moz-txt-link-abbreviated" \
href="http://www.appleipadwallpapers.com">www.appleipadwallpapers.com</a>" >> \
`- >>
>>
>> (I'm new to fail2ban and I was worried my timestamp might have been
>> nonstandard, but does the bit under "Date template hits" mean that I'm
>> in the clear there?) By the way, fail2ban-testcases fails a few tests
>> related to this:
>>
>> Regex for filter 'nginx-botsearch' has no samples: 2: '^\\[error\\]
>> \\d+#\\d+: \\*\\d+ \\S+\\(\\) \\"\\S+\\" (failed|is not found) \\(2\\:
>> No such file or directory\\), client\\:
>> (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w)\\, server\\: \\S*\\, request:
>> \\"(GET|POST|HEAD) \\/\\S+ \\S+\\"\\, .*?$'
>>
>> Regex for filter 'nginx-http-auth' has no samples: 1: '^ \\[error\\]
>> \\d+#\\d+: \\*\\d+ no user/password was provided for basic
>> authentication, client: (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w), \
server: >> \\S+, request: "\\S+ \\S+ HTTP/\\d+\\.\\d+", host: "\\S+"\\s*$'
>>
>> and so forth. Don't know if this specifically is relevant, but thought
>> I'd mention it. Thanks all.
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in \
San >> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> <a moz-do-not-send="true" href="http://sdm.link/attshape" \
rel="nofollow">http://sdm.link/attshape</a> >> \
_______________________________________________ >> Fail2ban-users mailing list
>> Fail2ban-users@...
>> <a moz-do-not-send="true" \
href="https://lists.sourceforge.net/lists/listinfo/fail2ban-users" \
rel="nofollow">https://lists.sourceforge.net/lists/listinfo/fail2ban-users</a> \
>> >></pre></blockquote>
</pre>
</blockquote>
<br>
</body>
</html>
["signature.asc" (application/pgp-signature)]
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic