[prev in list] [next in list] [prev in thread] [next in thread]
List: clamav-users
Subject: Re: [clamav-users] Scan for dummy file with /dev/zero takes longer
From: Andrew Williams <awillia2 () sourcefire ! com>
Date: 2019-07-16 16:26:52
Message-ID: CAN0foEXm+=inY7FqxXe_=WQQvczwgr+eJdrWLCAisFt-kR3T2w () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Taizo,
The reason for the difference is that there are a lot of subsignatures used
in the published set of logical signatures that begin with some number of
zeroes (more so than the majority of random byte sequences), so the ClamAV
pattern matcher has to do a lot more work on the all-zeroes file
determining that none of the signatures fully match. Also, there are
likely some short all-zero subsigs that get used based on how certain
subsignature features are implemented, and these can also affect
performance on large files consisting mainly of zeroes.
Hope that helps! Thanks for asking about this - your observation is a good
reminder to us that a large all-zero file makes a good test case for
catching signatures that might have egregious performance impacts. :)
-Andrew
Andrew Williams
Malware Research Team
Cisco Talos
On Tue, Jul 9, 2019 at 11:07 PM Taizo ITO <taizo.ito@hennge.com> wrote:
> Hello,
>
> I'm trying to get some stats on how long a scan takes by different
> size, but I encountered an unexpected behavior when scanning a file
> generated in a specific way.
> A scan for a dummy file filled with /dev/zero takes much longer than
> with /dev/urandom. I think the processing time should be the same or
> less.
> I'd like to know how to avoid this problem because that may cause stuck
> service.
> I'm using ClamAV version 0.101.2/25504.
>
> You can reproduce the problem by doing the following.
>
> A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.
> ```
> $ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1
> $ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1
> $ ls -ltr dummy-dd-10MB-with-*
> -rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-zero.iso
> -rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-urandom.iso
>
> $ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-zero.iso: OK
> real 0m4.056s
> user 0m0.008s
> sys 0m0.004s
>
> $ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-urandom.iso: OK
> real 0m0.569s
> user 0m0.012s
> sys 0m0.000s
> ```
>
>
> A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.
> ```
> $ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10
> $ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10
> $ ls -ltr dummy-dd-250MB-with-*
> -rw-r--r-- 1 user user 250000000 Jul 9 03:44
> dummy-dd-250MB-with-urandom.iso
> -rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-zero.iso
>
> $ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-zero.iso: OK
> real 1m42.949s
> user 0m0.009s
> sys 0m0.003s
>
> $time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-urandom.iso: OK
> real 0m12.905s
> user 0m0.004s
> sys 0m0.007s
> ```
>
> Thanks.
>
> --
> Taizo Ito
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr">Taizo,<div><br></div><div>The reason for the difference \
is that there are a lot of subsignatures used in the published set of logical \
signatures that begin with some number of zeroes (more so than the majority of random \
byte sequences), so the ClamAV pattern matcher has to do a lot more work on the \
all-zeroes file determining that none of the signatures fully match. Also, there \
are likely some short all-zero subsigs that get used based on how certain \
subsignature features are implemented, and these can also affect performance on large \
files consisting mainly of zeroes.</div><div><br></div><div>Hope that helps! Thanks \
for asking about this - your observation is a good reminder to us that a large \
all-zero file makes a good test case for catching signatures that might have \
egregious performance impacts. \
:)</div><div><br></div><div>-Andrew</div><div><br></div><div>Andrew \
Williams</div><div>Malware Research Team</div><div>Cisco Talos</div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 9, 2019 at 11:07 PM \
Taizo ITO <<a href="mailto:taizo.ito@hennge.com">taizo.ito@hennge.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br> <br>
I'm trying to get some stats on how long a scan takes by different<br>
size, but I encountered an unexpected behavior when scanning a file<br>
generated in a specific way.<br>
A scan for a dummy file filled with /dev/zero takes much longer than<br>
with /dev/urandom. I think the processing time should be the same or<br>
less.<br>
I'd like to know how to avoid this problem because that may cause stuck \
service.<br> I'm using ClamAV version 0.101.2/25504.<br>
<br>
You can reproduce the problem by doing the following.<br>
<br>
A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.<br>
```<br>
$ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1<br>
$ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1<br>
$ ls -ltr dummy-dd-10MB-with-*<br>
-rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-zero.iso<br>
-rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-urandom.iso<br>
<br>
$ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U<br>
/var/run/clamd.scan/clamd.sock)<br>
dummy-dd-10MB-with-zero.iso: OK<br>
real 0m4.056s<br>
user 0m0.008s<br>
sys 0m0.004s<br>
<br>
$ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U<br>
/var/run/clamd.scan/clamd.sock)<br>
dummy-dd-10MB-with-urandom.iso: OK<br>
real 0m0.569s<br>
user 0m0.012s<br>
sys 0m0.000s<br>
```<br>
<br>
<br>
A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.<br>
```<br>
$ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10<br>
$ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10<br>
$ ls -ltr dummy-dd-250MB-with-*<br>
-rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-urandom.iso<br>
-rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-zero.iso<br>
<br>
$ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U<br>
/var/run/clamd.scan/clamd.sock)<br>
dummy-dd-250MB-with-zero.iso: OK<br>
real 1m42.949s<br>
user 0m0.009s<br>
sys 0m0.003s<br>
<br>
$time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U<br>
/var/run/clamd.scan/clamd.sock)<br>
dummy-dd-250MB-with-urandom.iso: OK<br>
real 0m12.905s<br>
user 0m0.004s<br>
sys 0m0.007s<br>
```<br>
<br>
Thanks.<br>
<br>
-- <br>
Taizo Ito<br>
<br>
_______________________________________________<br>
<br>
clamav-users mailing list<br>
<a href="mailto:clamav-users@lists.clamav.net" \
target="_blank">clamav-users@lists.clamav.net</a><br> <a \
href="https://lists.clamav.net/mailman/listinfo/clamav-users" rel="noreferrer" \
target="_blank">https://lists.clamav.net/mailman/listinfo/clamav-users</a><br> <br>
<br>
Help us build a comprehensive ClamAV guide:<br>
<a href="https://github.com/vrtadmin/clamav-faq" rel="noreferrer" \
target="_blank">https://github.com/vrtadmin/clamav-faq</a><br> <br>
<a href="http://www.clamav.net/contact.html#ml" rel="noreferrer" \
target="_blank">http://www.clamav.net/contact.html#ml</a><br> \
</blockquote></div></div>
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic