[prev in list] [next in list] [prev in thread] [next in thread]
List: bro
Subject: Re: [Zeek] Memory usage climbs and never recovers
From: "Pearson, Carl (cpearson () uidaho ! edu)" <cpearson () uidaho ! edu>
Date: 2020-07-09 16:26:27
Message-ID: CY4PR04MB108118E84E165C732A659D5BA9640 () CY4PR04MB1081 ! namprd04 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
[Attachment #4 (text/plain)]
Hi Joseph,
We had similar memory utilization when we first deployed Zeek, our usage graph was \
like yours. We ended up disabling the scan detection script in local.zeek and memory \
usage stabilized afterwards. If you are loading the scan script, might be worth \
disabling it and see if it makes a difference? Changing @load misc/scan to #@load \
misc/scan in /<zeek install dir>/share/zeek/site/local.zeek stops the script from \
loading when Zeek starts.
Carl Pearson | IT Security Analyst | University of Idaho
(208) 885-0957 | 875 Perimeter Drive MS 3155 | Moscow, ID 83844
From: zeek-bounces@zeek.org <zeek-bounces@zeek.org> On Behalf Of Joseph Fischetti
Sent: Thursday, July 9, 2020 08:20
To: Justin Azoff <justin@corelight.com>
Cc: zeek@zeek.org
Subject: Re: [Zeek] Memory usage climbs and never recovers
Negative, perhaps I should?
bro@bro-master-1:/opt/zeek/etc$ zeekctl jeprof.check
Warning: ZeekControl plugin uses legacy BroControl API. Use
'import ZeekControl.plugin' instead of 'import BroControl.plugin'
Error: unknown command 'jeprof.check'
Wouldn't others experience the same issues if the software just didn't behave as \
expected? Is it that we're undersized (and still eventually get to the point where \
the memory is needed)? Should I remove swap (so that it just starts dropping things \
rather than consuming swap?)
From: Justin Azoff <justin@corelight.com<mailto:justin@corelight.com>>
Sent: Thursday, July 9, 2020 11:08 AM
To: Joseph Fischetti \
<Joseph.Fischetti@marist.edu<mailto:Joseph.Fischetti@marist.edu>>
Cc: zeek@zeek.org<mailto:zeek@zeek.org>
Subject: Re: [Zeek] Memory usage climbs and never recovers
[EXTERNAL EMAIL]
Are you building zeek against jemalloc?
This plugin I put together for zeekctl makes it easy to enable jemalloc profiling and \
really understand the memory usage:
https://github.com/JustinAzoff/zeek-jemalloc-profiling<https://urldefense.com/v3/__htt \
ps:/github.com/JustinAzoff/zeek-jemalloc-profiling__;!!JYXjzlvb!ytAoH2FY2nNAau06oiM10kVFwFj6Y8_aBIpOMkBQMec7HDTEphWZUHihl0ZVjIal0w$>
Sometimes the memory just doesn't go down because malloc doesn't necessarily return \
freed memory to the OS. I think jemalloc will, but might not if swap is enabled on \
the host.
On Thu, Jul 9, 2020 at 9:51 AM Joseph Fischetti \
<Joseph.Fischetti@marist.edu<mailto:Joseph.Fischetti@marist.edu>> wrote: Hi All,
Rather than resurrect an old thread that I had I wanted to start a new one.
Our cluster has been mostly stable thanks to some suggestions (thanks Justin!).
That said, we're still getting out of hand memory consumption and eventual swap \
usage. The (2) workers are bare metal, fully populated with 24 x 16GB memory \
modules. Attached is the relevant parts of node.cfg [1] (note, we WERE pinning the \
CPU's but had a terrible time getting things to start up with them that way, so now \
we're not. Packet loss and CPU usage is well within the acceptable range) [2].
The memory usage isn't what's confusing so much as the fact that it just never comes \
back down.
Where should we look?
[cid:image001.png@01D655D3.03D2B7D0]
========================
[1]
-------------
[worker-1]
type=worker
host=HOST_A
lb_method=custom
lb_procs=10
#pin_cpus=1,2,3,4,5,6,7,8,9,10
interface=myricom::eth4
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3
#
[worker-2]
type=worker
host=HOST_B
lb_method=custom
lb_procs=10
#pin_cpus=1,2,3,4,5,6,7,8,9,10
interface=myricom::eth4
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3
[worker-3]
type=worker
host=HOST_A
lb_method=custom
lb_procs=10
#pin_cpus=18,19,20,21,22,23,24,25,26,27
interface=myricom::eth5
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3
#
[worker-4]
type=worker
host=HOST_B
lb_method=custom
lb_procs=10
#pin_cpus=18,19,20,21,22,23,24,25,26,27
interface=myricom::eth5
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3
#
-----------------
[2]
(uptime – since Jul 2)
worker-1 dropped=39637970 rx=12144046412 0.33%
worker-2 dropped=13113250 rx=6088981004 0.22%
worker-3 dropped=0 rx=830605595 0.00%
worker-4 dropped=0 rx=781722953 0.00%
Totals dropped=52751220 rx=19845355964 0.27%
_______________________________________________
Zeek mailing list
zeek@zeek.org<mailto:zeek@zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek<https://urldefense.com/v3/__htt \
p:/mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek__;!!JYXjzlvb!ytAoH2FY2nNAau06oiM10kVFwFj6Y8_aBIpOMkBQMec7HDTEphWZUHihl0aV0q5QAg$>
--
Justin
[Attachment #5 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Joseph,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We had similar memory utilization when we first deployed Zeek, \
our usage graph was like yours. We ended up disabling the scan detection script in \
local.zeek and memory usage stabilized afterwards. If you are loading the scan \
script, might be worth disabling it and see if it makes a difference? Changing @load \
misc/scan to #@load misc/scan in /<zeek install dir>/share/zeek/site/local.zeek \
stops the script from loading when Zeek starts. <o:p></o:p></p>
<p class="MsoNormal"><span style="color:#171717"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:black">Carl Pearson</span></b><span \
style="color:gray"> </span>|<span style="color:gray"> </span><b>IT Security \
Analyst</b><span style="color:gray"> </span><span style="color:black">|</span><span \
style="color:gray"> </span><b><span style="color:black">University of \
Idaho</span></b><span style="color:#3B3838"><o:p></o:p></span></p> <p \
class="MsoNormal"><span style="color:#3B3838">(208) 885-0957 </span><span \
style="color:black">|</span><span style="color:gray"> </span><span \
style="color:#3B3838">875 Perimeter Drive MS 3155</span><span style="color:gray"> \
</span><span style="color:black">|</span><span style="color:gray"> </span><span \
style="color:#3B3838">Moscow, ID 83844</span><span \
style="color:#3B3838"><o:p></o:p></span></p> <p \
class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> zeek-bounces@zeek.org <zeek-bounces@zeek.org> \
<b> On Behalf Of </b>Joseph Fischetti<br>
<b>Sent:</b> Thursday, July 9, 2020 08:20<br>
<b>To:</b> Justin Azoff <justin@corelight.com><br>
<b>Cc:</b> zeek@zeek.org<br>
<b>Subject:</b> Re: [Zeek] Memory usage climbs and never recovers<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Negative, perhaps I should?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">bro@bro-master-1:/opt/zeek/etc$ zeekctl \
jeprof.check<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Warning: ZeekControl plugin uses legacy BroControl API. \
Use<o:p></o:p></p> <p class="MsoNormal">'import ZeekControl.plugin' instead of \
'import BroControl.plugin'<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Error: unknown command 'jeprof.check'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Wouldn't others experience the same issues if the software just \
didn't behave as expected? Is it that we're undersized (and still eventually \
get to the point where the memory is needed)? Should I remove swap (so that it \
just starts dropping things rather than consuming swap?)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Justin Azoff <<a \
href="mailto:justin@corelight.com">justin@corelight.com</a>> <br>
<b>Sent:</b> Thursday, July 9, 2020 11:08 AM<br>
<b>To:</b> Joseph Fischetti <<a \
href="mailto:Joseph.Fischetti@marist.edu">Joseph.Fischetti@marist.edu</a>><br> \
<b>Cc:</b> <a href="mailto:zeek@zeek.org">zeek@zeek.org</a><br> <b>Subject:</b> Re: \
[Zeek] Memory usage climbs and never recovers<o:p></o:p></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="background:#FF6700"><b><span style="font-size:12.0pt;color:black">[EXTERNAL \
EMAIL]</span></b><o:p></o:p></p> <div>
<div>
<p class="MsoNormal">Are you building zeek against jemalloc? <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This plugin I put together for zeekctl makes it easy to \
enable jemalloc profiling and really understand the memory usage:<o:p></o:p></p> \
</div> <div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a \
href="https://urldefense.com/v3/__https:/github.com/JustinAzoff/zeek-jemalloc-profilin \
g__;!!JYXjzlvb!ytAoH2FY2nNAau06oiM10kVFwFj6Y8_aBIpOMkBQMec7HDTEphWZUHihl0ZVjIal0w$">https://github.com/JustinAzoff/zeek-jemalloc-profiling</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sometimes the memory just doesn't go down because malloc doesn't \
necessarily return freed memory to the OS. I think jemalloc will, but \
might not if swap is enabled on the host.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jul 9, 2020 at 9:51 AM Joseph Fischetti <<a \
href="mailto:Joseph.Fischetti@marist.edu">Joseph.Fischetti@marist.edu</a>> \
wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi \
All,<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Rather than resurrect an \
old thread that I had I wanted to start a new one.<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Our \
cluster has been mostly stable thanks to some suggestions (thanks \
Justin!).<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">That \
said, we're still getting out of hand memory consumption and eventual swap \
usage. The (2) workers are bare metal, fully populated with 24 x 16GB memory \
modules. Attached is the relevant parts of node.cfg [1] (note, we WERE pinning the \
CPU's but had a terrible time getting things to start up with them that way, so now \
we're not. Packet loss and CPU usage is well within the acceptable range) \
[2].<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The \
memory usage isn't what's confusing so much as the fact that it just never comes back \
down. <br>
<br>
Where should we look?<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img \
border="0" width="694" height="251" style="width:7.2291in;height:2.6145in" \
id="gmail-m_5455189717699372446Picture_x0020_1" \
src="cid:image001.png@01D655D3.03D2B7D0"><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">========================<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[1]<br>
-------------<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[worker-1]<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">type=worker<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">host=HOST_A<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_method=custom<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_procs=10<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#pin_cpus=1,2,3,4,5,6,7,8,9,10<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">interface=myricom::eth4<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[worker-2]<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">type=worker<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">host=HOST_B<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_method=custom<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_procs=10<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#pin_cpus=1,2,3,4,5,6,7,8,9,10<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">interface=myricom::eth4<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[worker-3]<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">type=worker<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">host=HOST_A<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_method=custom<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_procs=10<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#pin_cpus=18,19,20,21,22,23,24,25,26,27<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">interface=myricom::eth5<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[worker-4]<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">type=worker<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">host=HOST_B<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_method=custom<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">lb_procs=10<o:p></o:p></p> \
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#pin_cpus=18,19,20,21,22,23,24,25,26,27<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">interface=myricom::eth5<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=8192MB,SNF_DESCRING_SIZE=4096MB,SNF_DEBUG_MASK=3<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-----------------<o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2]<o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">(uptime \
– since Jul 2)<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D;background:#F8F8F8">worker-1 \
dropped=39637970 rx=12144046412 0.33%</span><span \
style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><br> \
<span style="background:#F8F8F8">worker-2 dropped=13113250 rx=6088981004 \
0.22%</span><br> <span style="background:#F8F8F8">worker-3 dropped=0 rx=830605595 \
0.00%</span><br> <span style="background:#F8F8F8">worker-4 dropped=0 rx=781722953 \
0.00%</span></span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D;background:#F8F8F8"> </span><o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D;background:#F8F8F8">Totals \
dropped=52751220 rx=19845355964 0.27%</span><o:p></o:p></p> </div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="https://urldefense.com/v3/__http:/mailman.ICSI.Berkeley.EDU/mailman/listinfo/ \
zeek__;!!JYXjzlvb!ytAoH2FY2nNAau06oiM10kVFwFj6Y8_aBIpOMkBQMec7HDTEphWZUHihl0aV0q5QAg$" \
target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Justin<o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>
["image001.png" (image/png)]
_______________________________________________
Zeek mailing list
zeek@zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
--===============0733003822==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic