[prev in list] [next in list] [prev in thread] [next in thread]
List: ntop
Subject: Re: [Ntop] HIGH cpu util and system load
From: "Gary Gatten" <Ggatten () waddell ! com>
Date: 2009-06-18 20:43:47
Message-ID: 70C0964126D66F458E688618E1CD008A0793F017 () WADPEXV0 ! waddell ! com
[Download RAW message or body]
--===============7836958718429963752==
Content-class: urn:content-classes:message
Content-Type: multipart/alternative;
boundary="----_=_NextPart_001_01C9F055.7A9D5D00"
This is a multi-part message in MIME format.
More info: it SEEMED to be working fine (<25% CPU) for many hours -
came back from lunch today and once again it was > 95% CPU. I killed it
and restarted, and CPU is again "normal" - 10% - 25%. The traffic
loads, number of hosts, sessions, etc. have not drastically changed all
day - and in fact are near the high end right now and CPU load is fine.
So, this tells me that after some period of time ntop freaks out about
something and runs crazy. I DID have a problem with Firefox blowing up
on my system which had an active nTop tab. Could this MAYBE be causing
some issues? Also, memory footprint is about 400MB at this time and was
closer to 600MB when it blew up, I still have plenty of physical RAM and
SWAP - so that's not an issue. Actually.... Let me check ulimits to
make sure!!!!
Gary
________________________________
From: Gary Gatten
Sent: Wednesday, June 17, 2009 3:13 PM
To: 'ntop@unipi.it'; 'ntop-dev@unipi.it'
Subject: RE: HIGH cpu util and system load
I changed some start args: disabled decoders, set interface to none,
etc. Been babysitting all day, SO far cpu is < 30% and load is < .5,
however, it's doubled in the last 20 mins or so as has memory
utilization. Most of the CPU load us in the USER space, so I guess
that's "good". Context Switches have increased quite a bit as well.
I'm now showing 34,000 stored hosts (11,000 active) and 34,000 sessions
- these are rounded numbers.
Developers - I know there are many differences between 3.2 and 3.3.
Should any of these changes cause a "significant" increase in CPU
utilization?
Thanks!
Gary
________________________________
From: Gary Gatten
Sent: Wednesday, June 17, 2009 10:32 AM
To: 'ntop@unipi.it'; 'ntop-dev@unipi.it'
Subject: HIGH cpu util and system load
OK, similar to other posts but now I've all but ruled out hardware
issues.
When running v3.2 on FreeBSD 6.0 I was running 3 instances of ntop with
a total of about... 20 netflow interfaces. This was a PIII-750 and
load average and cpu was fine (50% and .5) until it ran out of memory
and started swapping. When I upgraded to 3.3.3-8 I immediately noticed
this system could no longer keep up with the netflow load - even though
our traffic remained pretty much the same. I messed with this for some
time disabling different features, including GeoIP - but no no avail.
So, I upgraded hardware.
Now I'm running Solaris 10 x86 on a VMWare VM. I only have one instance
of ntop 3.3.10 running, but it see's everything through our core LAN -
it now has about 12,000 hosts and 20,000 sessions. I don't know what
the physical hardware is, but the sysadmin tells me I have a single 3GHz
CPU - obviously MUCH more powerfule than a PIII-750, yet CPU util is
still > 90% and load is about 1.3. And I still have 2 other ntop
instances to move over to this platform....
So, what can I do to determine what is causing such a drastic increase
in 3.3.x cpu requirements? I guess more accurately, how can I determine
if ntop is working "correctly", or if there's a bug of some kind causing
a race condition, or maybe I linked against some library with problems?
I'm half tempted to compile v3.2 on this platform and see what happens,
but I'm guess I'll get similar results as the FreeBSD platform.
Any help on this would be GREATLY appreciated! If anyone else running
3.3.8+ is monitoring a large number of hosts with many dynamic sessions,
maybe you can reply with your CPU info?
Oh yea, I restarted this instance last night while traffic was low and
it was using < 10% cpu, when I came in this morning it was > 95% again.
prstat shows 8 LWP's: (1) zombie (libpcap - don't care), 6 mostly in
sleep, and one process in run and consuming 85% of the cpu. I tried to
check this out with gdb but it's not workin too well - it's not listing
thread 1!
Anyway, TIA!
Gary
<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" \
xmlns:st1="urn:schemas-microsoft-com:office:smarttags" \
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name="Microsoft Theme 2.00" content=".htm 011">
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Arial;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
color:black;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>More info: it SEEMED to be working
fine (<25% CPU) for many hours – came back from lunch today and once
again it was > 95% CPU. I killed it and restarted, and CPU is again \
“normal” – 10% - 25%. The traffic loads, number of hosts, \
sessions, etc. have not drastically changed all day – and in fact are near the \
high end right now and CPU load is fine.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>So, this tells me that after some period
of time ntop freaks out about something and runs crazy. I DID have a
problem with Firefox blowing up on my system which had an active nTop
tab. Could this MAYBE be causing some issues? Also, memory
footprint is about 400MB at this time and was closer to 600MB when it blew up,
I still have plenty of physical RAM and SWAP – so that’s not an
issue. Actually…. Let me check ulimits to make \
sure!!!!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><st1:City w:st="on"><st1:place w:st="on"><font size=3
color=black face="Times New Roman"><span \
style='font-size:12.0pt;color:black'>Gary</span></font></st1:place></st1:City><font \
color=black><span style='color:black'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Gary Gatten <br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, June 17, 2009
3:13 PM<br>
<b><span style='font-weight:bold'>To:</span></b> <st1:PersonName \
w:st="on">'ntop@unipi.it'</st1:PersonName>; <st1:PersonName \
w:st="on">'ntop-dev@unipi.it'</st1:PersonName><br> <b><span \
style='font-weight:bold'>Subject:</span></b> RE: HIGH cpu util and system \
load</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>I changed some start args: disabled decoders,
set interface to none, etc. Been babysitting all day, SO far cpu is <
30% and load is < .5, however, it’s doubled in the last 20 mins or so
as has memory utilization. Most of the CPU load us in the USER space, so
I guess that’s “good”. Context Switches have increased
quite a bit as well.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>I’m now showing 34,000 stored hosts
(11,000 active) and 34,000 sessions – these are rounded \
numbers.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>Developers – I know there are many
differences between 3.2 and 3.3. Should any of these changes cause a
“significant” increase in CPU utilization?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>Thanks!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><st1:place w:st="on"><st1:City w:st="on"><font size=3
color=black face="Times New Roman"><span \
style='font-size:12.0pt;color:black'>Gary</span></font></st1:City></st1:place><font \
color=black><span style='color:black'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Gary Gatten <br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, June 17, 2009
10:32 AM<br>
<b><span style='font-weight:bold'>To:</span></b> <st1:PersonName \
w:st="on">'ntop@unipi.it'</st1:PersonName>; <st1:PersonName \
w:st="on">'ntop-dev@unipi.it'</st1:PersonName><br> <b><span \
style='font-weight:bold'>Subject:</span></b> HIGH cpu util and system \
load</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>OK, similar to other posts but now I’ve all but ruled out
hardware issues.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>When running v3.2 on FreeBSD 6.0 I was running 3 instances
of ntop with a total of about… 20 netflow interfaces. This
was a PIII-750 and load average and cpu was fine (50% and .5) until it ran out
of memory and started swapping. When I upgraded to 3.3.3-8 I immediately
noticed this system could no longer keep up with the netflow load – even
though our traffic remained pretty much the same. I messed with
this for some time disabling different features, including GeoIP – but no
no avail. So, I upgraded hardware.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Now I’m running Solaris 10 x86 on a VMWare VM. I
only have one instance of ntop 3.3.10 running, but it see’s everything
through our core LAN – it now has about 12,000 hosts and 20,000
sessions. I don’t know what the physical hardware is, but the
sysadmin tells me I have a single 3GHz CPU – obviously MUCH more
powerfule than a PIII-750, yet CPU util is still > 90% and load is about
1.3. And I still have 2 other ntop instances to move over to this
platform….<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>So, what can I do to determine what is causing such a
drastic increase in 3.3.x cpu requirements? I guess more accurately, how
can I determine if ntop is working “correctly”, or if there’s
a bug of some kind causing a race condition, or maybe I linked against some library
with problems? I’m half tempted to compile v3.2 on this platform
and see what happens, but I’m guess I’ll get similar results as the
FreeBSD platform.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Any help on this would be GREATLY appreciated! If
anyone else running 3.3.8+ is monitoring a large number of hosts with many
dynamic sessions, maybe you can reply with your CPU info? \
<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Oh yea, I restarted this instance last night while traffic
was low and it was using < 10% cpu, when I came in this morning it was >
95% again. prstat shows 8 LWP’s: (1) zombie (libpcap –
don’t care), 6 mostly in sleep, and one process in run and consuming 85%
of the cpu. I tried to check this out with gdb but it’s not workin
too well – it’s not listing thread 1!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Anyway, TIA!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><st1:place w:st="on"><st1:City w:st="on"><font size=2
face=Arial><span style='font-size:10.0pt;font-family:Arial'>Gary</span></font></st1:City></st1:place><font
size=2 face=Arial><span \
style='font-size:10.0pt;font-family:Arial'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt \
0in'> </div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>
</body>
</html>
_______________________________________________
Ntop mailing list
Ntop@unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
--===============7836958718429963752==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic