[prev in list] [next in list] [prev in thread] [next in thread]
List: psad-discuss
Subject: Re: [psad-discuss] Zombie Processes
From: Michael Rash <mbr () cipherdyne ! org>
Date: 2014-06-12 12:48:33
Message-ID: CABv+sEfM5xcDscUNqRMRBK-EnCo0k09rRfjJNcCKO5jAwWLmng () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Wed, Jun 11, 2014 at 11:49 PM, Michael Rash <mbr@cipherdyne.org> wrote:
>
> On Wed, Jun 11, 2014 at 12:35 PM, Dan Dickey <ddickey@icecoldsoftware.com>
> wrote:
>
>> Mike -
>> The defunct processes have all called exit and are done done done.
>> They are still hanging around because the parent process (psad?) hasn't
>> done a wait() call on them to collect the exit information.
>> I haven't looked at the psad code in some time, but it may be worthwhile
>> in the loop logic to call waitpid(-1, &status, WNOHANG) periodically.
>> It would then clean up children processes who have exited.
>>
>
> Thanks for thinking of this, but should this be required given that psad
> just (currently anyway) uses system() to execute the whois client?
>
> https://github.com/mrash/psad/blob/master/psad#L7283
>
> I'll do some more digging - clearly zombies are getting created, and that
> implies exactly what you said about psad not doing a wait() against child
> processes.
>
Seems like what might be happening is that even though system() is being
used, psad is also wrapping system() with an alarm without also calling
waitpid().
Thanks,
--Mike
>
>
>>
>> Just trying to be helpful... I've been using psad on my systems for some
>> time.
>> Thanks for a quality product and the support you've given it over the
>> years!
>>
>
> Glad you like psad, and thanks for the feedback.
>
> --Mike
>
>
>
>
>> -Dan
>>
>> On Wednesday, June 11, 2014 08:52:43 AM Michael Rash wrote:
>> > On Tue, Jun 10, 2014 at 6:18 PM, 3Turtles <3turtles@videotron.ca>
>> wrote:
>> > > Here's what ps is showing me:
>> > >
>> > > UID PID PPID C STIME TTY TIME CMD
>> > > root 1167 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 4689 26489 0 13:59 ? 00:00:00 [sh] <defunct>
>> > > root 6781 26489 0 14:38 ? 00:00:00 [sh] <defunct>
>> > > root 7072 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 7390 26489 0 14:51 ? 00:00:00 [sh] <defunct>
>> > > root 7989 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 8715 26489 0 15:14 ? 00:00:00 [sh] <defunct>
>> > > root 10157 26489 0 15:46 ? 00:00:00 [sh] <defunct>
>> > > root 10249 26489 0 15:48 ? 00:00:00 [sh] <defunct>
>> >
>> > This is most likely an artifact of how psad gathers whois information
>> for
>> > IP's that is has flagged. The problem is that the whois client
>> sometimes
>> > takes a while to return data because it has to query upstream whois
>> > databases over the network. psad makes the tradeoff that if whois is
>> > taking too long to respond, then it doesn't wait around before moving
>> on so
>> > the process becomes a zombie. There is likely a better way to do this
>> > though. I may need to make this more configurable, and I'm hoping that
>> the
>> > whois client itself either already has a 'timeout' parameter (or one
>> can be
>> > added). There is a variable in the psad.conf file WHOIS_TIMEOUT which
>> is
>> > set to 60 seconds by default which seems pretty long. One thing you
>> could
>> > try is disabling whois lookups just to confirm that this is the problem
>> -
>> > use the --no-whois option.
>> >
>> > Thanks,
>> >
>> > --Mike
>> >
>> > > root 13369 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 13709 26489 0 16:53 ? 00:00:00 [sh] <defunct>
>> > > root 15342 26489 0 17:23 ? 00:00:00 [sh] <defunct>
>> > > root 15999 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 17398 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 19833 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 23286 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 25189 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 25546 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 26489 1 0 Jun09 ? 00:00:18 /usr/bin/perl -w
>> > > /usr/sbin/psad
>> > > root 26868 26489 0 00:00 ? 00:00:00 [sh] <defunct>
>> > > root 28371 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 35755 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 36124 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 36214 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 36484 26489 0 03:07 ? 00:00:00 [sh] <defunct>
>> > > root 41507 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 41513 26489 0 04:52 ? 00:00:00 [sh] <defunct>
>> > > root 42148 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 44183 26489 0 05:45 ? 00:00:00 [sh] <defunct>
>> > > root 44235 26489 0 05:46 ? 00:00:00 [sh] <defunct>
>> > > root 44280 26489 0 05:47 ? 00:00:00 [sh] <defunct>
>> > > root 44898 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 45006 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 47485 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 49095 26489 0 07:17 ? 00:00:00 [sh] <defunct>
>> > > root 49538 26489 0 07:27 ? 00:00:00 [sh] <defunct>
>> > > root 50873 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 51348 26489 0 08:03 ? 00:00:00 [sh] <defunct>
>> > > root 51767 26489 0 08:10 ? 00:00:00 [sh] <defunct>
>> > > root 52446 26489 0 08:25 ? 00:00:00 [sh] <defunct>
>> > > root 53859 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 55522 26489 0 09:27 ? 00:00:00 [sh] <defunct>
>> > > root 56889 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 57510 26489 0 10:05 ? 00:00:00 [sh] <defunct>
>> > > root 58433 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 59599 26489 0 10:51 ? 00:00:00 [sh] <defunct>
>> > > root 60515 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 60786 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 62869 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 63332 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 63646 26489 0 Jun09 ? 00:00:00 [sh] <defunct>
>> > > root 63774 26489 0 12:11 ? 00:00:00 [sh] <defunct>
>> > > root 65493 26489 0 12:49 ? 00:00:00 [sh] <defunct>
>> > >
>> > > How do i fix this?
>> > >
>> > > On 08/06/2014 8:51 PM, 3Turtles wrote:
>> > > > My Ubuntu servers are all currently suffering from zombie
>> processes. I
>> > > > narrowed down the culprit to PSAD (sh <defunct>'s parent is psad).
>> > > >
>> > > > In my psad.conf file i have the noemail configured, but emails are
>> still
>> > > > trying to send out and they are failing (i did this on purpose so my
>> > > > email doesnt get spammed to death) and being sent to my root mail
>> > >
>> > > instead.
>> > >
>> > > > Any idea how i can solve this? After a few hours i have around 35
>> > > > zombie processes.
>> > >
>> > >
>> --------------------------------------------------------------------------
>> > > ---->
>> > > > HPCC Systems Open Source Big Data Platform from LexisNexis Risk
>> > > > Solutions
>> > > > Find What Matters Most in Your Big Data with HPCC Systems
>> > > > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>> > > > Leverages Graph Analysis for Fast Processing & Easy Data Exploration
>> > > > http://www.hpccsystems.com
>> > > > _______________________________________________
>> > > > psad-discuss mailing list
>> > > > psad-discuss@lists.sourceforge.net
>> > > > https://lists.sourceforge.net/lists/listinfo/psad-discuss
>> > >
>> > >
>> --------------------------------------------------------------------------
>> > > ---- HPCC Systems Open Source Big Data Platform from LexisNexis Risk
>> > > Solutions Find What Matters Most in Your Big Data with HPCC Systems
>> > > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
>> > > Leverages Graph Analysis for Fast Processing & Easy Data Exploration
>> > > http://p.sf.net/sfu/hpccsystems
>> > > _______________________________________________
>> > > psad-discuss mailing list
>> > > psad-discuss@lists.sourceforge.net
>> > > https://lists.sourceforge.net/lists/listinfo/psad-discuss
>>
>> --
>> Dan A. Dickey
>> ddickey@icecoldsoftware.com
>>
>
>
>
> --
> Michael Rash | Founder
> http://www.cipherdyne.org/
> Key fingerprint = 53EA 13EA 472E 3771 894F AC69 95D8 5D6B A742 839F
>
--
Michael Rash | Founder
http://www.cipherdyne.org/
Key fingerprint = 53EA 13EA 472E 3771 894F AC69 95D8 5D6B A742 839F
[Attachment #5 (text/html)]
<div dir="ltr"><br><div class="gmail_extra">On Wed, Jun 11, 2014 at 11:49 PM, Michael \
Rash <span dir="ltr"><<a href="mailto:mbr@cipherdyne.org" \
target="_blank">mbr@cipherdyne.org</a>></span> wrote:<br><div class="gmail_quote"> \
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="">On \
Wed, Jun 11, 2014 at 12:35 PM, Dan Dickey <span dir="ltr"><<a \
href="mailto:ddickey@icecoldsoftware.com" \
target="_blank">ddickey@icecoldsoftware.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div class=""><blockquote class="gmail_quote" \
style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Mike \
-<br> The defunct processes have all called exit and are done done done.<br>
They are still hanging around because the parent process (psad?) hasn't<br>
done a wait() call on them to collect the exit information.<br>
I haven't looked at the psad code in some time, but it may be worthwhile<br>
in the loop logic to call waitpid(-1, &status, WNOHANG) periodically.<br>
It would then clean up children processes who have \
exited.<br></blockquote><div><br></div></div><div>Thanks for thinking of this, but \
should this be required given that psad just (currently anyway) uses system() to \
execute the whois client?</div>
<div><br></div><div><a href="https://github.com/mrash/psad/blob/master/psad#L7283" \
target="_blank">https://github.com/mrash/psad/blob/master/psad#L7283</a><br></div><div><br></div><div>I'll \
do some more digging - clearly zombies are getting created, and that implies exactly \
what you said about psad not doing a wait() against child processes.</div> \
</div></div></div></blockquote><div><br></div><div>Seems like what might be happening \
is that even though system() is being used, psad is also wrapping system() with an \
alarm without also calling waitpid().</div><div><br> \
</div><div>Thanks,</div><div><br></div><div>--Mike</div><div> </div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"> <div class="">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Just trying to be helpful... I've been using psad on my systems for some \
time.<br> Thanks for a quality product and the support you've given it over the \
years!<br></blockquote><div><br></div></div><div>Glad you like psad, and thanks for \
the feedback.</div><div><br></div><div>--Mike</div><div><div class="h5"> \
<div><br></div><div> <br></div><div> </div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
-Dan<br>
<div><div><br>
On Wednesday, June 11, 2014 08:52:43 AM Michael Rash wrote:<br>
> On Tue, Jun 10, 2014 at 6:18 PM, 3Turtles <<a \
href="mailto:3turtles@videotron.ca" target="_blank">3turtles@videotron.ca</a>> \
wrote:<br> > > Here's what ps is showing me:<br>
> ><br>
> > UID PID PPID C STIME TTY TIME CMD<br>
> > root 1167 26489 0 Jun09 ? 00:00:00 [sh] \
<defunct><br> > > root 4689 26489 0 13:59 ? 00:00:00 \
[sh] <defunct><br> > > root 6781 26489 0 14:38 ? \
00:00:00 [sh] <defunct><br> > > root <a \
href="tel:7072%2026489%20%C2%A00" value="+17072264890" target="_blank">7072 26489 \
0</a> Jun09 ? 00:00:00 [sh] <defunct><br> > > root \
7390 26489 0 14:51 ? 00:00:00 [sh] <defunct><br> > > root \
7989 26489 0 Jun09 ? 00:00:00 [sh] <defunct><br> > > root \
8715 26489 0 15:14 ? 00:00:00 [sh] <defunct><br> > > root \
10157 26489 0 15:46 ? 00:00:00 [sh] <defunct><br> > > root \
10249 26489 0 15:48 ? 00:00:00 [sh] <defunct><br> ><br>
> This is most likely an artifact of how psad gathers whois information for<br>
> IP's that is has flagged. The problem is that the whois client \
sometimes<br> > takes a while to return data because it has to query upstream \
whois<br> > databases over the network. psad makes the tradeoff that if whois \
is<br> > taking too long to respond, then it doesn't wait around before moving \
on so<br> > the process becomes a zombie. There is likely a better way to do \
this<br> > though. I may need to make this more configurable, and I'm hoping \
that the<br> > whois client itself either already has a 'timeout' \
parameter (or one can be<br> > added). There is a variable in the psad.conf file \
WHOIS_TIMEOUT which is<br> > set to 60 seconds by default which seems pretty long. \
One thing you could<br> > try is disabling whois lookups just to confirm that this \
is the problem -<br> > use the --no-whois option.<br>
><br>
> Thanks,<br>
><br>
> --Mike<br>
><br>
> > root <a href="tel:13369%2026489%20%C2%A00" value="+13369264890" \
target="_blank">13369 26489 0</a> Jun09 ? 00:00:00 [sh] \
<defunct><br> > > root 13709 26489 0 16:53 ? 00:00:00 \
[sh] <defunct><br> > > root 15342 26489 0 17:23 ? \
00:00:00 [sh] <defunct><br> > > root 15999 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 17398 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 19833 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 23286 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 25189 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 25546 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 26489 1 0 Jun09 ? \
00:00:18 /usr/bin/perl -w<br> > > /usr/sbin/psad<br>
> > root 26868 26489 0 00:00 ? 00:00:00 [sh] \
<defunct><br> > > root 28371 26489 0 Jun09 ? 00:00:00 \
[sh] <defunct><br> > > root 35755 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 36124 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 36214 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 36484 26489 0 03:07 ? \
00:00:00 [sh] <defunct><br> > > root 41507 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 41513 26489 0 04:52 ? \
00:00:00 [sh] <defunct><br> > > root 42148 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 44183 26489 0 05:45 ? \
00:00:00 [sh] <defunct><br> > > root 44235 26489 0 05:46 ? \
00:00:00 [sh] <defunct><br> > > root 44280 26489 0 05:47 ? \
00:00:00 [sh] <defunct><br> > > root 44898 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 45006 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 47485 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 49095 26489 0 07:17 ? \
00:00:00 [sh] <defunct><br> > > root 49538 26489 0 07:27 ? \
00:00:00 [sh] <defunct><br> > > root 50873 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 51348 26489 0 08:03 ? \
00:00:00 [sh] <defunct><br> > > root 51767 26489 0 08:10 ? \
00:00:00 [sh] <defunct><br> > > root 52446 26489 0 08:25 ? \
00:00:00 [sh] <defunct><br> > > root 53859 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 55522 26489 0 09:27 ? \
00:00:00 [sh] <defunct><br> > > root 56889 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 57510 26489 0 10:05 ? \
00:00:00 [sh] <defunct><br> > > root 58433 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 59599 26489 0 10:51 ? \
00:00:00 [sh] <defunct><br> > > root 60515 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 60786 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 62869 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 63332 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 63646 26489 0 Jun09 ? \
00:00:00 [sh] <defunct><br> > > root 63774 26489 0 12:11 ? \
00:00:00 [sh] <defunct><br> > > root 65493 26489 0 12:49 ? \
00:00:00 [sh] <defunct><br> > ><br>
> > How do i fix this?<br>
> ><br>
> > On 08/06/2014 8:51 PM, 3Turtles wrote:<br>
> > > My Ubuntu servers are all currently suffering from zombie processes. \
I<br> > > > narrowed down the culprit to PSAD (sh <defunct>'s \
parent is psad).<br> > > ><br>
> > > In my psad.conf file i have the noemail configured, but emails are \
still<br> > > > trying to send out and they are failing (i did this on \
purpose so my<br> > > > email doesnt get spammed to death) and being sent to \
my root mail<br> > ><br>
> > instead.<br>
> ><br>
> > > Any idea how i can solve this? After a few hours i have around \
35<br> > > > zombie processes.<br>
> ><br>
> > --------------------------------------------------------------------------<br>
> > ----><br>
> > > HPCC Systems Open Source Big Data Platform from LexisNexis Risk<br>
> > > Solutions<br>
> > > Find What Matters Most in Your Big Data with HPCC Systems<br>
> > > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.<br>
> > > Leverages Graph Analysis for Fast Processing & Easy Data \
Exploration<br> > > > <a href="http://www.hpccsystems.com" \
target="_blank">http://www.hpccsystems.com</a><br> > > > \
_______________________________________________<br> > > > psad-discuss \
mailing list<br> > > > <a href="mailto:psad-discuss@lists.sourceforge.net" \
target="_blank">psad-discuss@lists.sourceforge.net</a><br> > > > <a \
href="https://lists.sourceforge.net/lists/listinfo/psad-discuss" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/psad-discuss</a><br> \
> ><br> > > \
--------------------------------------------------------------------------<br> \
</div></div>> > ---- HPCC Systems Open Source Big Data Platform from LexisNexis \
Risk<br> <div>> > Solutions Find What Matters Most in Your Big Data with HPCC \
Systems<br> > > Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.<br>
> > Leverages Graph Analysis for Fast Processing & Easy Data \
Exploration<br> > > <a href="http://p.sf.net/sfu/hpccsystems" \
target="_blank">http://p.sf.net/sfu/hpccsystems</a><br> > > \
_______________________________________________<br> > > psad-discuss mailing \
list<br> > > <a href="mailto:psad-discuss@lists.sourceforge.net" \
target="_blank">psad-discuss@lists.sourceforge.net</a><br> > > <a \
href="https://lists.sourceforge.net/lists/listinfo/psad-discuss" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/psad-discuss</a><br> \
<br>
--<br>
</div>Dan A. Dickey<br>
<a href="mailto:ddickey@icecoldsoftware.com" \
target="_blank">ddickey@icecoldsoftware.com</a><br> \
</blockquote></div></div></div><br><br clear="all"><div class=""><div><br></div>-- \
<br><div dir="ltr">Michael Rash | Founder<br><a href="http://www.cipherdyne.org/" \
target="_blank">http://www.cipherdyne.org/</a><br>Key fingerprint = 53EA 13EA 472E \
3771 894F AC69 95D8 5D6B A742 839F<br>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Michael \
Rash | Founder<br><a href="http://www.cipherdyne.org/" \
target="_blank">http://www.cipherdyne.org/</a><br>Key fingerprint = 53EA 13EA 472E \
3771 894F AC69 95D8 5D6B A742 839F<br> </div>
</div></div>
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
psad-discuss mailing list
psad-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/psad-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic