[prev in list] [next in list] [prev in thread] [next in thread]
List: security-onion
Subject: Re: [security-onion] elsa cron job errors and problems
From: Martin Holste <mcholste () gmail ! com>
Date: 2013-03-28 17:11:18
Message-ID: CANpnLHhLKb3cYjYTa+kmM8BLz5LubYTaJFZUVYhjWMiW+a48QQ () mail ! gmail ! com
[Download RAW message or body]
Yes, that's expected (it's doing cleaning and consolidation). Open a
second console and use less or tail to watch
/nsm/elsa/data/elsa/log/node.log.
On Thu, Mar 28, 2013 at 12:01 PM, Richard <imageek72@gmail.com> wrote:
> Did as you suggested, and running it by hand for over 20 mins now and i
> have no output and it is still running. Is there a way to run this and be
> able to see some kind of debugging output or something?
>
>
>
> On Thursday, March 28, 2013 10:16:33 AM UTC-6, Martin wrote:
> > All of this mess is fixed in the brand new ELSA code released this week,
> but it will be a bit before it makes it into SO. In the meantime, the best
> way to troubleshoot is to disable the cron job, stop syslog-ng, make sure
> there are no perl processes running anywhere, then start syslog-ng.
> Lastly, manually run the cron.pl job to see if it is getting any errors
> (there may be a corrupt database table).
> >
> >
> >
> >
> >
> > On Thu, Mar 28, 2013 at 11:09 AM, Scott Runnels <srun...@gmail.com>
> wrote:
> >
> >
> >
> > Did you recently update any packages?
> >
> >
> >
> >
> > Scott Runnels
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 28, 2013 at 12:07 PM, Richard <imag...@gmail.com> wrote:
> >
> >
> >
> > Last night I started getting alerts for elsa cron jobs:
> >
> > Use of uninitialized value $epochdate in localtime at
> /opt/elsa/web/lib/Utils.pm line 135.
> >
> > Use of uninitialized value $epochdate in localtime at
> /opt/elsa/web/lib/Utils.pm line 135.
> >
> >
> >
> > Then shortly after these started, i started receiving the following
> errors:
> >
> >
> >
> > Use of uninitialized value $query in concatenation (.) or string at
> /opt/elsa/web/lib/Utils.pm line 99.
> >
> > DBI connect('database=elsa_web','elsa',...) failed: Too many connections
> QUERY: at /opt/elsa/web/lib/Utils.pm line 105
> >
> > Utils::_dbh_error_handler('DBI
> connect(\'database=elsa_web\',\'elsa\',...) failed: Too m...',
> 'DBI::dr=HASH(0x3794510)', undef) called at /usr/lib/perl5/DBI.pm line 677
> >
> > DBI::__ANON__(undef, undef) called at /usr/lib/perl5/DBI.pm line
> 734
> >
> > DBI::connect('DBI', 'dbi:mysql:database=elsa_web', 'elsa',
> 'biglog', 'HASH(0x36dc098)') called at /usr/lib/perl5/DBI.pm line 575
> >
> > DBI::connect_cached('DBI', 'dbi:mysql:database=elsa_web',
> 'elsa', 'biglog', 'HASH(0x2e21788)') called at /opt/elsa/web/lib/Utils.pm
> line 67
> >
> > Class::MOP::Class:::around('CODE(0x17e5090)', 'API',
> 'config_file', '/etc/elsa_web.conf') called at
> /usr/lib/perl5/Class/MOP/Method/Wrapped.pm line 162
> >
> > Class::MOP::Method::Wrapped::__ANON__('API', 'config_file',
> '/etc/elsa_web.conf') called at /usr/lib/perl5/Class/MOP/Method/Wrapped.pm
> line 91
> >
> > API::BUILDARGS('API', 'config_file', '/etc/elsa_web.conf')
> called at constructor API::new (defined at /opt/elsa/web/lib/API.pm line
> 4534) line 6
> >
> > API::new('API', 'config_file', '/etc/elsa_web.conf') called at
> /opt/elsa/web/cron.pl line 27
> >
> >
> >
> >
> >
> > The part that sticks out for that error is the second line where it says
> Too many connections. Why are there too many connections? I login and see
> that there are literally hundreds of perl processes for /opt/elsa/web/
> cron.pl last night and so i killed them off and ended up having to
> restart the system.
> >
> >
> >
> >
> >
> >
> > If i leave the the job uncommented in /etc/cron.d/elsa then each of
> these jobs jumps to the process list and eats up as much CPU as possible
> and the processes never die. I start seeing the errors again soon and the
> resources on the box become all used up eventually and i have to restart my
> server.
> >
> >
> >
> >
> >
> >
> > Are there any other logs i could be looking in for errors or to figure
> out why these processes seem to get hung and never complete? We do not
> really use Elsa yet, so totally disabling Elsa coule be a potential fix,
> but i would like to be able to use Elsa when i have the time to learn how
> to use it.
> >
> >
> >
> >
> >
> >
> > --
> >
> > You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> >
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com.
> >
> > To post to this group, send email to securit...@googlegroups.com.
> >
> > Visit this group at
> http://groups.google.com/group/security-onion?hl=en-US.
> >
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> >
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com.
> >
> > To post to this group, send email to securit...@googlegroups.com.
> >
> > Visit this group at
> http://groups.google.com/group/security-onion?hl=en-US.
> >
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
You received this message because you are subscribed to the Google Groups \
"security-onion" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to security-onion+unsubscribe@googlegroups.com. To post to this \
group, send email to security-onion@googlegroups.com. Visit this group at \
http://groups.google.com/group/security-onion?hl=en-US. For more options, visit \
https://groups.google.com/groups/opt_out.
[Attachment #3 (text/html)]
<div dir="ltr">Yes, that's expected (it's doing cleaning and consolidation). \
Open a second console and use less or tail to watch \
/nsm/elsa/data/elsa/log/node.log.</div><div class="gmail_extra"><br><br><div \
class="gmail_quote">
On Thu, Mar 28, 2013 at 12:01 PM, Richard <span dir="ltr"><<a \
href="mailto:imageek72@gmail.com" target="_blank">imageek72@gmail.com</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex">
Did as you suggested, and running it by hand for over 20 mins now and i have no \
output and it is still running. Is there a way to run this and be able to see some \
kind of debugging output or something?<br> <div class="im"><br>
<br>
<br>
On Thursday, March 28, 2013 10:16:33 AM UTC-6, Martin wrote:<br>
> All of this mess is fixed in the brand new ELSA code released this week, but it \
will be a bit before it makes it into SO. In the meantime, the best way to \
troubleshoot is to disable the cron job, stop syslog-ng, make sure there are no perl \
processes running anywhere, then start syslog-ng. Lastly, manually run the <a \
href="http://cron.pl" target="_blank">cron.pl</a> job to see if it is getting any \
errors (there may be a corrupt database table).<br>
><br>
><br>
><br>
><br>
><br>
</div><div class="im">> On Thu, Mar 28, 2013 at 11:09 AM, Scott Runnels <<a \
href="mailto:srun...@gmail.com">srun...@gmail.com</a>> wrote:<br> ><br>
><br>
><br>
> Did you recently update any packages?<br>
><br>
><br>
><br>
><br>
> Scott Runnels<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div><div class="im">> On Thu, Mar 28, 2013 at 12:07 PM, Richard <<a \
href="mailto:imag...@gmail.com">imag...@gmail.com</a>> wrote:<br> ><br>
><br>
><br>
> Last night I started getting alerts for elsa cron jobs:<br>
><br>
> Use of uninitialized value $epochdate in localtime at /opt/elsa/web/lib/Utils.pm \
line 135.<br> ><br>
> Use of uninitialized value $epochdate in localtime at /opt/elsa/web/lib/Utils.pm \
line 135.<br> ><br>
><br>
><br>
> Then shortly after these started, i started receiving the following errors:<br>
><br>
><br>
><br>
> Use of uninitialized value $query in concatenation (.) or string at \
/opt/elsa/web/lib/Utils.pm line 99.<br> ><br>
> DBI connect('database=elsa_web','elsa',...) failed: Too many \
connections QUERY: at /opt/elsa/web/lib/Utils.pm line 105<br> ><br>
> Utils::_dbh_error_handler('DBI \
connect(\'database=elsa_web\',\'elsa\',...) failed: Too m...', \
'DBI::dr=HASH(0x3794510)', undef) called at /usr/lib/perl5/DBI.pm line \
677<br> ><br>
> DBI::__ANON__(undef, undef) called at /usr/lib/perl5/DBI.pm line 734<br>
><br>
> DBI::connect('DBI', 'dbi:mysql:database=elsa_web', \
'elsa', 'biglog', 'HASH(0x36dc098)') called at \
/usr/lib/perl5/DBI.pm line 575<br> ><br>
> DBI::connect_cached('DBI', \
'dbi:mysql:database=elsa_web', 'elsa', 'biglog', \
'HASH(0x2e21788)') called at /opt/elsa/web/lib/Utils.pm line 67<br> ><br>
> Class::MOP::Class:::around('CODE(0x17e5090)', 'API', \
'config_file', '/etc/elsa_web.conf') called at \
/usr/lib/perl5/Class/MOP/Method/Wrapped.pm line 162<br> ><br>
> Class::MOP::Method::Wrapped::__ANON__('API', \
'config_file', '/etc/elsa_web.conf') called at \
/usr/lib/perl5/Class/MOP/Method/Wrapped.pm line 91<br> ><br>
> API::BUILDARGS('API', 'config_file', \
'/etc/elsa_web.conf') called at constructor API::new (defined at \
/opt/elsa/web/lib/API.pm line 4534) line 6<br> ><br>
> API::new('API', 'config_file', \
'/etc/elsa_web.conf') called at /opt/elsa/web/<a href="http://cron.pl" \
target="_blank">cron.pl</a> line 27<br> ><br>
><br>
><br>
><br>
><br>
> The part that sticks out for that error is the second line where it says Too \
many connections. Why are there too many connections? I login and see that there are \
literally hundreds of perl processes for /opt/elsa/web/<a href="http://cron.pl" \
target="_blank">cron.pl</a> last night and so i killed them off and ended up having \
to restart the system.<br>
><br>
><br>
><br>
><br>
><br>
><br>
> If i leave the the job uncommented in /etc/cron.d/elsa then each of these jobs \
jumps to the process list and eats up as much CPU as possible and the processes never \
die. I start seeing the errors again soon and the resources on the box become all \
used up eventually and i have to restart my server.<br>
><br>
><br>
><br>
><br>
><br>
><br>
> Are there any other logs i could be looking in for errors or to figure out why \
these processes seem to get hung and never complete? We do not really use Elsa yet, \
so totally disabling Elsa coule be a potential fix, but i would like to be able to \
use Elsa when i have the time to learn how to use it.<br>
><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> ><br>
</div>> To unsubscribe from this group and stop receiving emails from it, send an \
email to <a href="mailto:security-onio...@googlegroups.com">security-onio...@googlegroups.com</a>.<br>
><br>
> To post to this group, send email to <a \
href="mailto:securit...@googlegroups.com">securit...@googlegroups.com</a>.<br> <div \
class="im">><br> > Visit this group at <a \
href="http://groups.google.com/group/security-onion?hl=en-US" \
target="_blank">http://groups.google.com/group/security-onion?hl=en-US</a>.<br> \
><br> > For more options, visit <a \
href="https://groups.google.com/groups/opt_out" \
target="_blank">https://groups.google.com/groups/opt_out</a>.<br> ><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> ><br>
</div>> To unsubscribe from this group and stop receiving emails from it, send an \
email to <a href="mailto:security-onio...@googlegroups.com">security-onio...@googlegroups.com</a>.<br>
><br>
> To post to this group, send email to <a \
href="mailto:securit...@googlegroups.com">securit...@googlegroups.com</a>.<br> <div \
class="HOEnZb"><div class="h5">><br> > Visit this group at <a \
href="http://groups.google.com/group/security-onion?hl=en-US" \
target="_blank">http://groups.google.com/group/security-onion?hl=en-US</a>.<br> \
><br> > For more options, visit <a \
href="https://groups.google.com/groups/opt_out" \
target="_blank">https://groups.google.com/groups/opt_out</a>.<br> <br>
--<br>
You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br> To unsubscribe from this group and stop \
receiving emails from it, send an email to <a \
href="mailto:security-onion%2Bunsubscribe@googlegroups.com">security-onion+unsubscribe@googlegroups.com</a>.<br>
To post to this group, send email to <a \
href="mailto:security-onion@googlegroups.com">security-onion@googlegroups.com</a>.<br>
Visit this group at <a href="http://groups.google.com/group/security-onion?hl=en-US" \
target="_blank">http://groups.google.com/group/security-onion?hl=en-US</a>.<br> For \
more options, visit <a href="https://groups.google.com/groups/opt_out" \
target="_blank">https://groups.google.com/groups/opt_out</a>.<br> <br>
<br>
</div></div></blockquote></div><br></div>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups \
"security-onion" group.<br /> To unsubscribe from this group and stop \
receiving emails from it, send an email to \
security-onion+unsubscribe@googlegroups.com.<br /> To post to this group, send email \
to security-onion@googlegroups.com.<br /> Visit this group at <a \
href="http://groups.google.com/group/security-onion?hl=en-US">http://groups.google.com/group/security-onion?hl=en-US</a>.<br \
/> For more options, visit <a \
href="https://groups.google.com/groups/opt_out">https://groups.google.com/groups/opt_out</a>.<br \
/> <br />
<br />
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic