[prev in list] [next in list] [prev in thread] [next in thread]
List: bacula-bugs
Subject: [Bacula-bugs] [bacula 0001833]: 5.2.5 Director Crashes while backing up Catalog
From: Mantis Bug Tracker <nobody () bugs ! bacula ! org>
Date: 2012-04-18 11:37:13
Message-ID: 315699e2e663f40b18ecec54cb131789 () bugs ! bacula ! org
[Download RAW message or body]
The following issue has been CLOSED
======================================================================
http://bugs.bacula.org/view.php?id=1833
======================================================================
Reported By: stephen
Assigned To:
======================================================================
Project: bacula
Issue ID: 1833
Category: Director
Reproducibility: have not tried
Severity: crash
Priority: high
Status: closed
Resolution: unable to reproduce
Fixed in Version:
======================================================================
Date Submitted: 2012-02-15 15:37 GMT
Last Modified: 2012-04-18 12:37 BST
======================================================================
Summary: 5.2.5 Director Crashes while backing up Catalog
Description:
Daily operations running normally, then Director crashes with traceback while
backing up Catalog. Attaching traceback.
Steps to Reproduce:
None
Additional Information:
None
======================================================================
----------------------------------------------------------------------
(0006186) kern (administrator) - 2012-02-16 09:50
http://bugs.bacula.org/view.php?id=1833#c6186
----------------------------------------------------------------------
This is an interesting bug. I don't think it has anything to do with backing up
the Catalog. It appears to be related to someone from IP 128.32.149.5 trying
to connect to the Director with an incorrect log in. This may be a
monitoring program that probes Bacula incorrectly.
Of course, even an incorrect probe of the Director should not cause it to crash.
The crash seems to come either from a race condition or an improper Messages
configuration. Would you please post your bacula-dir.conf file (of particular
interest is the Messages resource(s)).
----------------------------------------------------------------------
(0006187) stephen (reporter) - 2012-02-16 17:39
http://bugs.bacula.org/view.php?id=1833#c6187
----------------------------------------------------------------------
We do have monitoring in place; it's not so much incorrect, as merely a port
check probe, so bacula sees it as an invalidated connection. We've been running
that for years without negative consequence.
Another thing worth mentioning, it's possible that a 'reload' of the Director
was issued while the Catalog was being dumped. Perhaps a race condition there,
where the reload is issued, but the database is locked, and the Director freaks
out?
Director {
Name = lawson-dir
Messages = Daemon
}
Messages {
Name = Standard
mailcommand = "/opt/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s
\"Bacula: %t %e of %c %l\" %r"
operatorcommand = "/opt/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\"
-s \"Bacula: Intervention needed for %j\" %r"
mail = bacula@seismo.berkeley.edu = all, !skipped, !saved
console = all, !skipped, !saved
append = "/var/bacula/work/log" = all, !skipped
}
Messages {
Name = Daemon
mailcommand = "/opt/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s
\"Bacula daemon message\" %r"
console = all, !skipped, !saved
append = "/var/bacula/work/log" = all, !skipped
}
----------------------------------------------------------------------
(0006193) stephen (reporter) - 2012-02-27 23:35
http://bugs.bacula.org/view.php?id=1833#c6193
----------------------------------------------------------------------
This has happened twice more and each time the scenario was that a 'reload' was
issued to the Director while the Catalog was being dumped, and the crash of the
Directory happened just after the Run Before script (call to mysqldump) returns.
There are some config changes we've been automating by having a script change
the job files each night before our fulls run (hence the need for the reload).
The timing of this was not intended to happen during the Catalog dump, but
sometimes does if the Catalog dump is delayed.
At any rate, I'm pretty sure the 'reload' during a Catalog dump is the
underlying common detail to all our recent Director crashes. Perhaps a reload
during a dump would not be advised, but should it crash the Director?
Thanks!
----------------------------------------------------------------------
(0006195) kern (administrator) - 2012-02-28 08:32
http://bugs.bacula.org/view.php?id=1833#c6195
----------------------------------------------------------------------
It is quite possible that the crash has something to do with a reload, but
the "real" cause is the probe in the traceback I saw. If you could post another
one or two, I can see if they are also due to the probe. Just the .traceback
file would be sufficient.
The probe generates an error message in Bacula, when Bacula attempts to write
it to your File log, it calls a glibc fputs() subroutine, and the crash
occurs in that routine. So in fact, it appears to be the system library
that crashes.
You can also comment out lines 302 and 303 in <bacula>/src/dird/authenticate.c
and that should solve the problem, providing it always manifests itself in the
same way.
----------------------------------------------------------------------
(0006290) kern (administrator) - 2012-04-18 12:37
http://bugs.bacula.org/view.php?id=1833#c6290
----------------------------------------------------------------------
If this is connected with a reload, the problem probably comes from an incorrect
bacula-dir.conf file. One should always run ./bacula-dir -t before reloading a
new conf file. From the dump, the problem is always created by someone probing
the director with an incorrect packet.
If you can show definitively that this comes from a reload, please reopen the
bug report. Otherwise, since I cannot reproduce it, and the crash is coming
from a comm connection, I cannot see any way to tie it down so it can be fixed.
Issue History
Date Modified Username Field Change
======================================================================
2012-02-15 15:37 stephen New Issue
2012-02-15 15:38 stephen File Added: lawson-dir.1169.bactrace
2012-02-15 15:39 stephen File Added: bacula.1169.traceback
2012-02-16 09:50 kern Note Added: 0006186
2012-02-16 09:50 kern Status new => feedback
2012-02-16 17:39 stephen Note Added: 0006187
2012-02-16 17:39 stephen Status feedback => new
2012-02-16 17:39 stephen Note Edited: 0006187
2012-02-27 23:35 stephen Note Added: 0006193
2012-02-28 08:32 kern Note Added: 0006195
2012-02-28 08:32 kern Status new => feedback
2012-04-18 12:37 kern Note Added: 0006290
2012-04-18 12:37 kern Status feedback => closed
2012-04-18 12:37 kern Resolution open => unable to
reproduce
======================================================================
------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Bacula-bugs mailing list
Bacula-bugs@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-bugs
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic