[prev in list] [next in list] [prev in thread] [next in thread]
List: bacula-users
Subject: [Bacula-users] Bacula fd authentication problem
From: Boris Petrovic <boris.petrovic () united ! cloud>
Date: 2018-01-15 15:29:07
Message-ID: B614CB23-355D-4328-BCE7-FCF51E3AEEF2 () united ! cloud
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hi everyone,
I'm experiencing strange problems on some of my bacula file daemons since I have \
pathed Ubuntu kernel for Meltdown/Spectre vulnerability. I'm getting following error \
message in bconsole after job fails:
15-Jan 15:24 tools JobId 5963: Fatal error: Authorization key rejected by Storage \
daemon. Please see http://www.bacula.org/en/rel-manual/Bacula_Freque_Asked_Questi.html#SECTION00260000000000000000 \
for help. 15-Jan 15:24 bacula-dir JobId 5963: Fatal error: Bad response to Storage \
command: wanted 2000 OK storage , got 2902 Bad storage
15-Jan 15:24 bacula-dir JobId 5963: Error: Bacula bacula-dir 7.0.5 (28Jul14):
Build OS: x86_64-pc-linux-gnu ubuntu 16.04
JobId: 5963
Job: Tools-Backup.2018-01-15_15.24.27_06
Backup Level: Full
Client: "tools" 7.0.5 (28Jul14) x86_64-pc-linux-gnu,ubuntu,16.04
FileSet: "Full-Fileset" 2017-10-09 14:24:49
Pool: "Full-Backup-Pool" (From Command input)
Catalog: "nvg-backup-catalog" (From Client resource)
Storage: "backup-storage" (From Pool resource)
Scheduled time: 15-Jan-2018 15:24:25
Start time: 15-Jan-2018 15:24:29
End time: 15-Jan-2018 15:24:40
Elapsed time: 11 secs
Priority: 10
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s):
Volume Session Id: 2
Volume Session Time: 1516026063
Last Volume Bytes: 1,674,336,771,120 (1.674 TB)
Non-fatal FD errors: 2
SD Errors: 0
FD termination status: Error
SD termination status: Waiting on FD
Termination: *** Backup Error ***
On client side, when bacula-fd is started as a foreground process with maximum debug \
output (bacula-fd -f -d 400 -c /etc/bacula/bacula-fd.conf) I see that file daemon \
menages to connect to bacula SD, but fails to authenticate. This is bacula-fd process \
output:
tools: job.c:1855-0 StorageCmd: storage address= baculaSD.my.domain port=9103 ssl=0
tools: job.c:1883-0 Connect to storage: baculaSD.my.domain:9103 ssl=0
tools: bsock.c:208-0 Current 127.0.0.1:9103 All 127.0.0.1:9103
tools: bsock.c:137-0 who=Storage daemon host=baculaSD.my.domain port=9103
tools: bsock.c:310-0 OK connected to server Storage daemon baculaSD.my.domain:9103.
tools: job.c:1899-0 Connection OK to SD.
tools: authenticate.c:237-0 Send to SD: Hello Bacula SD: Start Job \
Tools-Backup.2018-01-15_15.24.27_06 5
tools: authenticate.c:240-0 ==== respond to SD challenge
tools: authenticate.c:247-0 cram_respond failed for SD: Storage daemon
tools: job.c:306-0 Quit command loop. Canceled=1
tools: runscript.c:96-0 runscript: running all RUNSCRIPT object (ClientAfterJob) \
JobStatus=f
tools: job.c:417-0 Calling term_find_files
tools: job.c:420-0 Done with term_find_files
tools: jcr.c:171-0 write_last_jobs seek to 192
tools: job.c:423-0 Done with free_jcr
tools: mem_pool.c:366-0 garbage collect memory pool
baculaSD.my.domain resolvs to localhost because this file daemon is on public server, \
and bacula server is behind NAT so I have ssh tunnel between server and client and it \
works perfectly fine for other hosts. And as you can see in client's output it is \
able to connect to storage daemon so I belive networking is not the issue. I also \
have duble-checked director name and password ant it matches on director and client's \
side.
Bacula director version:
bacula-dir Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
Bacula client version:
bacula-fd:
Installed: 7.0.5+dfsg-4ubuntu0.1
Candidate: 7.0.5+dfsg-4ubuntu0.1
I would appreciate any help since I'm bit stuck here.
Boris
[Attachment #3 (text/html)]
<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi \
everyone, <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">I'm experiencing strange problems on some of my bacula file \
daemons since I have pathed Ubuntu kernel for Meltdown/Spectre \
vulnerability.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">I'm getting following error message in bconsole after job \
fails:<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">15-Jan 15:24 tools JobId 5963: Fatal error: Authorization \
key rejected by Storage daemon.</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">Please see <a \
href="http://www.bacula.org/en/rel-manual/Bacula_Freque_Asked_Questi.html#SECTION00260 \
000000000000000">http://www.bacula.org/en/rel-manual/Bacula_Freque_Asked_Questi.html#SECTION00260000000000000000</a> for
help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">15-Jan 15:24 bacula-dir JobId \
5963: Fatal error: Bad response to Storage command: wanted 2000 OK \
storage<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">, \
got 2902 Bad storage<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">15-Jan 15:24 bacula-dir JobId 5963: Error: Bacula bacula-dir \
7.0.5 (28Jul14):<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Build \
OS: \
x86_64-pc-linux-gnu ubuntu 16.04<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> \
JobId: \
5963<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Job: \
Tools-Backup.2018-01-15_15.24.27_06<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Backup \
Level: \
Full<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Client: \
"tools" 7.0.5 (28Jul14) \
x86_64-pc-linux-gnu,ubuntu,16.04<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> \
FileSet: \
"Full-Fileset" 2017-10-09 14:24:49<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> \
Pool: \
"Full-Backup-Pool" (From Command input)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> \
Catalog: \
"nvg-backup-catalog" (From Client resource)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> \
Storage: \
"backup-storage" (From Pool resource)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> Scheduled \
time: 15-Jan-2018 \
15:24:25<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Start \
time: \
15-Jan-2018 15:24:29<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> End \
time: \
15-Jan-2018 15:24:40<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Elapsed \
time: 11 \
secs<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Priority: \
10<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
FD Files Written: 0<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> SD Files \
Written: 0<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> FD Bytes \
Written: 0 (0 B)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> SD Bytes \
Written: 0 (0 B)<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> \
Rate: \
0.0 KB/s<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Software Compression: \
None<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
VSS: \
no<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Encryption: \
no<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Accurate: \
no<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Volume name(s):<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Volume Session Id: \
2<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Volume Session Time: 1516026063<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> Last Volume \
Bytes: 1,674,336,771,120 (1.674 \
TB)<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> \
Non-fatal FD errors: 2<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> SD \
Errors: \
0<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"> FD \
termination status: Error<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <b>SD termination status: Waiting on \
FD</b><o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> \
Termination: *** \
Backup Error ***<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">On client side, when bacula-fd is started as a foreground \
process with maximum debug output (bacula-fd -f -d 400 -c /etc/bacula/bacula-fd.conf) \
I see that file daemon menages to connect to bacula SD, but fails to authenticate. \
This is bacula-fd process output:<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: job.c:1855-0 StorageCmd: storage address= \
baculaSD.my.domain port=9103 ssl=0<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: job.c:1883-0 Connect to storage: \
baculaSD.my.domain:9103 ssl=0<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: bsock.c:208-0 Current 127.0.0.1:9103 All \
127.0.0.1:9103<o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">tools: bsock.c:137-0 who=Storage daemon \
host=baculaSD.my.domain port=9103</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">tools: bsock.c:310-0 OK connected to server Storage \
daemon baculaSD.my.domain:9103.</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">tools: job.c:1899-0 Connection OK to SD.</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">tools: authenticate.c:237-0 Send to SD: Hello Bacula SD: \
Start Job Tools-Backup.2018-01-15_15.24.27_06 5</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><b><span \
style="font-size:11.0pt">tools: authenticate.c:240-0 ==== respond to SD \
challenge</span></b><span style="font-size:11.0pt"><o:p></o:p></span></p> <p \
class="MsoNormal"><b><span style="font-size:11.0pt">tools: authenticate.c:247-0 \
cram_respond failed for SD: Storage daemon</span></b><span \
style="font-size:11.0pt"><o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: job.c:306-0 Quit command loop. \
Canceled=1<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: runscript.c:96-0 runscript: running all RUNSCRIPT \
object (ClientAfterJob) JobStatus=f<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: job.c:417-0 Calling \
term_find_files<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: job.c:420-0 Done with \
term_find_files<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: jcr.c:171-0 write_last_jobs seek to \
192<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">tools: \
job.c:423-0 Done with free_jcr<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">tools: mem_pool.c:366-0 garbage collect memory \
pool<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">baculaSD.my.domain resolvs to localhost because this file \
daemon is on public server, and bacula server is behind NAT so I have ssh tunnel \
between server and client and it works perfectly fine for other hosts. And as you \
can see in client's output it is able to connect to storage daemon so I belive \
networking is not the issue.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">I also have duble-checked director name and password ant it \
matches on director and client's side.<br> <br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Bacula director \
version:<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">bacula-dir Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu \
ubuntu 16.04<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">Bacula client version:<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt">bacula-fd:<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="font-size:11.0pt"> Installed: \
7.0.5+dfsg-4ubuntu0.1<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> Candidate: \
7.0.5+dfsg-4ubuntu0.1<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">I would appreciate any help since I'm bit stuck \
here.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt">Boris<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"> <o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt"><o:p> </o:p></span></p> </div>
</body>
</html>
[Attachment #4 (--===============2700726110322807483==)]
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic