[prev in list] [next in list] [prev in thread] [next in thread]
List: bacula-users
Subject: Re: [Bacula-users] Cloud s3 error
From: Ana_EmÃlia_M._Arruda <emiliaarruda () gmail ! com>
Date: 2023-01-12 20:18:23
Message-ID: CAAh-UZbNj5ZoQUAwmCw2YjZPys0iYRVFJGaLpVscp156p8nUxA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
Hello Ivan, Hello Chris,
Would it be possible this issue is related to having object lock configured
in this bucket?
-
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock. For more
information about Amazon S3 Object Lock, see Amazon S3 Object Lock
Overview
<https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html>
in
the *Amazon S3 User Guide*.
It is possible this happens when Bacula tries to reuse a volume before its
retention period set by using amazon s3 object lock. Do you think this can
be the case?
Maybe if you have object lock configured in the bucket, you may set the
VolumeRetention = 999 years, Recycle = No and AutoPrune = No. This should
avoid volumes to get recycled.
If this is not the case, which Bacula version are you using? Does it happen
with all part files or only some of them?
Best regards,
Ana
On Fri, Jan 6, 2023 at 12:46 PM Chris Wilkinson <winstonian36@gmail.com>
wrote:
> I posted a similar question to the group late last year but I had no
> response. The issue for me is intermittent and I've found no resolution to
> it. I just rerun the failed jobs and delete them when it occurs. That often
> leaves some orphaned volumes on S3 or the cache that I have to clean up
> manually with a couple of bash scripts.
>
> Regards
> Chris Wilkinson
>
> On Fri, 6 Jan 2023, 10:30 am Ivan Villalba via Bacula-users, <
> bacula-users@lists.sourceforge.net> wrote:
>
> > Hi there,
> >
> > I'm getting errors on the jobs configured with cloud s3:
> >
> > 06-Jan 09:03 mainbackupserver-sd JobId 4030: Error:
> > serverJob-CopyToS3-0781/part.16 state=error retry=1/10 size=2.064 MB
> > duration=0s msg= S3_put_object ERR=Content-MD5 OR x-amz-checksum- HTTP
> > header is required for Put Object requests with Object Lock parameters CURL
> > Effective URL: https://xxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16
> > CURL Effective URL:
> > https://xxxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16 RequestId :
> > KPQE1MJPVAK3XK6F HostId :
> > m8sQYlY4qJLDDThwKeDxnOWyksMR7bR1HJiukDmqf29ahPC6yc4x0LT0VWpmBfhObotCdX4T36M=
> >
> > I have the same error on all the jobs that uploads to s3 cloud.
> >
> > The thing is that it worked eventually, at least I have some uploaeds on
> > the s3 bucket from the earlier tests jobs, but it's not working anymore,
> > even I've not modified the configurations since then.
> >
> > What am I doing wrong?
> >
> > thanks in advance.
> >
> >
> > Configurations (sensitive data hidden):
> > SD:
> > Device {
> > Name = "backupserver-backups"
> > Device Type = "Cloud"
> > Cloud = "S3-cloud-eu-west-1"
> > Maximum Part Size = 2M
> > Maximum File Size = 2M
> > Media Type = Cloud
> > Archive Device = /backup/bacula-storage
> > LabelMedia = yes
> > Random Access = yes
> > AutomaticMount = yes
> > RemovableMedia = no
> > AlwaysOpen = no
> > }
> >
> > s3:
> > Cloud {
> > Name = "S3-cloud-eu-west-1"
> > Driver = "S3"
> > HostName = "s3.eu-west-1.amazonaws.com"
> > BucketName = "xxxx"
> > AccessKey = "xxxx"
> > SecretKey = "xxxx"
> > Protocol = HTTPS
> > UriStyle = "VirtualHost"
> > Truncate Cache = "AfterUpload"
> > Upload = "EachPart"
> > Region = "eu-west-1"
> > MaximumUploadBandwidth = 10MB/s
> > }
> >
> > Dir's storage:
> >
> > #CopyToS3
> > Storage {
> > Name = "CloudStorageS3"
> > Address = "xxxx"
> > SDPort = 9103
> > Password = "xxxx"
> > Device = "backupserver-backups"
> > Media Type = Cloud
> > Maximum Concurrent Jobs = 5
> > Heartbeat Interval = 10
> > }
> >
> >
> > --
> > Ivan Villalba
> > SysOps
> >
> > <https://www.marfeel.com/>
> >
> > <https://www.marfeel.com/>
> > [image: Inline images 4]
> > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-google-certified-publishing-partner>
> > [image: Inline images 3]
> > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-facebook-instant-articles-partner>
> >
> >
> > Avda. Josep Tarradellas 20-30, 4th Floor
> >
> > 08029 Barcelona, Spain
> >
> > ES: (34) 93 178 59 50
> > <%2834%29%2093%20178%2059%2050%20%C2%A0ext.%20107>
> > US: (1) 917-341-2540 <%281%29%20917-341-2540%20ext.%20107>
> > _______________________________________________
> > Bacula-users mailing list
> > Bacula-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
> _______________________________________________
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
[Attachment #7 (text/html)]
<div dir="ltr"><div dir="ltr"><div class="gmail_default" \
style="font-family:tahoma,sans-serif">Hello Ivan, Hello Chris,</div><div \
class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif">Would it be possible this \
issue is related to having object lock configured in this bucket?</div><div \
class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif"><ul \
class="gmail-itemizedlist" type="disc" style="margin:0px;padding:0px 0px 0px \
25px;color:rgb(22,25,31);font-family:"Amazon \
Ember";font-size:16px;background-color:rgba(241,250,255,0.8)"><li \
class="gmail-listitem" \
style="font-size:1em;line-height:1.5em;padding-left:5px;padding-top:5px"><p \
style="font-size:1em;line-height:1.5em;margin:5px 0px 0px;padding:0px">The <code \
class="gmail-code" style="font-family:Monaco,Menlo,Consolas,"Courier \
Prime",Courier,"Courier \
New",monospace;font-size:1em;background-color:rgb(227,237,243);border-radius:2px;padding:0px \
3px">Content-MD5</code> header is required for any request to upload an object with \
a retention period configured using Amazon S3 Object Lock. For more information about \
Amazon S3 Object Lock, see <a \
href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html" \
style="background-color:transparent">Amazon S3 Object Lock Overview</a> in the \
<em>Amazon S3 User Guide</em>.</p></li></ul></div></div><div><br></div><div><div \
class="gmail_default" style="font-family:tahoma,sans-serif">It is possible this \
happens when Bacula tries to reuse a volume before its retention period set by using \
amazon s3 object lock. Do you think this can be the case?</div><div \
class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div \
class="gmail_default" style="font-family:tahoma,sans-serif">Maybe if you have object \
lock configured in the bucket, you may set the VolumeRetention = 999 years, Recycle = \
No and AutoPrune = No. This should avoid volumes to get \
recycled.</div><br></div><div><div class="gmail_default" \
style="font-family:tahoma,sans-serif">If this is not the case, which Bacula version \
are you using? Does it happen with all part files or only some of \
them?</div><br></div><div><div class="gmail_default" \
style="font-family:tahoma,sans-serif">Best regards,</div><div class="gmail_default" \
style="font-family:tahoma,sans-serif">Ana</div><br></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 12:46 PM \
Chris Wilkinson <<a \
href="mailto:winstonian36@gmail.com">winstonian36@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I \
posted a similar question to the group late last year but I had no response. The \
issue for me is intermittent and I've found no resolution to it. I just rerun the \
failed jobs and delete them when it occurs. That often leaves some orphaned volumes \
on S3 or the cache that I have to clean up manually with a couple of bash scripts. \
<br><br><div>Regards<br>Chris Wilkinson</div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Fri, 6 Jan 2023, 10:30 am Ivan Villalba via \
Bacula-users, <<a href="mailto:bacula-users@lists.sourceforge.net" \
target="_blank">bacula-users@lists.sourceforge.net</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi \
there, <div><br></div><div>I'm getting errors on the jobs configured with cloud \
s3: </div><div><br></div><div>06-Jan 09:03 mainbackupserver-sd JobId 4030: Error: \
serverJob-CopyToS3-0781/part.16 state=error retry=1/10 size=2.064 MB \
duration=0s msg= S3_put_object ERR=Content-MD5 OR x-amz-checksum- HTTP header is \
required for Put Object requests with Object Lock parameters CURL Effective URL: <a \
href="https://xxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16" rel="noreferrer" \
target="_blank">https://xxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16</a> CURL \
Effective URL: <a href="https://xxxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16" \
rel="noreferrer" target="_blank">https://xxxxxxx.s3.eu-west-1.amazonaws.com/xxxxxxx/part.16</a> \
RequestId : KPQE1MJPVAK3XK6F HostId : \
m8sQYlY4qJLDDThwKeDxnOWyksMR7bR1HJiukDmqf29ahPC6yc4x0LT0VWpmBfhObotCdX4T36M=</div><div><br></div><div>I \
have the same error on all the jobs that uploads to s3 \
cloud.</div><div><br></div><div>The thing is that it worked eventually, at least I \
have some uploaeds on the s3 bucket from the earlier tests jobs, but it's not \
working anymore, even I've not modified the configurations since \
then.</div><div><br></div><div>What am I doing wrong?</div><div><br></div><div>thanks \
in advance.</div><div><br></div><div><br></div><div>Configurations (sensitive data \
hidden): </div><div>SD: </div><div>Device {<br> Name = \
"backupserver-backups"<br> Device Type = "Cloud"<br> Cloud = \
"S3-cloud-eu-west-1"<br> Maximum Part Size = 2M<br> Maximum File Size = \
2M<br> Media Type = Cloud<br> Archive Device = /backup/bacula-storage<br> \
LabelMedia = yes<br> Random Access = yes<br> AutomaticMount = yes<br> \
RemovableMedia = no<br> AlwaysOpen = \
no<br>}<br><div><br></div><div>s3:</div><div>Cloud {<br> Name = \
"S3-cloud-eu-west-1"<br> Driver = "S3"<br> HostName = \
"<a href="http://s3.eu-west-1.amazonaws.com" rel="noreferrer" \
target="_blank">s3.eu-west-1.amazonaws.com</a>"<br> BucketName = \
"xxxx"<br> AccessKey = "xxxx"<br> SecretKey = \
"xxxx"<br> Protocol = HTTPS<br> UriStyle = "VirtualHost"<br> \
Truncate Cache = "AfterUpload"<br> Upload = "EachPart"<br> \
Region = "eu-west-1"<br> MaximumUploadBandwidth = \
10MB/s<br>}<br></div><div><br></div><div>Dir's storage: \
</div><div><br>#CopyToS3<br>Storage {<br> Name = "CloudStorageS3"<br> \
Address = "xxxx"<br> SDPort = 9103<br> Password = "xxxx"<br> \
Device = "backupserver-backups"<br> Media Type = Cloud<br> Maximum \
Concurrent Jobs = 5<br> Heartbeat Interval = \
10<br>}<br></div><div><br></div><div><br></div>-- <br><div dir="ltr"><div \
dir="ltr"><div style="padding-top:0px;padding-right:0px;padding-left:0px;display:flex;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div \
style="margin:0px;padding:0px 0px 20px;width:651px"><div><div \
style="font-size:12.8px;margin:8px 0px 0px;padding:0px"><div><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
style="font-size:12.8px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div style="font-size:12.8px"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div><div style="font-size:12.8px"><span \
style="color:rgb(102,102,102);font-size:12.8px">Ivan Villalba</span><br></div><div \
style="font-size:12.8px"><span \
style="color:rgb(102,102,102);font-size:12.8px">SysOps</span></div><div \
style="font-size:12.8px"><span \
style="color:rgb(102,102,102);font-size:12.8px"><br></span></div><div \
style="font-size:12.8px"><span style="color:rgb(102,102,102);font-size:12.8px"><a \
href="https://www.marfeel.com/" rel="noreferrer" target="_blank"><img \
src="https://docs.google.com/uc?export=download&id=1vFglQFs-qNLCd-y2VDSle8mmT_EPeeM-&revid=0B1j-XpuPCmnAa3krWnN2Y3F0L0JTaGpJQjY4NEhaY2hFNzRVPQ" \
width="200" height="47"></a><br></span></div></div><div dir="ltr"><div \
style="font-size:12.8px;color:rgb(80,0,80)"><div dir="ltr" \
style="margin-left:0pt"><div style="color:rgb(34,34,34);font-size:12.8px"><div \
style="color:rgb(136,136,136);font-family:arial;font-size:small"><div><div \
style="font-family:arial,sans-serif;font-size:12.8px;color:rgb(80,0,80)"><a \
href="https://www.marfeel.com/" rel="noreferrer" target="_blank"><br></a></div><div \
style="font-family:arial,sans-serif;font-size:12.8px;color:rgb(80,0,80)"><a \
href="https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-google-certified-publishing-partner" \
rel="noreferrer" target="_blank"><img src="cid:ii_15c0d318c953c6b8" alt="Inline \
images 4" width="113" height="44" style="color: rgb(34, 34, 34); font-size: small; \
margin-right: 0px;"></a><span style="color:rgb(34,34,34);font-size:small"> </span><a \
href="https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-facebook-instant-articles-partner" \
rel="noreferrer" target="_blank"><img src="cid:ii_15c0d3183f47c265" alt="Inline \
images 3" width="113" height="44" style="color: rgb(34, 34, 34); font-size: small; \
margin-right: 0px;"></a><br></div><div \
style="font-family:arial,sans-serif;font-size:12.8px;color:rgb(80,0,80)"><div \
dir="ltr" style="margin-left:0pt"><table \
style="border:none;border-collapse:collapse"><colgroup><col width="24"><col \
width="571"></colgroup><tbody><tr style="height:26px"><td colspan="2" \
style="border:0px solid rgb(0,0,0);vertical-align:top;padding:0px"><table \
style="font-size:12.8px;border:none;border-collapse:collapse"><colgroup><col \
width="24"><col width="571"></colgroup><tbody><tr style="height:26px"><td colspan="2" \
style="border:0px solid rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" \
style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span \
style="font-size:12.8px"></span><br></p><p dir="ltr" \
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span \
style="vertical-align:baseline"><font color="#666666" face="tahoma, sans-serif"><span \
style="font-size:12.6667px;white-space:pre-wrap">Avda. Josep Tarradellas \
20-30</span></font><font face="tahoma, sans-serif" color="#666666" \
style="font-size:12.6667px;white-space:pre-wrap">, 4th Floor</font></span></p><p \
dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span \
style="font-size:12.6667px;vertical-align:baseline;white-space:pre-wrap"><font \
face="tahoma, sans-serif" color="#666666">08029 Barcelona, \
Spain</font></span></p></td></tr></tbody></table></td></tr></tbody></table></div></div></div><div><br></div></div></div><div \
style="color:rgb(34,34,34);font-size:12.8px"><div><div style="font-size:12.8px"><div \
style="color:rgb(80,0,80);font-size:12.8px"><font color="#999999">ES: <a \
href="tel:%2834%29%2093%20178%2059%2050%20%C2%A0ext.%20107" value="+34931785950" \
rel="noreferrer" target="_blank">(34) 93 178 59 50<br></a></font></div><div \
style="color:rgb(80,0,80);font-size:12.8px"><font color="#999999">US: <a \
href="tel:%281%29%20917-341-2540%20ext.%20107" value="+19173412540" rel="noreferrer" \
target="_blank">(1) 917-341-2540</a></font></div></div></div></div></div></div></div>< \
/div></div></div></div></div></div></div></div></div></div></div></div></div></div></d \
iv></div></div></div></div></div></div></div></div></div></div></div></div></div></div \
></div></div></div></div></div></div></div></div></div></div></div></div></div></div>< \
> /div></div></div></div><div></div><div></div></div></div><div></div></div></div><div></div></div><div \
> style="font-size:12.8px;border-bottom-left-radius:0px;border-bottom-right-radius:0px \
> ;border-top:none;background-image:initial;background-position:initial;background-repeat:initial;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><div><div \
> style="padding:16px 0px;border-top:none"></div></div></div></div></div></div></div>
_______________________________________________<br>
Bacula-users mailing list<br>
<a href="mailto:Bacula-users@lists.sourceforge.net" rel="noreferrer" \
target="_blank">Bacula-users@lists.sourceforge.net</a><br> <a \
href="https://lists.sourceforge.net/lists/listinfo/bacula-users" rel="noreferrer \
noreferrer" target="_blank">https://lists.sourceforge.net/lists/listinfo/bacula-users</a><br>
</blockquote></div>
_______________________________________________<br>
Bacula-users mailing list<br>
<a href="mailto:Bacula-users@lists.sourceforge.net" \
target="_blank">Bacula-users@lists.sourceforge.net</a><br> <a \
href="https://lists.sourceforge.net/lists/listinfo/bacula-users" rel="noreferrer" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/bacula-users</a><br> \
</blockquote></div></div>
--00000000000047779f05f216d26c--
["facebook2.png" (image/png)]
["google-3.png" (image/png)]
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic