[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-general
Subject: Re: [HACKERS] Updated backup APIs for non-exclusive backups
From: Magnus Hagander <magnus () hagander ! net>
Date: 2016-03-31 12:21:28
Message-ID: CABUevEye=+q_CZ_sKB6uVMr12Rj9rZ9h_wtEykxJnDf0198xWA () mail ! gmail ! com
[Download RAW message or body]
On Thu, Mar 31, 2016 at 2:19 PM, Magnus Hagander <magnus@hagander.net>
wrote:
> On Thu, Mar 31, 2016 at 4:00 AM, David Steele <david@pgmasters.net> wrote:
>
>> On 3/30/16 4:18 AM, Magnus Hagander wrote:
>> >
>> > On Wed, Mar 30, 2016 at 4:10 AM, David Steele <david@pgmasters.net
>> > <mailto:david@pgmasters.net>> wrote:
>> >
>> > This certainly looks like it would work but it raises the barrier
>> for
>> > implementing backups by quite a lot. It's fine for backrest or
>> barman
>> > but it won't be pleasant for anyone who has home-grown scripts.
>> >
>> >
>> > How much does it really raise the bar, though?
>> >
>> > It would go from "copy all files and make damn sure you copy pg_control
>> > last, and rename it to pg_control.backup" to "take this binary blob you
>> > got from the server and write it to pg_control.backup"?
>> >
>> > Also, the target of these APIs is specifically the backup tools and not
>> > homewritten scripts.
>>
>> Then what would home-grown scripts use, the deprecated API that we know
>> has issues?
>>
>
> They should use either pg_basebackup/pg_receivexlog, or they should use a
> tool like backrest or barman.
>
>
>
>> > A simple shellscript will have trouble enough using
>> > it in the first place since it requires a persistent connection to the
>> > database.
>>
>> All that's required is to spawn a psql process. I'm no shell expert but
>> that's simple enough.
>>
>
> Oh, it's certainly doable, but you also need to come back and talk to it
> at a later time (to call stop backup), and do something useful with a
> multiline output. None of that is particularly easy. Certainly doable, but
> it becomes the wrong tool for the job.
>
>
>
>> > But those scripts are likely broken anyway.
>>
>> Yes, probably. Backup and especially archiving correctly are harder
>> than most people realize.
>>
>
> Exactly. Which is why we should discourage people from doing that, and
> instead point them to the tools that do the job right. This is the whole
> reason we're making this change in the first place.
>
>
>
>> > The main reason for Heikki to suggest this one over the other basic one
>> > is that it brings protection against the "backup script/program crashed
>> > halfway through but the user still tried to restore from that". They
>> will
>> > outright fail because there is no pg_control.backup in that case.
>>
>> But if we are going to make this complicated I'm not sure it's a big
>> deal just to require pg_control to be copied last. pgBackRest already
>> does that and Barman probably does, too.
>>
>
> It does (I did doublecheck that at some point).
>
>
>
>> I don't see "don't copy pg_control" and "copy pg_control last" as all
>> that different in terms of complexity.
>>
>> pgBackRest also *restores* pg_control last which I think is pretty
>> important and would not be addressed by this solution.
>>
>
> So maybe we should at least start that way. And just document that
> clearly, and then use the patch that we have right now?
>
> Except we add so it still returns the stoppoint() as well (which is
> returned from the current pg_stop_backup, but not in the new one).
>
Eh, nevermind. We do already return the stoppoint. I was looking at the
wrong version of the patch. So basically that's current version of patch,
but with proper docs of course.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
[Attachment #3 (text/html)]
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Mar 31, 2016 \
at 2:19 PM, Magnus Hagander <span dir="ltr"><<a href="mailto:magnus@hagander.net" \
target="_blank">magnus@hagander.net</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><span class="">On Thu, Mar 31, 2016 at 4:00 AM, David Steele \
<span dir="ltr"><<a href="mailto:david@pgmasters.net" \
target="_blank">david@pgmasters.net</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span>On 3/30/16 4:18 AM, Magnus Hagander wrote:<br> ><br>
> On Wed, Mar 30, 2016 at 4:10 AM, David Steele <<a \
href="mailto:david@pgmasters.net" target="_blank">david@pgmasters.net</a><br> \
</span><span>> <mailto:<a href="mailto:david@pgmasters.net" \
target="_blank">david@pgmasters.net</a>>> wrote:<br> ><br>
> This certainly looks like it would work but it raises the barrier for<br>
> implementing backups by quite a lot. It's fine for backrest or \
barman<br> > but it won't be pleasant for anyone who has home-grown \
scripts.<br> ><br>
><br>
> How much does it really raise the bar, though?<br>
><br>
> It would go from "copy all files and make damn sure you copy pg_control<br>
> last, and rename it to pg_control.backup" to "take this binary blob \
you<br> > got from the server and write it to pg_control.backup"?<br>
><br>
> Also, the target of these APIs is specifically the backup tools and not<br>
> homewritten scripts.<br>
<br>
</span>Then what would home-grown scripts use, the deprecated API that we know<br>
has issues?<br></blockquote><div><br></div></span><div>They should use either \
pg_basebackup/pg_receivexlog, or they should use a tool like backrest or \
barman.</div><span class=""><div><br></div><div> </div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span> > A simple shellscript will have trouble enough \
using<br> > it in the first place since it requires a persistent connection to \
the<br> > database.<br>
<br>
</span>All that's required is to spawn a psql process. I'm no shell expert \
but<br> that's simple enough.<br></blockquote><div><br></div></span><div>Oh, \
it's certainly doable, but you also need to come back and talk to it at a later \
time (to call stop backup), and do something useful with a multiline output. None of \
that is particularly easy. Certainly doable, but it becomes the wrong tool for the \
job.</div><span class=""><div><br></div><div> </div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span> > But \
those scripts are likely broken anyway.<br> <br>
</span>Yes, probably. Backup and especially archiving correctly are harder<br>
than most people realize.<br></blockquote><div><br></div></span><div>Exactly. Which \
is why we should discourage people from doing that, and instead point them to the \
tools that do the job right. This is the whole reason we're making this change in \
the first place.</div><span class=""><div><br></div><div> </div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span> > The main reason for Heikki to suggest this one \
over the other basic one<br> > is that it brings protection against the \
"backup script/program crashed<br> > halfway through but the user still tried \
to restore from that". They will<br> </span>> outright fail because there is \
no pg_control.backup in that case.<br> <br>
But if we are going to make this complicated I'm not sure it's a big<br>
deal just to require pg_control to be copied last. pgBackRest already<br>
does that and Barman probably does, \
too.<br></blockquote><div><br></div></span><div>It does (I did doublecheck that at \
some point).</div><span class=""><div><br></div><div> </div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> I don't see "don't copy pg_control" and \
"copy pg_control last" as all<br> that different in terms of \
complexity.<br> <br>
pgBackRest also *restores* pg_control last which I think is pretty<br>
important and would not be addressed by this \
solution.<br></blockquote><div><br></div></span><div>So maybe we should at least \
start that way. And just document that clearly, and then use the patch that we have \
right now?</div><div><br></div><div>Except we add so it still returns the stoppoint() \
as well (which is returned from the current pg_stop_backup, but not in the new \
one).</div></div></div></div></blockquote><div><br></div><div>Eh, nevermind. We do \
already return the stoppoint. I was looking at the wrong version of the patch. So \
basically that's current version of patch, but with proper docs of \
course.</div></div><div><br></div>-- <br><div class="gmail_signature"> Magnus \
Hagander<br> Me: <a href="http://www.hagander.net/" \
target="_blank">http://www.hagander.net/</a><br> Work: <a \
href="http://www.redpill-linpro.com/" \
target="_blank">http://www.redpill-linpro.com/</a></div> </div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic