[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-general
Subject: =?UTF-8?B?UmU6IFJFOiBbRVhURVJOQUxdIFJlOiBJbnNlcnRzIGFuZCBiYWQgcGVyZm9y?= =?UTF-8?B?bWFuY2U=?=
From: Ali . <mp.x () bk ! ru>
Date: 2021-11-30 6:01:20
Message-ID: 1638252080.149803539 () f520 ! i ! mail ! ru
[Download RAW message or body]
[Attachment #2 (text/plain)]
Ok, thanks
--
Sent from Mail.ru app for Android Wednesday, 24 November 2021, 11:28pm +03:00 from \
Godfrin, Philippe E philippe.godfrin@nov.com :
> The notion of COPY blocks and asynchronously is very interesting
>
> From: Gavin Roy < gavinr@aweber.com>
> Sent: Wednesday, November 24, 2021 1:50 PM
> To: Godfrin, Philippe E < Philippe.Godfrin@nov.com>
> Cc: pgsql-general@lists.postgresql.org
> Subject: [EXTERNAL] Re: Inserts and bad performance
>
>
>
>
> On Wed, Nov 24, 2021 at 2:15 PM Godfrin, Philippe E < Philippe.Godfrin@nov.com> \
> wrote:
> > Greetings
> > I am inserting a large number of rows, 5,10, 15 million. The python code commits \
> > every 5000 inserts. The table has partitioned children.
>
> On the Python client side, if you're using psycopg, you should consider using using \
> COPY instead of INSERT if you're not:
> https://www.psycopg.org/psycopg3/docs/basic/copy.html#copy
>
> And if using psycopg2, execute_batch might be of value:
>
> https://www.psycopg.org/docs/extras.html?highlight=insert#psycopg2.extras.execute_batch
>
> Regards,
>
> Gavin
>
>
[Attachment #3 (text/html)]
<HTML><BODY><p style="margin-top: 0px;" dir="ltr">Ok, thanks</p>
<div id="mail-app-auto-default-signature">
<p dir="ltr">--<br> Sent from Mail.ru app for Android</p>
</div>Wednesday, 24 November 2021, 11:28pm +03:00 from Godfrin, Philippe E <a \
href="mailto:philippe.godfrin@nov.com">philippe.godfrin@nov.com</a>:<br><br><blockquote \
id="mail-app-auto-quote" cite="16377856811068591122" style="border-left:1px solid \
#005FF9; margin:0px 0px 0px 10px; padding:0px 0px 0px 10px;"><div class="js-helper \
js-readmsg-msg">
<style type="text/css"></style>
<base target="_self" href="https://e.mail.ru/" />
<div id="style_16377856811068591122">
<div id="style_16377856811068591122_BODY"><div class="cl_284167">
<div class="WordSection1_mr_css_attr">
<p class="MsoNormal_mr_css_attr">The notion of COPY blocks and asynchronously is very \
interesting</p> <p class="MsoNormal_mr_css_attr"> </p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal_mr_css_attr"><b>From:</b> Gavin Roy <<a \
href="/compose?To=gavinr@aweber.com">gavinr@aweber.com</a>> <br> <b>Sent:</b> \
Wednesday, November 24, 2021 1:50 PM<br> <b>To:</b> Godfrin, Philippe E <<a \
href="/compose?To=Philippe.Godfrin@nov.com">Philippe.Godfrin@nov.com</a>><br> \
<b>Cc:</b> <a href="/compose?To=pgsql%2dgeneral@lists.postgresql.org">pgsql-general@lists.postgresql.org</a><br>
<b>Subject:</b> [EXTERNAL] Re: Inserts and bad performance</p>
</div>
<p class="MsoNormal_mr_css_attr"> </p>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt;display:none"> \
</span></p>
<div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
<div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt">On Wed, Nov 24, 2021 \
at 2:15 PM Godfrin, Philippe E <<a \
href="//e.mail.ru/compose/?mailto=mailto%3aPhilippe.Godfrin@nov.com" target="_blank" \
rel=" noopener noreferrer" >Philippe.Godfrin@nov.com</a>> wrote:</span></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<div>
<p class="MsoNormal_mr_css_attr" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Greetings</p> <p \
class="MsoNormal_mr_css_attr" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I am inserting a large \
number of rows, 5,10, 15 million. The python code commits every 5000 inserts. The \
table has partitioned children.</p> </div>
</div>
</blockquote>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt">On the Python client \
side, if you're using psycopg, you should consider using using COPY instead of INSERT \
if you're not:</span></p> </div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"><a \
href="https://www.psycopg.org/psycopg3/docs/basic/copy.html#copy" target="_blank" \
rel=" noopener noreferrer" \
>https://www.psycopg.org/psycopg3/docs/basic/copy.html#copy</a></span></p> </div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt">And if using \
psycopg2, execute_batch might be of value:</span></p> </div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"><a \
href="https://www.psycopg.org/docs/extras.html?highlight=insert#psycopg2.extras.execute_batch" \
target="_blank" rel=" noopener noreferrer" \
>https://www.psycopg.org/docs/extras.html?highlight=insert#psycopg2.extras.execute_batch</a></span></p>
>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt">Regards,</span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt">Gavin</span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
<div>
<p class="MsoNormal_mr_css_attr"><span style="font-size:10.0pt"> </span></p>
</div>
</div>
</div>
</div>
</div></div>
</div>
</div></blockquote></BODY></HTML>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic