[prev in list] [next in list] [prev in thread] [next in thread] 

List:       pgsql-hackers
Subject:    Re: [HACKERS] Parallel COPY FROM execution
From:       Pavel Stehule <pavel.stehule () gmail ! com>
Date:       2017-06-30 13:47:49
Message-ID: CAFj8pRDm3OMxnPmBQGef77_Aj63x-_d8Dp7=UYRzVBDVKV1=Wg () mail ! gmail ! com
[Download RAW message or body]

2017-06-30 15:45 GMT+02:00 Pavel Stehule <pavel.stehule@gmail.com>:

>
>
> 2017-06-30 15:42 GMT+02:00 Alex K <kondratov.aleksey@gmail.com>:
>
>> On Fri, Jun 30, 2017 at 3:35 PM, Pavel Stehule <pavel.stehule@gmail.com>
>> wrote:
>> >
>> >
>> > 2017-06-30 14:23 GMT+02:00 Alex K <kondratov.aleksey@gmail.com>:
>> >>
>> >> Thus, it results in a ~60% performance boost per each x2
>> multiplication of
>> >> parallel processes, which is consistent with the initial estimation.
>> >>
>> >
>> > the important use case is big table with lot of indexes. Did you test
>> > similar case?
>>
>> Not yet, I will try it, thank you for a suggestion. But how much is it
>> 'big table' and 'lot of indexes' in numbers approximately?
>>
>
> the size is about 1/3 RAM size, 60 columns, 30 indexes
>

maybe some variants can be interesting .. 1/30 RAM, 1/20 RAM, 1/10 RAM, 1/3
RAM

and a) when bottleneck is IO, b) when bottleneck is CPU

Pavel


>
> Regards
>
> Pavel
>
>
>>
>> Also, index updates and constraint checks performance are what I cannot
>> control during COPY execution, so probably I have not to care too much
>> about that. But of course, it is interesting, how does COPY perform in
>> that case.
>>
>>
>> Alexey
>>
>
>

[Attachment #3 (text/html)]

<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2017-06-30 \
15:45 GMT+02:00 Pavel Stehule <span dir="ltr">&lt;<a \
href="mailto:pavel.stehule@gmail.com" \
target="_blank">pavel.stehule@gmail.com</a>&gt;</span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div \
class="gmail_quote"><span class="">2017-06-30 15:42 GMT+02:00 Alex K <span \
dir="ltr">&lt;<a href="mailto:kondratov.aleksey@gmail.com" \
target="_blank">kondratov.aleksey@gmail.com</a>&gt;</span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span>On Fri, Jun 30, 2017 at 3:35 PM, Pavel Stehule &lt;<a \
href="mailto:pavel.stehule@gmail.com" target="_blank">pavel.stehule@gmail.com</a>&gt; \
wrote:<br> &gt;<br>
&gt;<br>
&gt; 2017-06-30 14:23 GMT+02:00 Alex K &lt;<a \
href="mailto:kondratov.aleksey@gmail.com" \
target="_blank">kondratov.aleksey@gmail.com</a>&gt;:<br> &gt;&gt;<br>
</span><span>&gt;&gt; Thus, it results in a ~60% performance boost per each x2 \
multiplication of<br> &gt;&gt; parallel processes, which is consistent with the \
initial estimation.<br> &gt;&gt;<br>
&gt;<br>
&gt; the important use case is big table with lot of indexes. Did you test<br>
&gt; similar case?<br>
<br>
</span>Not yet, I will try it, thank you for a suggestion. But how much is it<br>
&#39;big table&#39; and &#39;lot of indexes&#39; in numbers \
approximately?<br></blockquote><div><br></div></span><div>the size is about 1/3 RAM \
size, 60 columns, 30 \
indexes</div></div></div></div></blockquote><div><br></div><div>maybe some variants \
can be interesting .. 1/30 RAM, 1/20 RAM, 1/10 RAM, 1/3 RAM  \
</div><div><br></div><div>and a) when bottleneck is IO, b) when bottleneck is \
CPU</div><div><br></div><div>Pavel</div><div>  <br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><div><br></div><div>Regards</div><span class="HOEnZb"><font \
color="#888888"><div><br></div><div>Pavel</div></font></span><span class=""><div>  \
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <br>
Also, index updates and constraint checks performance are what I cannot<br>
control during COPY execution, so probably I have not to care too much<br>
about that. But of course, it is interesting, how does COPY perform in<br>
that case.<br>
<br>
<br>
Alexey<br>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>



[prev in list] [next in list] [prev in thread] [next in thread]