[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-general
Subject: Re: [HACKERS] Relation extension scalability
From: Dilip Kumar <dilipbalaut () gmail ! com>
Date: 2016-03-31 13:15:45
Message-ID: CAFiTN-s0p79Oho4XQx6iv_U02BXiNNDhUjurPo9kaV_JUst5OA () mail ! gmail ! com
[Download RAW message or body]
On Thu, Mar 31, 2016 at 4:28 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Yeah, kind of. But obviously if we could make the limit smaller
> without hurting performance, that would be better.
>
> Per my note yesterday about performance degradation with parallel
> COPY, I wasn't able to demonstrate that this patch gives a consistent
> performance benefit on hydra - the best result I got was speeding up a
> 9.5 minute load to 8 minutes where linear scalability would have been
> 2 minutes. And I found cases where it was actually slower with the
> patch. Now maybe hydra is just a crap machine, but I'm feeling
> nervous.
>
I see the performance gain when either
"*complete data is in SSD*"
or *"data on MD and WAL on SSD"*
or *unlogged table*.
What machines did you use to test this? Have you tested really large
> data loads, like many MB or even GB of data?
>
With INSERT Script within 2 mins run data size is 18GB I am running 5-10
Mins means at least 85GB data.
(Inserts 1000 1KB tuples in each transactions)
With COPY Script within 2 mins run data size is 23GB and I am running 5-10
Mins means at least 100GB data.
(Inserts 10000 tuples in each transactions (tuple size is 1byte to 5 bytes)
Machine Details
-----------------------
I tested in 8 socket NUMA machine with 64 core.
Machine Details:
----------------------
[dilip.kumar@cthulhu ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 8
NUMA node(s): 8
Vendor ID: GenuineIntel
CPU family: 6
Model: 47
Model name: Intel(R) Xeon(R) CPU E7- 8830 @ 2.13GHz
Stepping: 2
CPU MHz: 1064.000
BogoMIPS: 4266.62
If you need some more information please let me know ?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
[Attachment #3 (text/html)]
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Mar 31, 2016 \
at 4:28 PM, Robert Haas <span dir="ltr"><<a href="mailto:robertmhaas@gmail.com" \
target="_blank">robertmhaas@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
style="overflow:hidden">Yeah, kind of. But obviously if we could make the limit \
smaller<br> without hurting performance, that would be better.<br>
<br>
Per my note yesterday about performance degradation with parallel<br>
COPY, I wasn't able to demonstrate that this patch gives a consistent<br>
performance benefit on hydra - the best result I got was speeding up a<br>
9.5 minute load to 8 minutes where linear scalability would have been<br>
2 minutes. And I found cases where it was actually slower with the<br>
patch. Now maybe hydra is just a crap machine, but I'm feeling<br>
nervous.<br></div></blockquote><div><br></div><div><br></div><div>I see the \
performance gain when either</div><div> "<b>complete data is in SSD</b>" \
</div><div>or <b>"data on MD and WAL on SSD"</b> </div><div>or <b>unlogged \
table</b>.</div><div><br></div><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
style="overflow:hidden">What machines did you use to test this? Have you tested \
really large<br> data loads, like many MB or even GB of \
data?</div></blockquote></div><div class="gmail_extra"><br></div><div \
class="gmail_extra"><br></div><div class="gmail_extra">With INSERT Script within 2 \
mins run data size is 18GB I am running 5-10 Mins means at least 85GB \
data.</div><div class="gmail_extra">(Inserts 1000 1KB tuples in each \
transactions)</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div \
class="gmail_extra">With COPY Script within 2 mins run data size is 23GB and I am \
running 5-10 Mins means at least 100GB data.</div><div class="gmail_extra">(Inserts \
10000 tuples in each transactions (tuple size is 1byte to 5 bytes)</div><div \
class="gmail_extra"><div><br></div><div>Machine \
Details</div><div>-----------------------</div></div></div><div class="gmail_extra">I \
tested in 8 socket NUMA machine with 64 core.</div><div class="gmail_extra"><div \
class="gmail_extra">Machine Details:</div><div \
class="gmail_extra">----------------------</div><div \
class="gmail_extra">[dilip.kumar@cthulhu ~]$ lscpu</div><div \
class="gmail_extra">Architecture: x86_64</div><div \
class="gmail_extra">CPU op-mode(s): 32-bit, 64-bit</div><div \
class="gmail_extra">Byte Order: Little Endian</div><div \
class="gmail_extra">CPU(s): 128</div><div \
class="gmail_extra">On-line CPU(s) list: 0-127</div><div \
class="gmail_extra">Thread(s) per core: 2</div><div class="gmail_extra">Core(s) \
per socket: 8</div><div class="gmail_extra">Socket(s): \
8</div><div class="gmail_extra">NUMA node(s): 8</div><div \
class="gmail_extra">Vendor ID: GenuineIntel</div><div \
class="gmail_extra">CPU family: 6</div><div \
class="gmail_extra">Model: 47</div><div \
class="gmail_extra">Model name: Intel(R) Xeon(R) CPU E7- 8830 @ \
2.13GHz</div><div class="gmail_extra">Stepping: 2</div><div \
class="gmail_extra">CPU MHz: 1064.000</div><div \
class="gmail_extra">BogoMIPS: \
4266.62</div><div><br></div><div><br></div><div>If you need some more information \
please let me know ?</div></div><div class="gmail_extra"><br></div>-- <br><div><div \
dir="ltr"><span style="color:rgb(80,0,80);font-size:12.8px">Regards,</span><br \
style="color:rgb(80,0,80);font-size:12.8px"><span \
style="color:rgb(80,0,80);font-size:12.8px">Dilip Kumar</span><br \
style="color:rgb(80,0,80);font-size:12.8px"><span \
style="color:rgb(80,0,80);font-size:12.8px">EnterpriseDB: </span><a \
href="http://www.enterprisedb.com/" style="color:rgb(17,85,204);font-size:12.8px" \
target="_blank">http://www.enterprisedb.com</a><br></div></div> </div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic