[prev in list] [next in list] [prev in thread] [next in thread]
List: pgsql-performance
Subject: Re: [PERFORM] Substantial different index use between 9.5 and 9.6
From: Bill Measday <bill () measday ! com>
Date: 2016-12-03 20:42:02
Message-ID: 6586dbf1-3c1e-e7b1-bdae-c2a3d2914f49 () measday ! com
[Download RAW message or body]
Seems to be a replicable issue in PostGis - ticket raised at their end,
so I'll wait for a resolution of the root cause.
Thanks for your help/thoughts.
Rgds
Bill
On 3/12/2016 2:41 AM, Daniel Blanch Bataller wrote:
> ANALYZE takes samples at random, so statistics might be different even
> with same postgresql version:
>
> https://www.postgresql.org/docs/current/static/sql-analyze.html
>
> For large tables, ANALYZE takes a random sample of the table
> contents, rather than examining every row. This allows even very
> large tables to be analyzed in a small amount of time. Note,
> however, that the statistics are only approximate, and will change
> slightly each time ANALYZE is run, even if the actual table
> contents did not change. This might result in small changes in the
> planner's estimated costs shown by EXPLAIN
> <https://www.postgresql.org/docs/current/static/sql-explain.html>.
> In rare situations, this non-determinism will cause the planner's
> choices of query plans to change after ANALYZE is run. To avoid
> this, raise the amount of statistics collected by ANALYZE, as
> described below.
>
>
> Though, having that round (x 1000) difference, my bet is that you have
> different statistics target whether on database, table or columns, see:
>
> The extent of analysis can be controlled by adjusting the
> default_statistics_target
> <https://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET> \
> configuration variable, or on a column-by-column basis by setting the per-column
> statistics target with ALTER TABLE ... ALTER COLUMN ... SET
> STATISTICS (see ALTER TABLE
> <https://www.postgresql.org/docs/current/static/sql-altertable.html>).
> The target value sets the maximum number of entries in the
> most-common-value list and the maximum number of bins in the
> histogram. The default target value is 100, but this can be
> adjusted up or down to trade off accuracy of planner estimates
> against the time taken for ANALYZE and the amount of space
> occupied in pg_statistic. In particular, setting the statistics
> target to zero disables collection of statistics for that column.
> It might be useful to do that for columns that are never used as
> part of the WHERE, GROUP BY, or ORDER BY clauses of queries, since
> the planner will have no use for statistics on such columns.
>
>
> Here is some help on how to see statistics per column:
>
> http://stackoverflow.com/questions/15034622/check-statistics-targets-in-postgresql
>
> Check if this is the case.
>
>
>
>
>
>
> > El 2 dic 2016, a las 1:26, Bill Measday <bill@measday.com
> > <mailto:bill@measday.com>> escribió:
> >
> > Thanks Tom.
> >
> > First, this wasn't a migration but new db loaded from scratch (if
> > that matters).
> >
> > As per the end of the original post "I have vacuum analysed both
> > tables". I assume this is what you meant?
> >
> > My gut feel was that it isn't a postgis issue since the third example
> > I gave uses the index, but I will take it up with them too.
> >
> > Rgds
> >
> >
> > Bill
> >
> > On 2/12/2016 10:48 AM, Tom Lane wrote:
> > > Bill Measday <bill@measday.com <mailto:bill@measday.com>> writes:
> > > > Substantial different index use between 9.5 and 9.6
> > > Maybe you missed an ANALYZE after migrating? The plan difference
> > > seems to be due to a vast difference in rowcount estimate for the
> > > m_elevations condition:
> > >
> > > > -> Bitmap Heap Scan on m_elevations e
> > > > (cost=282802.21..37401439.43 rows=3512160 width=8)
> > > > -> Seq Scan on m_elevations e
> > > > (cost=10000000000.00..13296950520.12 rows=3512159563 width=8)
> > > If you don't know where that factor-of-1000 came from, maybe take
> > > it up with the postgis folk. It'd mostly be coming out of their
> > > selectivity estimation routines.
> > >
> > > regards, tom lane
> >
> >
> >
> > --
> > Sent via pgsql-performance mailing list
> > (pgsql-performance@postgresql.org
> > <mailto:pgsql-performance@postgresql.org>)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-performance
>
[Attachment #3 (text/html)]
<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Seems to be a replicable issue in PostGis - ticket raised at
their end, so I'll wait for a resolution of the root cause.</p>
<p><br>
</p>
<p>Thanks for your help/thoughts.</p>
<p><br>
Rgds</p>
<p><br>
Bill<br>
</p>
<br>
<div class="moz-cite-prefix">On 3/12/2016 2:41 AM, Daniel Blanch
Bataller wrote:<br>
</div>
<blockquote
cite="mid:AB2D19D3-2900-453A-AC90-ECB5B3EF4205@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="">ANALYZE takes samples at random, so statistics might
be different even with same postgresql version:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><a moz-do-not-send="true"
href="https://www.postgresql.org/docs/current/static/sql-analyze.html"
class="">https://www.postgresql.org/docs/current/static/sql-analyze.html</a></div>
<div class=""><br class="">
</div>
<div class=""><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class="">For large tables, </span><tt
class="COMMAND" style="font-size: 1.3em;">ANALYZE</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> takes a random sample of the table contents,
rather than examining every row. This allows even very large
tables to be analyzed in a small amount of time. Note,
however, that the statistics are only approximate, and will
change slightly each time </span><tt class="COMMAND"
style="font-size: 1.3em;">ANALYZE</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> is run, even if the actual table contents did not
change. This might result in small changes in the planner's
estimated costs shown by </span><a moz-do-not-send="true"
href="https://www.postgresql.org/docs/current/static/sql-explain.html"
style="color: rgb(0, 78, 102); font-family: verdana,
sans-serif; font-size: 12.15999984741211px;" class="">EXPLAIN</a><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class="">. In rare situations, this non-determinism will
cause the planner's choices of query plans to change after </span><tt
class="COMMAND" style="font-size: 1.3em;">ANALYZE</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> is run. To avoid this, raise the amount of
statistics collected by </span><tt class="COMMAND"
style="font-size: 1.3em;">ANALYZE</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class="">, as described below.</span></div>
</blockquote>
<div class=""><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 12.15999984741211px;
background-color: rgb(255, 255, 255);" class="">Though, having
that round (x 1000) difference, my bet is that you have
different statistics target whether on database, table or
columns, see:</span></div>
<div class=""><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class=""><br class="">
</span></div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class="">The extent of analysis can be
controlled by adjusting the </span><a moz-do-not-send="true"
href="https://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET"
style="color: rgb(0, 78, 102); font-family: verdana,
sans-serif; font-size: 12.15999984741211px;" \
class="">default_statistics_target</a><span style="font-family: verdana, sans-serif; \
font-size: 12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> configuration variable, or on a column-by-column
basis by setting the per-column statistics target with </span><tt
class="COMMAND" style="font-size: 1.3em;">ALTER TABLE ...
ALTER COLUMN ... SET STATISTICS</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> (see </span><a moz-do-not-send="true"
href="https://www.postgresql.org/docs/current/static/sql-altertable.html"
style="color: rgb(0, 78, 102); font-family: verdana,
sans-serif; font-size: 12.15999984741211px;" class="">ALTER
TABLE</a><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class="">). The target value sets the maximum
number of entries in the most-common-value list and the
maximum number of bins in the histogram. The default target
value is 100, but this can be adjusted up or down to trade
off accuracy of planner estimates against the time taken
for </span><tt class="COMMAND" style="font-size: \
1.3em;">ANALYZE</tt><span style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class=""> and the amount of space occupied in </span><tt
class="LITERAL" style="font-size: 1.3em;">pg_statistic</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class="">. In particular, setting the statistics target to
zero disables collection of statistics for that column. It
might be useful to do that for columns that are never used
as part of the </span><tt class="LITERAL" style="font-size:
1.3em;">WHERE</tt><span style="font-family: verdana,
sans-serif; font-size: 12.15999984741211px;
background-color: rgb(255, 255, 255);" class="">, </span><tt
class="LITERAL" style="font-size: 1.3em;">GROUP BY</tt><span
style="font-family: verdana, sans-serif; font-size:
12.15999984741211px; background-color: rgb(255, 255, 255);"
class="">, or </span><tt class="LITERAL" style="font-size:
1.3em;">ORDER BY</tt><span style="font-family: verdana,
sans-serif; font-size: 12.15999984741211px;
background-color: rgb(255, 255, 255);" class=""> clauses of
queries, since the planner will have no use for statistics
on such columns.</span></div>
</blockquote>
<div class=""><span style="font-family: verdana, sans-serif;
font-size: 12.15999984741211px; background-color: rgb(255,
255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 12.15999984741211px;
background-color: rgb(255, 255, 255);" class="">Here is some
help on how to see statistics per column:</span></div>
<div class=""><font class="" face="verdana, sans-serif"><span
style="font-size: 12.15999984741211px; background-color:
rgb(255, 255, 255);" class=""><br class="">
</span></font></div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><span style="background-color: rgb(255, 255, 255);
font-size: 12.15999984741211px;" class=""><font class=""
face="verdana, sans-serif"><a moz-do-not-send="true"
href="http://stackoverflow.com/questions/15034622/check-statistics-targets-in-postgresql"
class="">http://stackoverflow.com/questions/15034622/check-statistics-targets-in-postgresql</a></font></span></div>
<div class=""><br class="">
</div>
</blockquote>
<span style="font-size: 12.15999984741211px; background-color:
rgb(255, 255, 255);" class="">Check if this is the case.<br
class="">
</span>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><br class="">
</div>
</blockquote>
<span style="font-size: 12.15999984741211px; background-color:
rgb(255, 255, 255);" class="">
<div class=""><span style="font-size: 12.15999984741211px;
background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<br class="">
</span>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">El 2 dic 2016, a las 1:26, Bill Measday <<a
moz-do-not-send="true" href="mailto:bill@measday.com"
class="">bill@measday.com</a>> escribió:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Thanks Tom.<br class="">
<br class="">
First, this wasn't a migration but new db loaded from
scratch (if that matters).<br class="">
<br class="">
As per the end of the original post "I have vacuum
analysed both tables". I assume this is what you meant?<br
class="">
<br class="">
My gut feel was that it isn't a postgis issue since the
third example I gave uses the index, but I will take it up
with them too.<br class="">
<br class="">
Rgds<br class="">
<br class="">
<br class="">
Bill<br class="">
<br class="">
On 2/12/2016 10:48 AM, Tom Lane wrote:<br class="">
<blockquote type="cite" class="">Bill Measday <<a
moz-do-not-send="true" href="mailto:bill@measday.com"
class="">bill@measday.com</a>> writes:<br class="">
<blockquote type="cite" class="">Substantial different
index use between 9.5 and 9.6<br class="">
</blockquote>
Maybe you missed an ANALYZE after migrating? The plan
difference<br class="">
seems to be due to a vast difference in rowcount
estimate for the<br class="">
m_elevations condition:<br class="">
<br class="">
<blockquote type="cite" class=""> -> Bitmap
Heap Scan on m_elevations e<br class="">
(cost=282802.21..37401439.43 rows=3512160 width=8)<br
class="">
-> Seq Scan on m_elevations e<br class="">
(cost=10000000000.00..13296950520.12 rows=3512159563
width=8)<br class="">
</blockquote>
If you don't know where that factor-of-1000 came from,
maybe take<br class="">
it up with the postgis folk. It'd mostly be coming out
of their<br class="">
selectivity estimation routines.<br class="">
<br class="">
<span class="Apple-tab-span" style="white-space:pre"> </span><span \
class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" \
style="white-space:pre"> </span>regards, tom lane<br class="">
</blockquote>
<br class="">
<br class="">
<br class="">
-- <br class="">
Sent via pgsql-performance mailing list (<a
moz-do-not-send="true"
href="mailto:pgsql-performance@postgresql.org" \
class="">pgsql-performance@postgresql.org</a>)<br class="">
To make changes to your subscription:<br class="">
<a moz-do-not-send="true"
href="http://www.postgresql.org/mailpref/pgsql-performance"
class="">http://www.postgresql.org/mailpref/pgsql-performance</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic