[prev in list] [next in list] [prev in thread] [next in thread]
List: postgis-users
Subject: Re: [postgis-users] ST_value slow
From: J Payne <jcpayne () uw ! edu>
Date: 2017-02-04 23:21:53
Message-ID: 03173A87-5E99-4628-A153-90B55B4A97FA () uw ! edu
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Rémi,
My query was:
UPDATE hourly_positions hp SET landform =
(SELECT ST_Value(rast,hp.geom4326) FROM landforms4326_128 lf where (lf.rast && \
hp.geom4326) AND ST_Intersects(lf.rast,hp.geom4326))
You wrote: "your update is not safe because a point may intersect several tiles." \
I'd like to understand that better. Is this an edge-condition problem (i.e., a \
point falls right on an edge between two tiles)? If so, my query would fail because \
the subquery would return more than one value, correct?
If a point is right on the edge between two tiles, is there any situation where \
ST_Value(rast,point) might return different values from the two tiles? I saw your \
reply to a similar problem in this post,
http://lists.osgeo.org/pipermail/postgis-users/2014-February/038660.html,
where you suggested adding something like this:
WITH (your computing)
SELECT DISTINCT ON (line_id, poly_id) , poly, line
FROM your computing
ORDER BY ST_Length(line) ASC
…but if ST_Value() ever returns more than one value, DISTINCT will return \
both(all), which might just add to your problems. I notice that the questioner \
responded "my fix was to detect such lines and intersect them with the original \
polygon instead of the tiled one." I guess I'm just confused about what is really \
happening with the tiles.
Thanks,
John
P.S. I'm not sure why, but using the bbox did speed up the query a little bit.
From: postgis-users <postgis-users-bounces@lists.osgeo.org> on behalf of Rémi Cura \
<remi.cura@gmail.com>
Reply-To: PostGIS Users Discussion <postgis-users@lists.osgeo.org>
Date: Saturday, February 4, 2017 at 11:23 AM
To: PostGIS Users Discussion <postgis-users@lists.osgeo.org>
Subject: Re: [postgis-users] ST_value slow
You're welcome !
Actually I guess that the function is intended to work on one tile and one point.
So using a bbox would have no sense.
Now if you use it on a whole table instead of one tile,
you have to add intersect conditions and so.
By the way your update is not safe because a point may intersects several tiles.
Cheers
,Rémi-C
2017-02-04 20:00 GMT+01:00 J Payne <jcpayne@uw.edu>:
Hi Rémi,
Thanks for the suggestion. I tiled the raster and added a ST_Intersects, and the \
whole operation finished in just a couple of minutes (the previous version hadn't \
completed when I left it running overnight). The command I used was this:
UPDATE hourly_positions hp SET landform =
(SELECT ST_Value(rast,hp.geom4326) FROM landforms4326_128 lf where (lf.rast && \
hp.geom4326) AND ST_Intersects(lf.rast,hp.geom4326))
So, I don't mean to sound critical of PostGIS, because I love the program and I have \
the utmost respect for everyone who works on it, but it appears to me that ST_Value \
is a "dumb" function; i.e., it doesn't know how to find pixels in a raster with any \
reasonable speed. Is there any technical reason that its speed couldn't be improved? \
As I mentioned in my original question, this operation (finding pixel values at many \
individual points) is commonly needed for certain classes of problems.
Thanks again for saving my bacon!
John
From: postgis-users <postgis-users-bounces@lists.osgeo.org> on behalf of Rémi Cura \
<remi.cura@gmail.com>
Reply-To: PostGIS Users Discussion <postgis-users@lists.osgeo.org>
Date: Saturday, February 4, 2017 at 3:01 AM
To: PostGIS Users Discussion <postgis-users@lists.osgeo.org>
Subject: Re: [postgis-users] ST_value slow
Hey,
I suppose your raster is correctly cut into small pieces.
You should first try to clock one point value retrieval.
It should be < 50ms (or you have an index issue)
Then you could force the use of index by writting a function
update_one_point(point) $$ update XXX set XXX WHERE ST_Intersects(rast,point);$$
You would then call your function on all points:
SELECT update_one_point(hp.geom4326)
FROM hp
Cheers
Rémi-C
2017-02-04 6:44 GMT+01:00 J Payne <jcpayne@uw.edu>:
Hello. I have an animal track that wanders around on a landscape, and I want to \
extract the pixel values for each location on the track. This kind of operation is \
very common for animal studies. The landscape is represented by a one-row raster map \
("landforms4326"), and the GPS positions are in a table called "hourly_positions". I \
am using a recent version of PostGIS and a very simple query to find the pixel \
values:
UPDATE hourly_positions hp SET landform = (SELECT ST_Value(rast,hp.geom4326) FROM \
landforms4326)
I have about 450,000 points, and both the landscape raster and the point geometries \
are indexed. For some reason, the operation is *incredibly* slow (it's already been \
running for more than 8 hours on a nearly new Macbook Pro, and still isn't finished). \
Am I missing something? I can't for the life of me imagine why it would be so slow, \
since PostGIS is very fast with all sorts of other operations on the same data \
(intersections, etc.).
Thanks,
John
_______________________________________________
postgis-users mailing list
postgis-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________ postgis-users mailing list \
postgis-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________
postgis-users mailing list
postgis-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________ postgis-users mailing list \
postgis-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/postgis-users
[Attachment #5 (text/html)]
<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"><head><meta name=Title content=""><meta \
name=Keywords content=""><meta http-equiv=Content-Type content="text/html; \
charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered \
medium)"><style><!-- /* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:Courier;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div \
class=WordSection1><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>Hi Rémi,<o:p></o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>My query \
was:<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>UPDATE hourly_positions hp SET landform \
= <o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'> (SELECT \
ST_Value(rast,hp.geom4326) FROM landforms4326_128 lf where (lf.rast && \
hp.geom4326) AND ST_Intersects(lf.rast,hp.geom4326))<o:p></o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>You wrote: \
“your update is not safe because a point may intersect several tiles.” \
I’d like to understand that better. Is this an edge-condition problem (i.e., \
a point falls right on an edge between two tiles)? If so, my query would fail \
because the subquery would return more than one value, correct? \
<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>If a point is \
right on the edge between two tiles, is there any situation where \
ST_Value(rast,point) might return different values from the two tiles? I saw your \
reply to a similar problem in this post,<o:p></o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'><a \
href="http://lists.osgeo.org/pipermail/postgis-users/2014-February/038660.html">http:/ \
/lists.osgeo.org/pipermail/postgis-users/2014-February/038660.html</a>,<o:p></o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>where you \
suggested adding something like this:<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>WITH (your \
computing)<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>SELECT DISTINCT ON (line_id, poly_id) , \
poly, line<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>FROM your \
computing<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>ORDER BY ST_Length(line) \
ASC<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>…but if \
ST_Value() ever returns more than one value, DISTINCT will return both(all), which \
might just add to your problems. I notice that the questioner responded “my \
fix was to detect such lines and intersect them with the original polygon instead of \
the tiled one.” I guess I’m just confused about what is really \
happening with the tiles.<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>Thanks,<o:p></o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'>John<o:p></o:p></span></p><p \
class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><p \
class=MsoNormal><span style='font-size:11.0pt;font-family:Calibri'>P.S. I’m not \
sure why, but using the bbox did speed up the query a little bit. \
<o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt;font-family:Calibri'><o:p> </o:p></span></p><div \
style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p \
class=MsoNormal><b><span style='font-family:Calibri;color:black'>From: \
</span></b><span style='font-family:Calibri;color:black'>postgis-users \
<postgis-users-bounces@lists.osgeo.org> on behalf of Rémi Cura \
<remi.cura@gmail.com><br><b>Reply-To: </b>PostGIS Users Discussion \
<postgis-users@lists.osgeo.org><br><b>Date: </b>Saturday, February 4, 2017 at \
11:23 AM<br><b>To: </b>PostGIS Users Discussion \
<postgis-users@lists.osgeo.org><br><b>Subject: </b>Re: [postgis-users] ST_value \
slow<o:p></o:p></span></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal><span \
style='font-family:Courier'>You're welcome !<o:p></o:p></span></p></div><div><p \
class=MsoNormal><span style='font-family:Courier'>Actually I guess that the function \
is intended to work on one tile and one point.<o:p></o:p></span></p></div><div><p \
class=MsoNormal><span style='font-family:Courier'>So using a bbox would have no \
sense.<o:p></o:p></span></p></div><div><p class=MsoNormal><span \
style='font-family:Courier'>Now if you use it on a whole table instead of one \
tile, <o:p></o:p></span></p></div><div><p class=MsoNormal><span \
style='font-family:Courier'>you have to add intersect conditions and \
so.<o:p></o:p></span></p></div><div><p class=MsoNormal \
style='margin-bottom:12.0pt'><span style='font-family:Courier'>By the way your update \
is not safe because a point may intersects several \
tiles.<o:p></o:p></span></p></div><div><p class=MsoNormal><span \
style='font-family:Courier'>Cheers<o:p></o:p></span></p></div><div><p \
class=MsoNormal><span \
style='font-family:Courier'>,Rémi-C<o:p></o:p></span></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>2017-02-04 20:00 \
GMT+01:00 J Payne <<a href="mailto:jcpayne@uw.edu" \
target="_blank">jcpayne@uw.edu</a>>:<o:p></o:p></p><blockquote \
style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>Hi Rémi,</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>Thanks for the suggestion. I tiled \
the raster and added a ST_Intersects, and the whole operation finished in just a \
couple of minutes (the previous version hadn’t completed when I left it running \
overnight). The command I used was this:</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>UPDATE hourly_positions hp SET landform \
= </span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> (SELECT \
ST_Value(rast,hp.geom4326) FROM landforms4326_128 lf where (lf.rast && \
hp.geom4326) AND ST_Intersects(lf.rast,hp.geom4326))</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>So, I don’t mean to sound critical \
of PostGIS, because I love the program and I have the utmost respect for everyone who \
works on it, but it appears to me that ST_Value is a “dumb” function; \
i.e., it doesn’t know how to find pixels in a raster with any reasonable \
speed. Is there any technical reason that its speed couldn’t be \
improved? As I mentioned in my original question, this operation (finding pixel \
values at many individual points) is commonly needed for certain classes of \
problems.</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>Thanks again for saving my \
bacon!</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'>John</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt;font-family:Calibri'> </span><o:p></o:p></p><div \
style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span \
style='font-family:Calibri;color:black'>From: </span></b><span \
style='font-family:Calibri;color:black'>postgis-users <<a \
href="mailto:postgis-users-bounces@lists.osgeo.org" \
target="_blank">postgis-users-bounces@lists.osgeo.org</a>> on behalf of Rémi Cura \
<<a href="mailto:remi.cura@gmail.com" \
target="_blank">remi.cura@gmail.com</a>><br><b>Reply-To: </b>PostGIS Users \
Discussion <<a href="mailto:postgis-users@lists.osgeo.org" \
target="_blank">postgis-users@lists.osgeo.org</a>><br><b>Date: </b>Saturday, \
February 4, 2017 at 3:01 AM<br><b>To: </b>PostGIS Users Discussion <<a \
href="mailto:postgis-users@lists.osgeo.org" \
target="_blank">postgis-users@lists.osgeo.org</a>><br><b>Subject: </b>Re: \
[postgis-users] ST_value slow</span><o:p></o:p></p></div><div><div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>Hey,</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><span \
style='font-family:Courier'>I suppose your raster is correctly cut into small \
pieces.</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>You should first try to clock one point value \
retrieval.</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>It should be < 50ms (or you have an index \
issue)</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'> </span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>Then you could force the use of index by writting a \
function <br>update_one_point(point) $$ update XXX set XXX WHERE \
ST_Intersects(rast,point);$$</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'> </span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><span \
style='font-family:Courier'>You would then call your function on all \
points:</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>SELECT \
update_one_point(hp.geom4326)</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><span \
style='font-family:Courier'>FROM hp</span><o:p></o:p></p></div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>Cheers</span><o:p></o:p></p></div><div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'>Rémi-C</span><o:p></o:p></p></div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-family:Courier'> </span><o:p></o:p></p></div></div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>2017-02-04 \
6:44 GMT+01:00 J Payne <<a href="mailto:jcpayne@uw.edu" \
target="_blank">jcpayne@uw.edu</a>>:<o:p></o:p></p><blockquote \
style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt'>Hello. I have an animal track that wanders around on a \
landscape, and I want to extract the pixel values for each location on the \
track. This kind of operation is very common for animal studies. The \
landscape is represented by a one-row raster map (“landforms4326”), and \
the GPS positions are in a table called “hourly_positions”. I am \
using a recent version of PostGIS and a very simple query to find the pixel \
values:</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt'>UPDATE hourly_positions hp SET landform = (SELECT \
ST_Value(rast,hp.geom4326) FROM landforms4326)</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:11.0pt'>I have about 450,000 points, and both the landscape raster \
and the point geometries are indexed. For some reason, the operation is \
*<b>incredibly</b>* slow (it’s already been running for more than 8 hours on a \
[Attachment #6 (text/plain)]
_______________________________________________
postgis-users mailing list
postgis-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic