[prev in list] [next in list] [prev in thread] [next in thread]
List: postgis-devel
Subject: Re: [postgis-devel] Sharding Rasters in Postgis
From: Paul Ramsey <pramsey () cleverelephant ! ca>
Date: 2014-09-23 21:14:57
Message-ID: etPan.5421e2d1.431bd7b7.297a () Butterfly ! local
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
If you're just doing queries that don't take advantage of spatial correlation at all, \
using any hashkey on the contents as the sharding key should work just great. Then \
you can use pl/proxy to run roll-ups against spatial polygons, etc, easily and know \
you won't end up with hot nodes.
P
--
Paul Ramsey
http://cleverelephant.ca
http://postgis.net
On September 23, 2014 at 2:11:18 PM, David Haynes II (dahaynes@umn.edu) wrote:
Yes, we are going to use the more finished product Postgres-xl, but even that is not \
production ready. So I would think the most logical solution is to generate the \
process ourselves.
Mostly wondering how to distribute the tables to various locations, once I do the \
difficulty part of aligning tiles to exist with a defined coordinate space.
On Mon, Sep 22, 2014 at 11:12 AM, Rémi Cura <remi.cura@gmail.com> wrote:
Do you mean somethiong like postgres-xc \
:http://postgresxc.wikia.com/wiki/Postgres-XC_Wiki
Cheers,
Rémi-C
2014-09-22 17:04 GMT+02:00 David Haynes II <dahaynes@umn.edu>:
Hello,
I was wondering if there any helpful examples for distributing rasters tables in \
postgis. Are there other items that I need to consider? Will the PostGIS functions \
need to be re-written if we want to parallelize the processing, using something like \
pg_proxy?
I came an example using a different software package, (slide 16)
http://spark-summit.org/wp-content/uploads/2014/07/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf
I would prefer to stay within postgresql community.
--
David Haynes, Ph.D.
Research Associate Terra Populous
Minnesota Population Center
_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
--
David Haynes, Ph.D.
Research Associate Terra Populous
Minnesota Population Center
_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
[Attachment #5 (text/html)]
<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body \
style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space;"><div id="bloop_customfont" \
style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: \
0px; line-height: auto;">If you're just doing queries that don't take advantage of \
spatial correlation at all, using any hashkey on the contents as the sharding key \
should work just great. Then you can use pl/proxy to run roll-ups against spatial \
polygons, etc, easily and know you won't end up with hot nodes.</div><div \
id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: \
rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div \
id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: \
rgba(0,0,0,1.0); margin: 0px; line-height: auto;">P</div> <div \
id="bloop_sign_1411506836761731072" class="bloop_sign"> <title></title>
<div>
<br>
</div>
-- <br>
Paul Ramsey<br>
http://cleverelephant.ca<div>http://postgis.net
</div>
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "em",
"name": "John Doe",
"jobTitle": "Graduate research assistant",
"affiliation": "University of Dreams",
"additionalName": "Johnny",
"url": "http://www.example.com",
"address": {
"@type": "PostalAddress",
"streetAddress": "1234 Peach Drive",
"addressLocality": "Wonderland",
"addressRegion": "Georgia"
}
}
</script>
</div> <br><p style="color:#000;">On September 23, 2014 at 2:11:18 PM, David Haynes \
II (<a href="mailto:dahaynes@umn.edu">dahaynes@umn.edu</a>) wrote:</p> <blockquote \
type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr">Yes, we are going to use the more finished product
Postgres-xl, but even that is not production ready.
<div>So I would think the most logical solution is to generate the
process ourselves.</div>
<div><br></div>
<div>Mostly wondering how to distribute the tables to various
locations, once I do the difficulty part of aligning tiles to exist
with a defined coordinate space.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 22, 2014 at 11:12 AM, Rémi
Cura <span dir="ltr"><<a href="mailto:remi.cura@gmail.com" \
target="_blank">remi.cura@gmail.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <div dir="ltr">
<div>Do you mean somethiong like postgres-xc :<a \
href="http://postgresxc.wikia.com/wiki/Postgres-XC_Wiki" \
target="_blank">http://postgresxc.wikia.com/wiki/Postgres-XC_Wiki</a><br>
<br></div>
Cheers,<br>
Rémi-C<br></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2014-09-22 17:04 GMT+02:00 David Haynes II
<span dir="ltr"><<a href="mailto:dahaynes@umn.edu" \
target="_blank">dahaynes@umn.edu</a>></span>:<br> <blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div \
dir="ltr">Hello, <div><br></div>
<div>I was wondering if there any helpful examples for distributing
rasters tables in postgis. Are there other items that I need to
consider? Will the PostGIS functions need to be re-written if we
want to parallelize the processing, using something like
pg_proxy? </div>
<div><br></div>
<div>I came an example using a different software package, (slide
16)</div>
<div><a href="http://spark-summit.org/wp-content/uploads/2014/07/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf" \
target="_blank">http://spark-summit.org/wp-content/uploads/2014/07/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf</a></div>
<div>I would prefer to stay within postgresql
community.<span class="HOEnZb"><font color="#888888"><span><font color="#888888"><br \
clear="all"></font></span></font></span> <div><span class="HOEnZb"><font \
color="#888888"><font color="#888888"><br></font></font></span></div> <span \
class="HOEnZb"><font color="#888888"><font \
color="#888888">--<br></font></font></span> <div dir="ltr"><span class="HOEnZb"><font \
color="#888888"><font color="#888888">David Haynes, Ph.D.</font></font></span>
<div><span class="HOEnZb"><font color="#888888"><font color="#888888">Research \
Associate Terra Populous</font></font></span></div>
<div><span class="HOEnZb"><font color="#888888"><font color="#888888">Minnesota \
Population Center</font></font></span></div> </div>
</div>
</div>
<span class="HOEnZb"><font color="#888888"><br>
_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" \
target="_blank">postgis-devel@lists.osgeo.org</a><br> <a \
href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br> \
</font></span></blockquote> </div>
<br></div>
<br>
_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br> \
</blockquote> </div>
<br>
<br clear="all">
<div><br></div>
--<br>
<div dir="ltr">David Haynes, Ph.D.
<div>Research Associate Terra Populous</div>
<div>Minnesota Population Center</div>
</div>
</div>
_______________________________________________
<br>postgis-devel mailing list
<br>postgis-devel@lists.osgeo.org
<br>http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</div></div></span></blockquote></body></html>
_______________________________________________
postgis-devel mailing list
postgis-devel@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic