[prev in list] [next in list] [prev in thread] [next in thread] 

List:       postgis-users
Subject:    [postgis-users] Parallelisation provides powerful postgis performance perks (script + ppt slides)
From:       Mark Wynter <mark () dimensionaledge ! com>
Date:       2015-07-23 22:55:05
Message-ID: 7750D188-A203-4A8A-9D1A-490919BF2F99 () dimensionaledge ! com
[Download RAW message or body]

A couple of tutorials on the subject... With full code on github

http://dimensionaledge.com/intro-vector-tiling-map-reduce-postgis/

http://dimensionaledge.com/from-days-to-minutes-geoprocessing-of-alberta-land-use-data/


> When I briefly look at the text you have written in the "Quick Example" It
> seems that you are distributing your query by an ID field. I am wondering
> how your method would apply to raster datasets? Distributing geographic
> data by an ID can get you into problems because of the dependency for
> certain analytical functions.

Gnu parallel great for processing complex pipelines. Mix and match with PostGis \
vectors and rasters, grass, R, gDAL etc 

iD is simplest way...  But your job list can have multiple arguments which you can \
feed into plpgsql function that would be called in the worker function. 

You can build in as much sophistication as you like into the plpgsql function.

Some things to bear in mind - get your querys working efficiently before scaling out \
- otherwise you are scaling out bad practice. Batch processing faster than individual \
processing And Dump your multipolgons into individual polygons if do intersection \
analysis.

Another parsllelusation tool is R via pl/r, which I'm using for routing analysis. \
More specialized and not always as versatile as Gnu parallel.

hTH.
Mark

_______________________________________________
postgis-users mailing list
postgis-users@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users


[prev in list] [next in list] [prev in thread] [next in thread]