[prev in list] [next in list] [prev in thread] [next in thread]
List: postgis-users
Subject: Re: [postgis-users] Better storage strategy for 1 million tile of 50*50 pixels, 24 bands?
From: guido lemoine <guido.lemoine () jrc ! ec ! europa ! eu>
Date: 2014-04-18 11:50:45
Message-ID: 7400b2f93ef.53512db5 () jrc ! ec ! europa ! eu
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Remi=2C
I access postgis raster data with Java=2C using JDBC and ST=5FAsTiff for=
raster results and then JAI (Java Advanced Imaging)=2E JAI allows me to=
do stuff which can now be done mostly with ST=5FMapAlgebra as well=2E S=
ince I am using=A0OpenCV (which has most functions accessible through Ja=
va wrappers) in other approaches=2C I don=27t expect to have any issue w=
ith using that with postgis=2E In particular=2C I expect to use GPU acce=
lerated OpenCV routines (template matching=2C segmentation=2C classifica=
tion) to work on chunks I pull out of the Landsat-8 data base=2E
I will definitely try both rasdaman and SciDB at some stage=2C but for n=
ow=2C the MVC possibilities provided by the postgis/java/grails suite ar=
e pretty solid=2E
Guido
=A0
On 04/18/14=2C R=E9mi Cura =3Cremi=2Ecura=40gmail=2Ecom=3E wrote=3A
=3E =
=3E =
=3E =
=3E =
=3E =
=3E =
=3E Hey=2C thanks for sharing your experience !
=3E =
=3E =
=3E =
=3E 2014-04-18 0=3A54 GMT+02=3A00 Peter Baumann =3Cp=2Ebaumann=40jacobs-=
university=2Ede =3Cp=2Ebaumann=40jacobs-university=2Ede=3E=3E=3A
=3E =
=3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E On 04/17/2014 08=3A11 PM=2C guido lemoine
=3E =3E wrote=3A
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E So what do you use instead=3F Your data set is not exa=
ctly common
=3E =3E =3E (400+ Gb=2C 24 band doubles (complex data=3F)=2C overl=
apping tiles=2C
=3E =3E =3E which are half empty)=2E Is this LiDAR data=3F What ex=
actly are you
=3E =3E =3E trying to visualise=3F
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =
=3E =
=3E Yep =2C good guess=2E =
=3E Postgres + pointcloud seems to work good (I have a 5=2E4 billion poi=
nts test database=2C no problem) to handle lidar=2C =
=3E but visualisation within GIS software is an issue=2E Having a raster=
ized view could be usefull for this=2E
=3E =
=3E =
=3E How course importing millions of points in QGIS is not really an opt=
ion (it works up to a million if style is not too complex)=2E
=3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E We have little issue with postgis raster speed using
=3E =3E =3E multi-temporal=2C multi-frame=2C all bands Landsat=
-8=2C either in-db
=3E =3E =3E or out-db=2E=A0
=3E =3E =3E =
=3E =3E =3E Over 100 Gb by now=2E We prefer out-of-db=2C because it keep=
s
=3E =3E =3E the raster db of manageable size=2E We expect it t=
o work up to
=3E =3E =3E Terrabyte limits (I am told I should use SciDB =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E when doing that=2C give rasdaman a try and compare then=2E I=
t=27s
=3E =3E operational on n-D multi-TB objects =3A)
=3E =3E =
=3E =3E -Peter
=3E =3E =A0
=3E =3E =
=3E =3E =
=3E =
=3E I know PostGIS raster is a powerful tool=2C
=3E =
=3E I just say that in-base multi-band raster creation is slow
=3E =
=3E =2Cany raster processing is slow (need to rewrite entire raster)
=3E =
=3E =
=3E =2Cimage processing options are limited (it could be so cool to plug=
a good image processing lib like itk or otb)
=3E =
=3E =2Caccessing in-base raster is still fragile
=3E =
=3E =2CLevel of Detail strategy are not fully integrated=2E
=3E =
=3E =
=3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E beyond that)=2E This works with mounted network disk as well=
=2C
=3E =3E =3E which is even nicer=2E
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =
=3E =
=3E The whole postgres data folder can be put on network disk (with cave=
at)=2C which is very usefull !
=3E =
=3E =A0
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E GL
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E On 04/17/14=2C R=E9mi Cura =
=3E =3E =3E =3Cremi=2Ecura=40gmail=2Ecom=3E =3Cremi=2Ecura=
=40gmail=2Ecom=3E wrote=3A
=3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E Hey=2C
=3E =3E =3E =3E =
=3E =3E =3E =3E from what I tried PostGis Ra=
ster is
=3E =3E =3E =3E (relatively) slow and not ad=
apted to multi
=3E =3E =3E =3E bands
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E (for instance=2C no way to set=
multiple bands at
=3E =3E =3E =3E once=2E For my use case this w=
ould mean about
=3E =3E =3E =3E 300k*0=2E1sec i=2Ee=2E about 8=
hours at best)=2E
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E The GDAL driver + QGIS should be=
considered an
=3E =3E =3E =3E experimental function at the mom=
ent=2C =
=3E =3E =3E =3E =
=3E =3E =3E =3E because it is still easily broke=
n (and I=27m
=3E =3E =3E =3E speaking cutting edge gdal + qgi=
s v 2=2C 2=2E2=2C 2=2E3
=3E =3E =3E =3E on win and linux)=2E
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E More generaly using QGIS with postgis is
=3E =3E =3E =3E easily broken (i restart qgis se=
veral dozen
=3E =3E =3E =3E times a day)
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E Taking that into consideration=2C =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E My conclusion is that for the mome=
nt there is no
=3E =3E =3E =3E incentive to use =5F=5Fin base=5F=5F=
postgis raster=2C =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E because there is no stable way to access raster
=3E =3E =3E =3E from outside base=2E
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E Sadly in my general use case=2C this=
translate to no
=3E =3E =3E =3E incentive to use postgis raster at a=
ll =3A-(
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E Cheers=2C
=3E =3E =3E =3E =
=3E =3E =3E =3E R=E9mi-C
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E 2014-04-17 17=3A43 GMT+02=3A00
=3E =3E =3E =3E Tumasgiu Rossini =3Crossini=2Et=40gm=
ail=2Ecom =3Crossini=2Et=40gmail=2Ecom=3E
=3E =3E =3E =3E =3Crossini=2Et=40gmail=2Ecom=3E =3C=
rossini=2Et=40gmail=2Ecom=3E=3E=3A
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E Remi=2C
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E I=27m interested in the wa=
y you handled this
=3E =3E =3E =3E =3E problem=2E
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E In my opinion=2C the probl=
em is on the duality
=3E =3E =3E =3E =3E of your needs=2E
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E Out-db is known to be faster for
=3E =3E =3E =3E =3E visualisation=2E
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E In-db is better for analysis=2E
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E So=2C a compromise should be made Maybe
=3E =3E =3E =3E =3E storing each tile per =22b=
and type=22 =3F So
=3E =3E =3E =3E =3E accessing data from qgis w=
ould be less painful
=3E =3E =3E =3E =3E =3F
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E 2014-03-10 10=3A54
=3E =3E =3E =3E =3E GMT+01=3A00 R=E9mi Cura =3C=
remi=2Ecura=40gmail=2Ecom =3Cremi=2Ecura=40gmail=2Ecom=3E
=3E =3E =3E =3E =3E =3Cremi=2Ecura=40gmail=2E=
com=3E =3Cremi=2Ecura=40gmail=2Ecom=3E=3E=3A
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=
=3E =3E =3E =3E =3E =3E Hey Dear List=2C
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=
=3E =3E =3E =3E =3E =3E =
I would appreciate
=3E =3E =3E =3E =3E =3E =
some advice about the
=3E =3E =3E =3E =3E =3E =
best way to store my
=3E =3E =3E =3E =3E =3E =
raster =3A
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
1 million tiles=2C
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E 50=
*50 pixels each (1 m2 or
=3E =3E =3E =3E =3E =3E le=
ss in real world)=2C
=3E =3E =3E =3E =3E =3E ar=
ound 24 bands (mostly
=3E =3E =3E =3E =3E =3E do=
ubles)
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =2Cin db=2E
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E About half the pixels
=3E =3E =3E =3E =3E =3E ar=
e empty=2C some tiles
=3E =3E =3E =3E =3E =3E ov=
erlaps=2C but most are
=3E =3E =3E =3E =3E =3E re=
gularly spaced=2E
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E I wo=
uld query it mainly by
=3E =3E =3E =3E =3E =3E loca=
lisation (intersects)=2C
=3E =3E =3E =3E =3E =3E and =
also based on id of the
=3E =3E =3E =3E =3E =3E tile=
=2E
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E The use would be fast
=3E =3E =3E =3E =3E =3E visu=
alisation with qgis (and
=3E =3E =3E =3E =3E =3E late=
st gdal)=2C interpolation=2C
=3E =3E =3E =3E =3E =3E clas=
sification=2C matching and
=3E =3E =3E =3E =3E =3E so=2E=
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E What i=
s the best strategy=3F
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E 1 table =
with many lines and
=3E =3E =3E =3E =3E =3E indexes=2C=
indb=2C
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E 1 table=2C=
out db
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E 1 table=2C 1 line
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E multiple tab=
les=2C heritage=3F
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E Thanks for inputs!
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E Cheers=2C
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E R=E9mi-C
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E postgis-users mailin=
g list
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E postgis-users=40list=
s=2Eosgeo=2Eorg =3Cpostgis-users=40lists=2Eosgeo=2Eorg=3E
=3E =3E =3E =3E =3E =3E =3Cpostgis-users=40l=
ists=2Eosgeo=2Eorg=3E =3Cpostgis-users=40lists=2Eosgeo=2Eorg=3E
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E http=3A//lists=2Eosg=
eo=2Eorg/cgi-bin/mailman/listinfo/postgis-users
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E postgis-users mailing list
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E postgis-users=40lists=2Eosgeo=2E=
org =3Cpostgis-users=40lists=2Eosgeo=2Eorg=3E
=3E =3E =3E =3E =3E =3Cpostgis-users=40lists=2Eosg=
eo=2Eorg=3E =3Cpostgis-users=40lists=2Eosgeo=2Eorg=3E
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E http=3A//lists=2Eosgeo=2Eorg/c=
gi-bin/mailman/listinfo/postgis-users
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F
=3E =3E =3E postgis-users mailing list
=3E =3E =3E postgis-users=40lists=2Eosgeo=2Eorg =3Cpostgis-users=40lists=
=2Eosgeo=2Eorg=3E
=3E =3E =3E http=3A//lists=2Eosgeo=2Eorg/cgi-bin/mailman/listinfo/postgi=
s-users
=3E =3E =3E =
=3E =3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E -- =
=3E =3E Dr=2E Peter Baumann
=3E =3E - Professor of Computer Science=2C Jacobs University Bremen
=3E =3E www=2Efaculty=2Ejacobs-university=2Ede/pbaumann(http=3A//www=2E=
faculty=2Ejacobs-university=2Ede/pbaumann)
=3E =3E mail=3A p=2Ebaumann=40jacobs-university=2Ede =3Cp=2Ebaumann=40=
jacobs-university=2Ede=3E
=3E =3E tel=3A +49-421-200-3178(tel=3A=252B49-421-200-3178)=2C fax=3A=
+49-421-200-493178(tel=3A=252B49-421-200-493178)
=3E =3E - Executive Director=2C rasdaman GmbH Bremen (HRB 26793)
=3E =3E www=2Erasdaman=2Ecom(http=3A//www=2Erasdaman=2Ecom)=2C mail=3A=
baumann=40rasdaman=2Ecom =3Cbaumann=40rasdaman=2Ecom=3E
=3E =3E tel=3A 0800-rasdaman=2C fax=3A 0800-rasdafax=2C mobile=3A +49=
-173-5837882(tel=3A=252B49-173-5837882)
=3E =3E =22Si forte in alienas manus oberraverit hec peregrina epistola =
incertis ventis dimissa=2C sed Deo commendata=2C precamur ut ei reddatur=
cui soli destinata=2C nec preripiat quisquam non sibi parata=2E=22 (mai=
l disclaimer=2C AD 1083)
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =
=3E =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F
=3E =3E =
=3E =3E postgis-users mailing list
=3E =3E =
=3E =3E postgis-users=40lists=2Eosgeo=2Eorg =3Cpostgis-users=40lists=2Eo=
sgeo=2Eorg=3E
=3E =3E =
=3E =3E http=3A//lists=2Eosgeo=2Eorg/cgi-bin/mailman/listinfo/postgis-us=
ers
=3E =3E =
=3E =
=3E =
=3E
[Attachment #5 (text/html)]
<div style="font-family: 'Times New Roman'; font-size: 16px; \
"><div>Remi,</div><div><br /></div><div>I access postgis raster data with Java, using \
JDBC and ST_AsTiff for raster results and then JAI (Java Advanced Imaging). JAI \
allows me to do stuff which can now be done mostly with ST_MapAlgebra as well. Since \
I am using OpenCV (which has most functions accessible through Java wrappers) in \
other approaches, I don't expect to have any issue with using that with postgis. In \
particular, I expect to use GPU accelerated OpenCV routines (template matching, \
segmentation, classification) to work on chunks I pull out of the Landsat-8 data \
base.</div><div><br /></div><div>I will definitely try both rasdaman and SciDB at \
some stage, but for now, the MVC possibilities provided by the postgis/java/grails \
suite are pretty solid.</div><div><br /></div><div>Guido</div><div><br \
/></div><div><br /></div><div ><br/></div><div><br /></div><div><br /></div><div>On \
04/18/14, <b class="name">Rémi Cura </b> <remi.cura@gmail.com> \
wrote:</div><blockquote \
cite="mid:CAJvUf_u99D8XpuAJjKSdpxbHeSE+rkO8D=c35vg88=tQ1ck1jg@mail.gmail.com" \
class="iwcQuote" style="border-left: 1px solid #00F; padding-left: 13px; margin-left: \
0;" type="cite"><div class="mimepart text html"><div dir="ltr"><br /><div \
class="gmail_extra"><br /></div><div class="gmail_extra">Hey, thanks for sharing your \
experience !<br /><br /></div><div class="gmail_extra"><div \
class="gmail_quote">2014-04-18 0:54 GMT+02:00 Peter Baumann <span \
dir="ltr"><p.baumann@jacobs-university.de \
<p.baumann@jacobs-university.de>></span>:<br /> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div class="">
<div>On 04/17/2014 08:11 PM, guido lemoine
wrote:<br />
</div>
<blockquote type="cite">
So what do you use instead? Your data set is not exactly common
(400+ Gb, 24 band doubles (complex data?), overlapping tiles,
which are half empty). Is this LiDAR data? What exactly are you
trying to visualise?
<div ><br/></div></blockquote></div></div></blockquote><div>Yep , good guess. \
<br />Postgres + pointcloud seems to work good (I have a 5.4 billion points test \
database, no problem) to handle lidar, <br />but visualisation within GIS software is \
an issue. Having a rasterized view could be usefull for this.<br /> </div><div>How \
course importing millions of points in QGIS is not really an option (it works up to a \
million if style is not too complex).<br /></div><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div><div \
class=""><blockquote type="cite"><div> <div>We have little issue with postgis raster \
speed using multi-temporal, multi-frame, all bands Landsat-8, either in-db
or out-db. </div>
<div>Over 100 Gb by now. We prefer out-of-db, because it keeps
the raster db of manageable size. We expect it to work up to
Terrabyte limits (I am told I should use SciDB </div>
</div>
</blockquote>
<br /></div>
when doing that, give rasdaman a try and compare then. It's
operational on n-D multi-TB objects :)<br />
-Peter<br />
<br /></div></blockquote><div>I know PostGIS raster is a powerful tool,<br \
/></div><div>I just say that in-base multi-band raster creation is slow<br \
/></div><div>,any raster processing is slow (need to rewrite entire raster)<br /> \
</div><div>,image processing options are limited (it could be so cool to plug a good \
image processing lib like itk or otb)<br /></div><div>,accessing in-base raster is \
still fragile<br /></div><div>,Level of Detail strategy are not fully integrated.<br \
/> </div><br /><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" \
text="#000000"> <blockquote type="cite"><div><div class="h5">
<div>
<div>beyond that). This works with mounted network disk as well,
which is even nicer.</div>
<div><br /></div></div></div></div></blockquote></div></blockquote><div>The \
whole postgres data folder can be put on network disk (with caveat), which is very \
usefull !<br /></div><div ><br/></div><blockquote class="gmail_quote" style="margin:0 \
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" \
text="#000000"><blockquote type="cite"><div><div class="h5"><div><div ><br/></div> \
<div>GL</div> <div>
<div style="font-size:16px">
<div><br />
</div>
<div><br />
</div>
<div>On 04/17/14, <b>Rémi Cura </b>
<remi.cura@gmail.com> <remi.cura@gmail.com> wrote:</div>
<blockquote style="border-left:1px solid \
#00f;padding-left:13px;margin-left:0" type="cite"> <div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hey,<br />
from what I tried PostGis Raster is
(relatively) slow and not adapted to multi
bands<br />
</div>
(for instance, no way to set multiple bands at
once. For my use case this would mean about
300k*0.1sec i.e. about 8 hours at best).<br />
<br />
</div>
The GDAL driver + QGIS should be considered an
experimental function at the moment, <br />
because it is still easily broken (and I'm
speaking cutting edge gdal + qgis v 2, 2.2, 2.3
on win and linux).<br />
</div>
<div>More generaly using QGIS with postgis is
easily broken (i restart qgis several dozen
times a day)<br />
</div>
<div><br />
</div>
<div>Taking that into consideration, <br />
</div>
My conclusion is that for the moment there is no
incentive to use __in base__ postgis raster, <br />
</div>
<div>because there is no stable way to access raster
from outside base.<br />
<br />
</div>
Sadly in my general use case, this translate to no
incentive to use postgis raster at all :-(<br />
<br />
</div>
Cheers,<br />
Rémi-C<br />
<div>
<div>
<div><br />
<br />
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br />
<br />
<div class="gmail_quote">2014-04-17 17:43 GMT+02:00
Tumasgiu Rossini <span dir="ltr"><rossini.t@gmail.com \
<rossini.t@gmail.com>
<rossini.t@gmail.com> \
<rossini.t@gmail.com>></span>:<br />
<blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr">
<div>
<div>Remi,<br />
<br />
</div>
I'm interested in the way you handled this
problem.<br />
</div>
<div><br />
In my opinion, the problem is on the duality
of your needs.<br />
</div>
<div>Out-db is known to be faster for
visualisation.<br />
</div>
<div>In-db is better for analysis.<br />
<br />
</div>
<div>So, a compromise should be made Maybe
storing each tile per "band type" ? So
accessing data from qgis would be less painful
?<br />
</div>
<div><br />
</div>
<div>
<br />
</div>
</div>
<div class="gmail_extra"><br />
<br />
<div class="gmail_quote">2014-03-10 10:54
GMT+01:00 Rémi Cura <span dir="ltr"><remi.cura@gmail.com \
<remi.cura@gmail.com>
<remi.cura@gmail.com> \
<remi.cura@gmail.com>></span>:<br />
<blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hey Dear List,<br />
<br />
</div>
I would appreciate
some advice about the
best way to store my
raster :<br />
<br />
</div>
1 million tiles,<br />
</div>
50*50 pixels each (1 m2 or
less in real world),
around 24 bands (mostly
doubles)<br />
</div>
<div>,in db.<br />
</div>
<div><br />
</div>
<div>About half the pixels
are empty, some tiles
overlaps, but most are
regularly spaced.<br />
</div>
<div><br />
</div>
I would query it mainly by
localisation (intersects),
and also based on id of the
tile.<br />
<br />
</div>
<div>The use would be fast
visualisation with qgis (and
latest gdal), interpolation,
classification, matching and
so.<br />
</div>
<div><br />
</div>
What is the best strategy?<br />
</div>
1 table with many lines and
indexes, indb,<br />
</div>
1 table, out db<br />
</div>
<div>1 table, 1 line<br />
</div>
multiple tables, heritage?<br />
<br />
<br />
</div>
<div>Thanks for inputs!<br />
</div>
Cheers,<br />
<br />
Rémi-C<br />
</div>
<br />
</div>
</div>
_______________________________________________<br />
postgis-users mailing list<br />
postgis-users@lists.osgeo.org \
<postgis-users@lists.osgeo.org>
<postgis-users@lists.osgeo.org> \
<postgis-users@lists.osgeo.org><br /> <a \
href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users</a><br \
/> </blockquote>
</div>
<br />
</div>
<br />
_______________________________________________<br />
postgis-users mailing list<br />
postgis-users@lists.osgeo.org \
<postgis-users@lists.osgeo.org>
<postgis-users@lists.osgeo.org> \
<postgis-users@lists.osgeo.org><br /> <a \
href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users</a><br \
/> </blockquote>
</div>
<br />
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br />
<fieldset></fieldset>
<br />
</div></div><pre><div><div \
class="h5">_______________________________________________ postgis-users mailing list
postgis-users@lists.osgeo.org <postgis-users@lists.osgeo.org>
</div></div><a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users</a></pre><span \
class="HOEnZb"><font color="#888888"> </font></span></blockquote><span \
class="HOEnZb"><font color="#888888"> <br />
<pre cols="80">--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
<a href="http://www.faculty.jacobs-university.de/pbaumann" \
target="_blank">www.faculty.jacobs-university.de/pbaumann</a> mail: \
p.baumann@jacobs-university.de <p.baumann@jacobs-university.de> tel: <a \
href="tel:%2B49-421-200-3178" target="1" value="+494212003178">+49-421-200-3178</a>, \
fax: <a href="tel:%2B49-421-200-493178" target="1" \
value="+49421200493178">+49-421-200-493178</a>
- Executive Director, rasdaman GmbH Bremen (HRB 26793)
<a href="http://www.rasdaman.com" target="_blank">www.rasdaman.com</a>, mail: \
baumann@rasdaman.com <baumann@rasdaman.com> tel: 0800-rasdaman, fax: \
0800-rasdafax, mobile: <a href="tel:%2B49-173-5837882" target="1" \
value="+491735837882">+49-173-5837882</a> "Si forte in alienas manus oberraverit \
hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei \
reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail \
disclaimer, AD 1083)
</pre>
</font></span></div>
<br />_______________________________________________<br />
postgis-users mailing list<br />
postgis-users@lists.osgeo.org <postgis-users@lists.osgeo.org><br />
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users" \
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users</a><br \
/></blockquote></div><br /></div></div> </div></blockquote></div>
_______________________________________________
postgis-users mailing list
postgis-users@lists.osgeo.org
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic