[prev in list] [next in list] [prev in thread] [next in thread]
List: osgeo-discuss
Subject: Re: [OSGeo-Discuss] fastest option of serving huge imagery on
From: "Bob Basques" <Bob.Basques () ci ! stpaul ! mn ! us>
Date: 2010-05-24 15:19:08
Message-ID: 4BFA529C.163B.00A8.0 () ci ! stpaul ! mn ! us
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
All,
Related to this . . .
> From our perspective, which was to allow for the most flexibilty with regard to \
> queries in the size of the images, we were willing to sacrifice the highest level \
> of performance. Most of the effort here was in organizing the tiles (tree index) \
> behind the MapServer service. Setting up the Mapfiles to generally drop pixels vs \
> interpolating them (adding new ones) was a big piece of the performance issue in \
> our configuration for example. Simply setting the MIN/MAX scale params accordingly \
> helped a great deal. MapServer (and most other products BTW), seems to be better \
> at dropping pixels (reducing the image size) on the fly, vs interpolating to add \
> them in to scale the image up. There is still some pxel replacement going on, but \
> generally making an image smaller is faster than scaling one up. Since we wanted to \
> be able to query for any image size, we concentrated on setting up Mapserver to \
> build from a tree index as fast as possible.
This is our mainstay setup at this point and how all services run in production mode. \
Our biggest bottleneck recently is network lag. I'm looking at the various tiling \
conventions as well for caching, probably a dynamic on demand caching system would \
be best in the end, and there is no reason this couldn't be applied to our current \
setup as a front end to tile based clients.
One big reason we went with the architecture that we did, was to allow for high \
resolution importing for our AutoCAD based online customers for raster imagery and \
such. There were also some groups of layer data that would prove much to costly to \
implement as separately configured cartographic layers as web services, See this \
layer output for example:
http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property_p \
ublic.map?mapext=563896.8404047444%20154119.8027878841%20568990.907012015%20157551.40786712753&mapsize=1146%20772&mode=map \
( http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property \
_public.map?mapext=563896.8404047444%20154119.8027878841%20568990.907012015%20157551.40786712753&mapsize=1146%20772&mode=map \
) http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/propert \
y_public.map?mapext=565159.2094367743%20155007.330481188%20567467.6297618784%20156562.39199164204&mapsize=1146%20772&mode=map \
( http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property \
_public.map?mapext=565159.2094367743%20155007.330481188%20567467.6297618784%20156562.39199164204&mapsize=1146%20772&mode=map \
) http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/propert \
y_public.map?mapext=566363.9818826721%20153592.63001268354%20568111.2314970943%20154769.66029395218&mapsize=1146%20772&mode=map \
( http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property \
_public.map?mapext=566363.9818826721%20153592.63001268354%20568111.2314970943%20154769.66029395218&mapsize=1146%20772&mode=map \
) http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/propert \
y_public.map?mapext=567200.2527496096%20153998.18707850928%20568073.8775568207%20154586.7022191436&mapsize=1146%20772&mode=map \
( http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property \
_public.map?mapext=567200.2527496096%20153998.18707850928%20568073.8775568207%20154586.7022191436&mapsize=1146%20772&mode=map \
)
These layers (there's 50 plus ion there for example) would be a bear to maintain. So \
I decided to cheat and use a raster presentation layer instead that is generated \
automatically from our ACAD DWGs.
These were major considerations for our design of the system, and may not apply to \
your situation.
bobb
> > > Jason Birch <jason@jasonbirch.com> wrote:
I've often wondered if power-of-two was the best approach from a
perception viewpoint. It definitely makes the most sense from a code
perspective. Anyone know of any research on this?
On 2010-05-22, Christopher Schmidt <crschmidt@crschmidt.net> wrote:
> On Fri, May 21, 2010 at 06:24:26PM -0300, Fabio Renzo Panettieri wrote:
> > On Fri, 2010-05-21 at 13:17 -0700, Karsten-3-2 wrote:
> > > Yes. What I want to do is simply to find out the fastest options to
> > > render on
> > > the fly from raw data imagery
> > > (no tiles whatsoever stored on disk in addition to the raw data ). I
> > > will
> > > check out what SpatialCache is...
>
> > From raw aerial imagery:
> 1. Store everything as uncompressed tiffs.
> 2. Make images as large as possible. (This probably requires BigTIFF
> support.)
> 3. Use overviews -- usually one for every power-of-two level from the base
> image up to the point where you have 256 x 256 overviews
> (gdaladdo)
> 4. If you have too many images to make one large image practical, create
> one reduced size image that you use at lower zoom levels.
>
> All of this is based around serving with MapServer. I have no experience
> using other imagery servers to solve this problem.
>
> Regards,
> --
> Christopher Schmidt
> Web Developer
> _______________________________________________
> Discuss mailing list
> Discuss@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
_______________________________________________
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
[Attachment #5 (text/html)]
<html>
<head>
<style type="text/css">
<!--
body { margin-bottom: 1px; margin-left: 4px; font-variant: normal; \
margin-top: 4px; line-height: normal; margin-right: 4px } p { margin-bottom: 0; \
margin-top: 0 }
-->
</style>
</head>
<body style="margin-bottom: 1px; margin-left: 4px; margin-top: 4px; margin-right: \
4px"> <p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">All,</font> </p>
<br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">Related to this . . .</font> </p>
<br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">From our perspective, which was to \
allow for the most flexibilty with regard to queries in the size of the images, \
we were willing to sacrifice the highest level of performance. Most of the \
effort here was in organizing the tiles (tree index) behind the MapServer \
service. Setting up the Mapfiles to generally drop pixels vs interpolating them \
(adding new ones) was a big piece of the performance issue in our \
configuration for example. Simply setting the MIN/MAX scale params accordingly \
helped a great deal. MapServer (and most other products BTW), seems \
to be better at dropping pixels (reducing the image size) on the fly, vs \
interpolating to add them in to scale the image up. There is still some pxel \
replacement going on, but generally making an image smaller is faster than \
scaling one up. Since we wanted to be able to query for any image size, we \
concentrated on setting up Mapserver to build from a tree index as fast as \
possible.</font> </p> <br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">This is our mainstay setup at this point \
and how all services run in production mode. Our biggest bottleneck recently is \
network lag. I'm looking at the various tiling conventions as well for \
caching, probably a dynamic on demand caching system would be best in the \
end, and there is no reason this couldn't be applied to our current setup as \
a front end to tile based clients.</font> </p> <br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">One big reason we went with the \
architecture that we did, was to allow for high resolution importing for our \
AutoCAD based online customers for raster imagery and such. There were also \
some groups of layer data that would prove much to costly to implement as separately \
configured cartographic layers as web services, See this layer output for \
example:</font> </p> <br>
<p style="margin-top: 0; margin-bottom: 0">
<font color="#0000ff" size="3" face="Comic Sans MS"><i><u><a \
href="http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/prop \
erty_public.map?mapext=563896.8404047444%20154119.8027878841%20568990.907012015%201575 \
51.40786712753&mapsize=1146%20772&mode=map">http://gis.ci.stpaul.mn.us/datasets/RASTER \
/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property_public.map?mapext=563896.8404047444 \
%20154119.8027878841%20568990.907012015%20157551.40786712753&mapsize=1146%20772&mode=map</a></u></i></font><font \
size="3" face="Comic Sans MS"></font> </p> <p style="margin-top: 0; \
margin-bottom: 0"> <font color="#0000ff" size="3" face="Comic Sans MS"><i><u><a \
href="http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/prop \
erty_public.map?mapext=565159.2094367743%20155007.330481188%20567467.6297618784%201565 \
62.39199164204&mapsize=1146%20772&mode=map">http://gis.ci.stpaul.mn.us/datasets/RASTER \
/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property_public.map?mapext=565159.2094367743 \
%20155007.330481188%20567467.6297618784%20156562.39199164204&mapsize=1146%20772&mode=map</a></u></i></font><font \
size="3" face="Comic Sans MS"></font> </p> <p style="margin-top: 0; \
margin-bottom: 0"> <font color="#0000ff" size="3" face="Comic Sans MS"><i><u><a \
href="http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/prop \
erty_public.map?mapext=566363.9818826721%20153592.63001268354%20568111.2314970943%2015 \
4769.66029395218&mapsize=1146%20772&mode=map">http://gis.ci.stpaul.mn.us/datasets/RAST \
ER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property_public.map?mapext=566363.98188267 \
21%20153592.63001268354%20568111.2314970943%20154769.66029395218&mapsize=1146%20772&mode=map</a></u></i></font><font \
size="3" face="Comic Sans MS"></font> </p> <p style="margin-top: 0; \
margin-bottom: 0"> <font color="#0000ff" size="3" face="Comic Sans MS"><i><u><a \
href="http://gis.ci.stpaul.mn.us/datasets/RASTER/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/prop \
erty_public.map?mapext=567200.2527496096%20153998.18707850928%20568073.8775568207%2015 \
4586.7022191436&mapsize=1146%20772&mode=map">http://gis.ci.stpaul.mn.us/datasets/RASTE \
R/SAINT_PAUL/PUBLIC_WORKS/PROPERTY/property_public.map?mapext=567200.252749609 \
6%20153998.18707850928%20568073.8775568207%20154586.7022191436&mapsize=1146%20772&mode=map</a></u></i></font><font \
size="3" face="Comic Sans MS"></font> </p> <br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">These layers (there's 50 plus ion \
there for example) would be a bear to maintain. So I decided to cheat and use a \
raster presentation layer instead that is generated automatically from our ACAD \
DWGs.</font> </p> <br>
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">These were major considerations for our \
design of the system, and may not apply to your situation.</font> </p> <br> \
<p style="margin-top: 0; margin-bottom: 0">
<font size="3" face="Comic Sans MS">bobb</font> </p>
<br>
<p style="margin-top: 0; margin-bottom: 0">
<br>
<br>
>>> Jason Birch <jason@jasonbirch.com> wrote:<br> </p>
<div style="margin-bottom: 0; margin-left: 15px; margin-top: 0; padding-left: \
7px; background-color: #f3f3f3; margin-right: 0; border-left: solid 1px #050505"> <p \
style="margin-top: 0; margin-bottom: 0"> I've often wondered if power-of-two was \
the best approach from a<br>perception viewpoint. It definitely makes the most sense \
from a code<br>perspective. Anyone know of any research on this?<br><br>On \
2010-05-22, Christopher Schmidt <crschmidt@crschmidt.net> wrote:<br>> On \
Fri, May 21, 2010 at 06:24:26PM -0300, Fabio Renzo Panettieri \
wrote:<br>>> On Fri, 2010-05-21 at 13:17 -0700, Karsten-3-2 \
wrote:<br>>> > Yes. What I want to do is simply to find out the fastest \
options to<br>>> > render on<br>>> > the fly from raw data \
imagery<br>>> > (no tiles whatsoever  stored on disk in \
addition to the raw data ). I<br>>> > will<br>>> > check out \
what SpatialCache is...<br>><br>> >From raw aerial \
imagery:<br>>  1. Store everything as uncompressed \
tiffs.<br>>  2. Make images as large as possible. (This probably \
requires BigTIFF<br>> support.)<br>>  3. Use overviews -- usually \
one for every power-of-two level from the \
base<br>>     image up to the point where you have 256 x \
256 overviews<br>>     (gdaladdo)<br>>  4. \
If you have too many images to make one large image practical, \
create<br>>     one reduced size image that you use at \
lower zoom levels.<br>><br>> All of this is based around serving with \
MapServer. I have no experience<br>> using other imagery servers to solve this \
problem.<br>><br>> Regards,<br>> --<br>> Christopher Schmidt<br>> \
Web Developer<br>> _______________________________________________<br>> Discuss \
mailing list<br>> Discuss@lists.osgeo.org<br>> <a \
href="http://lists.osgeo.org/mailman/listinfo/discuss">http://lists.osgeo.org/mailman/ \
listinfo/discuss</a><br>><br>_______________________________________________<br>Discuss \
mailing list<br>Discuss@lists.osgeo.org<br><a \
href="http://lists.osgeo.org/mailman/listinfo/discuss">http://lists.osgeo.org/mailman/listinfo/discuss</a><br>
</p>
</div>
</body>
</html>
_______________________________________________
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic