'[dav-dev] Large Collections & Response Times'

[prev in list] [next in list] [prev in thread] [next in thread] 

List: dav-dev
Subject: [dav-dev] Large Collections & Response Times
From: "Hendler, Josh" <Josh.Hendler () mlb ! com>
Date: 2002-11-07 16:51:13
[Download RAW message or body]

Greetings,

I'm currently in the process of investigating a large-scale DAV
Implementation using mod_dav running under Apache 2.

While most everything has been working well, I have been encountering some
degree of latency while dealing with large collections. For example, when
doing a PROPFIND on a collection with approximately 1000 resources within it
(with a depth of 1), Apache returns the request within about 9 seconds.
After the first request, it returns the request in about 7 seconds. While
this response time is not really a problem, things get a bit hairier as the
directory size goes up. When dealing with a directory of 10k files, the
response time goes up to between 70 seconds and 120 seconds. 

I'm somewhat curious about these results because of the variance between the
response times of the PROPFIND and the response time of a standard GET of
the directory. For example, the response time for the get of the directory
with 10k files is approximately 8-10 seconds, while the PROPFIND is
approximately 8-10x slower. 

I realize that using directories with such a large number of files is quite
problematic in the first place (old problem of inheriting bad solutions),
but I'm interested if anyone has any suggestions for how this could be tuned
to respond more quickly to such requests. Any help or comments regarding the
issue would be greatly appreciated.

Thanks so much

Josh Hendler

==============================================================================
"MLB Mail Domain" made the following
 annotations on 11/07/02 11:45:00
------------------------------------------------------------------------------------------------------------------------------------------------------------
 [INFO] -- Virus Manager:
No Viruses were detected in this message.

==============================================================================

[Attachment #3 (text/html)]

<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns="http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">

<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 10">
<meta name=Originator content="Microsoft Word 10">
<link rel=File-List href="cid:filelist.xml@01C28651.6B135FE0">
<!--[if gte mso 9]><xml>
 <o:OfficeDocumentSettings>
  <o:DoNotRelyOnCSS/>
 </o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
 <w:WordDocument>
  <w:SpellingState>Clean</w:SpellingState>
  <w:GrammarState>Clean</w:GrammarState>
  <w:DocumentKind>DocumentEmail</w:DocumentKind>
  <w:EnvelopeVis/>
  <w:Compatibility>
   <w:BreakWrappedTables/>
   <w:SnapToGridInCell/>
   <w:WrapTextWithPunct/>
   <w:UseAsianBreakRules/>
  </w:Compatibility>
  <w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
 </w:WordDocument>
</xml><![endif]-->
<style>
<!--
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{mso-style-parent:"";
	margin:0in;
	margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;
	text-underline:single;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	mso-style-noshow:yes;
	mso-ansi-font-size:10.0pt;
	mso-bidi-font-size:10.0pt;
	font-family:Arial;
	mso-ascii-font-family:Arial;
	mso-hansi-font-family:Arial;
	mso-bidi-font-family:Arial;
	color:windowtext;}
span.SpellE
	{mso-style-name:"";
	mso-spl-e:yes;}
span.GramE
	{mso-style-name:"";
	mso-gram-e:yes;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;
	mso-header-margin:.5in;
	mso-footer-margin:.5in;
	mso-paper-source:0;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 10]>
<style>
 /* Style Definitions */ 
 table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:10.0pt;
	font-family:"Times New Roman";}
</style>
<![endif]-->
</head>

<body lang=EN-US link=blue vlink=purple style='tab-interval:.5in'>

<div class=Section1>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Greetings,<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I'm currently in the process of investigating a
large-scale DAV Implementation using <span class=SpellE>mod_dav</span> running
under Apache 2.<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

While most everything has been working well, I have been
encountering some degree of latency while dealing with large collections. For
example, when doing a PROPFIND on a collection with approximately 1000
resources within it (with a depth of 1), Apache returns the request within
about 9 seconds. After the first request, it returns the request in about 7
seconds. While this response time is not really a problem, things get a bit
hairier as the directory size goes up. When dealing with a directory of 10k
files, the response time goes up to between 70 seconds and 120 seconds. \
<o:p></o:p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

I'm somewhat curious about these results because of
the variance between the response times of the PROPFIND and the response time
of a standard GET of the directory. For example, the response time for the get
of the directory with 10k files is approximately 8-10 seconds, while the
PROPFIND is approximately 8-10x slower. \
<o:p></o:p>

I realize that using directories with such a large number of files is quite
problematic in the first place (old problem of inheriting bad solutions), but I'm
interested if anyone has any suggestions for how this could be tuned to respond
more quickly to such requests. Any help or comments regarding the issue would
be greatly appreciated.<o:p></o:p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks so much<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Josh Hendler<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

<P>==============================================================================<br>
"MLB Mail Domain" made the following<br>
 annotations on 11/07/02 11:45:00<br>
------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
 [INFO] -- Virus Manager:<br>
No Viruses were detected in this message.<br>
<br>
==============================================================================<br>
</P>

_______________________________________________
dav-dev maillist  -  dav-dev@lyra.org
http://dav.lyra.org/mailman/listinfo/dav-dev

[prev in list] [next in list] [prev in thread] [next in thread] 