[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lucene-dev
Subject:    Query performance with DateFilter
From:       Scott Stanley <sastanley3 () yahoo ! com>
Date:       2001-11-30 22:57:30
[Download RAW message or body]

I have found that searching with date filtering is much slower since
shifting from Lucene 1.1b to lucene 1.2 rc2 (basically from com.lucene
to org.apache.lucene).
 
With 1.1b, search time was : 700ms
With 1.2rc2 : 11,000 ms!
(15 times slower)
(with  50,000 files indexed)
 
However, searching  with no filtering seems to be a bit faster with
1.2rc2.
 
To be sure  that the DateFilter was responsible for the performance
hit, I tested this:
 
    DateFilter df = new DateFilter("DOC_DATE", 1000087883595L,
                                   1009087883595L)
    BitSet bs = df.bits(IndexReader.open("/index");
 
With Lucene 1.1b : 668 ms
With Lucene 1.2 rc2 : 9000 ms
 
Running this under JProbe, I noticed that the performance difference
was coming from the call to SegmentTermDocs.next().  This method call
seems to be much slower because InputStream.readByte() is slower...
 
I noticed that InputStream.refill() and InputStream.readInternal() take
much more time.  I finally narrowed down to
RandomAccessFile.read(byte[], int, int) which is called around 50 times
more often in 1.2 RC2  than in the earlier version.
 
Is there an issue with the way FSDirectory handles bufferization of the
bytes read from the index files?  Is all of this related to the Thread
Safety fix?   I guess the bottom line is,  is there anything we can do
to bring the performance back up with the DateFilter? 

Scott

__________________________________________________
Do You Yahoo!?
Buy the perfect holiday gifts at Yahoo! Shopping.
http://shopping.yahoo.com

--
To unsubscribe, e-mail:   <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

[prev in list] [next in list] [prev in thread] [next in thread]