[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lucene-user
Subject:    Re: Question about threading in search
From:       Erick Erickson <erickerickson () gmail ! com>
Date:       2018-08-17 14:58:49
Message-ID: CAN4YXvdb5_4DHQgkxvZZNZpeAf2h+egFu0AzMs+HdRs2w3dS2A () mail ! gmail ! com
[Download RAW message or body]

Please don't optimize to 1 segment unless you can afford to do it
quite regularly, see:
https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

(NOTE: this is changing as of 7.5, see:
https://lucidworks.com/2018/06/20/solr-and-optimizing-your-index-take-ii/).

bq. It seems that search is sometimes faster with multiple segments.

In addition to what Toke said, this may just be an autowarming
problem. Measurements mean little/nothing unless they're performed on
a warmed-up index since there's quite a bit of reading from disk into
the heap and OS memory space that's required. You may just be seeing
that.

Best,
Erick


On Fri, Aug 17, 2018 at 2:26 AM, Toke Eskildsen <toes@kb.dk> wrote:
> On Sat, 2017-09-02 at 18:33 -0700, Peilin Yang wrote:
>> we're comparing two different indexes on the same collection - one
>> with lots of different segments (default settings), and one with a
>> force merged into one segment. It seems that search is sometimes
>> faster with multiple segments.
>
> If you are using Lucene 7+ and if some of the fields you are requesting
> as part of your search result are stored as DocValues, you might have
> encountered a performance regression with the streaming API:
> https://issues.apache.org/jira/browse/LUCENE-8374
>
> One peculiar effect of this issue is that fewer larger segments gets
> slower DocValues retrieval, compared to more smaller segments. So a
> force merge to 1 segment can result in worse performance.
>
> - Toke Eskildsen, the Royal Danish Library, Denmark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

[prev in list] [next in list] [prev in thread] [next in thread]