[prev in list] [next in list] [prev in thread] [next in thread] 

List:       solr-user
Subject:    Re: Index Segments not Merging
From:       Mike Thomsen <mikerthomsen () gmail ! com>
Date:       2017-02-27 13:22:48
Message-ID: CALAQ4HtT9-Bm=VvKP8Bmfucs6P_+Ru-V5hhBoaTV85UBEWdD6g () mail ! gmail ! com
[Download RAW message or body]


Just barely skimmed the documentation, but it looks like the tool generates
its own shards and pushes them into the collection by manipulating the
configuration of the cluster.

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/search_mapreduceindexertool.html

If that reading is correct, it would stand to reason that Solr (at least as
of Solr 4.10 which is what CDH ships) would not be doing the periodic
cleanup it normally does when building shards through its APIs.

On Thu, Feb 23, 2017 at 10:01 PM, Jordan Drake <jordan.drake@exterro.com>
wrote:

> We have solr with the index stored in HDFS. We are running MapReduce jobs
> to build the index using the MapReduceIndexerTool from Cloudera with the
> go-live option to merge into our live index.
>
> We are seeing an issue where the number of segments in the index never
> reduces. It continues to grow until we manually do an optimize.
>
> We are using the following solr config for merge policy
>
>
>
>
>
>
>
>
>
>
>
> *<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">    <int
> name="maxMergeAtOnce">10</int>    <int
> name="segmentsPerTier">10</int></mergePolicy><!--<
> mergeFactor>10</mergeFactor>--><mergeScheduler
> class="org.apache.lucene.index.ConcurrentMergeScheduler">    <int
> name="maxThreadCount">1</int>    <int
> name="maxMergeCount">6</int></mergeScheduler>*
>
> If we add documents into solr without using MapReduce the segments merge
> properly as expected.
>
> Any ideas on why we see this behavior? Does the solr index merge prevent
> the segments from merging?
>
>
> Thanks,
> Jordan
>


[prev in list] [next in list] [prev in thread] [next in thread]