[prev in list] [next in list] [prev in thread] [next in thread] 

List:       solr-dev
Subject:    [jira] [Commented] (SOLR-10277) On 'downnode', lots of wasteful mutations are done to ZK
From:       "Varun Thacker (JIRA)" <jira () apache ! org>
Date:       2017-03-31 18:04:41
Message-ID: JIRA.13055739.1489442149000.178552.1490983481643 () Atlassian ! JIRA
[Download RAW message or body]


    [ https://issues.apache.org/jira/browse/SOLR-10277?page=com.atlassian.jira.plugin. \
system.issuetabpanels:comment-tabpanel&focusedCommentId=15951407#comment-15951407 ] 

Varun Thacker commented on SOLR-10277:
--------------------------------------

Hi Scott,

Any comments on the last patch apart from addressing the nocommit ?

Joshua curious if you deployed this on your cluster and how much improvements did you \
see?

Regardless this should be a good candidate for Solr 6.5.1 as the fix is pretty safe. \
[~joel.bernstein] let me know if you're okay to commit this to the 6_5 branch as well

> On 'downnode', lots of wasteful mutations are done to ZK
> --------------------------------------------------------
> 
> Key: SOLR-10277
> URL: https://issues.apache.org/jira/browse/SOLR-10277
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public) 
> Components: SolrCloud
> Affects Versions: 5.5.3, 5.5.4, 6.0.1, 6.2.1, 6.3, 6.4.2
> Reporter: Joshua Humphries
> Assignee: Scott Blum
> Labels: leader, zookeeper
> Attachments: SOLR-10277-5.5.3.patch, SOLR-10277.patch
> 
> 
> When a node restarts, it submits a single 'downnode' message to the overseer's \
> state update queue. When the overseer processes the message, it does way more \
> writes to ZK than necessary. In our cluster of 48 hosts, the majority of \
> collections have only 1 shard and 1 replica. So a single node restarting should \
> only result in ~1/40th of the collections being updated with new replica states (to \
> indicate the node that is no longer active). However, the current logic in \
> NodeMutator#downNode always updates *every* collection. So we end up having to do \
> rolling restarts very slowly to avoid having a severe outage due to the overseer \
> having to do way too much work for each host that is restarted. And subsequent \
> shards becoming leader can't get processed until the `downnode` message is fully \
> processed. So a fast rolling restart can result in the overseer queue growing \
> incredibly large and nearly all shards winding up in a leader-less state until that \
> backlog is processed. The fix is a trivial logic change to only add a \
> ZkWriteCommand for collections that actually have an impacted replica.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]