[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-devel
Subject:    mon load..
From:       Sage Weil <sage () inktank ! com>
Date:       2013-05-31 2:50:48
Message-ID: alpine.DEB.2.00.1305301817310.26396 () cobra ! newdream ! net
[Download RAW message or body]

I'm playing with Mark's cluster, where he is seeing high ceph-mon cpu 
utilization when he creates new big pools.  I'm able to fairly reliably 
reproduce a livelock where it is stuck checking is_readable on queued auth 
requests long enough that it times out on the election and has to start 
all over again.

I see two issues:

- The PaxosService stuff is pulling values directly out of leveldb, and 
that is slow in this case.  Not completely sure why (compaction in teh 
background?  who knows.)  But, it's also unnecessary.. except that there 
is currently not a notification of the PaxosService instances when the 
underlying data changes.  That most easily plugs into a fwe places in teh 
Paxos class and on startup, but the way the layering is structured it's 
not very clean.  Not sure what teh right way to fix this up is.. but I 
think we do want some sort of PaxosService::refresh() that tells us 
whenever things changed; it can be the one to call the child's 
pdate_from_paxos().

- We should be able to discard those auth messages (and others!) if the 
original connection they came from has disconnected.. which is normally 
will after the client disconnects after 3 seconds (by default).

There is also wip-mon-trim that will lower the trim periodicity (and 
compaction) for the paxos states; that ought to help some as well, but I 
haven't tried it yet...

sage

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
[prev in list] [next in list] [prev in thread] [next in thread]