[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] Testing CephFS
From:       gfarnum () redhat ! com (Gregory Farnum)
Date:       2015-08-31 12:02:15
Message-ID: CAJ4mKGaZb32fgvrrW6XWByoKff=hyzxizvhK-CLAaUyCtXfbsQ () mail ! gmail ! com
[Download RAW message or body]

On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng <ukernel at gmail.com> wrote:
> On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum <gfarnum at redhat.com> wrote:
> > On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam <sha at pml.ac.uk> wrote:
> > > Hi Greg,
> > > 
> > > The MDS' detect that the other one went down and started the replay.
> > > 
> > > I did some further testing with 20 client machines. Of the 20 client machines, \
> > > 5 hung with the following error: 
> > > [Aug24 10:53] ceph: mds0 caps stale
> > > [Aug24 10:54] ceph: mds0 caps stale
> > > [Aug24 10:58] ceph: mds0 hung
> > > [Aug24 11:03] ceph: mds0 came back
> > > [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
> > > [  +0.000018] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
> > > [Aug24 11:04] ceph: mds0 reconnect start
> > > [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
> > > [  +0.008475] ceph: mds0 reconnect denied
> > 
> > Oh, this might be a kernel bug, failing to ask for mdsmap updates when
> > the connection goes away. Zheng, does that sound familiar?
> > -Greg
> > 
> 
> I reproduced this locally (use SIGSTOP to stop the monitor) . I think
> the root cause is that kernel client does not implement
> CEPH_FEATURE_MSGR_KEEPALIVE2. So the kernel client couldn't reliably
> detect the event that network cable got unplugged. It kept waiting for
> new events from the disconnected connection.

Yeah, the userspace client maintains an ongoing MDSMap subscription
from the monitors in order to hear about this. It puts more load on
the monitors but right now that's the solution we're going with: the
monitor times out the MDS, publishes a series of new maps (pushed to
the clients) in order to activate a standby, and the clients see that
they need to connect to the new MDS instance.
-Greg


[prev in list] [next in list] [prev in thread] [next in thread]