[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ssic-linux-devel
Subject:    Re: [SSI-devel] Re: SSI-1.9.2 release drawing near
From:       Roger Tsang <roger.tsang () gmail ! com>
Date:       2005-10-06 17:24:09
Message-ID: 498263350510061024g60417c7du87b8c9b1822ee3b6 () mail ! gmail ! com
[Download RAW message or body]

On 10/6/05, Roger Tsang <roger.tsang@gmail.com> wrote:
>
> On 10/6/05, John Hughes <john@calva.com> wrote:
> >
> > Roger Tsang wrote:
> >
> > >
> > > > * workaround (not fix) for rare cfs_writepages() deadlock on
> > > LVM2
> > > > backed ext3 fs under load.
> > > >
> > > I would like to use LVM2 on my 1.9 cluster, could you give more
> > > details?
> > >
> > >
> > > If I remember right besides the usual steps to setup LVM, all I had t=
o
> > > do was tweak the rc.sysinit* scripts to setup LVM2 on condition that
> > > the LVM2 config directory is a CDSL.
> >
> > Interesting. Any details of what the deadlock problem was? What's the
> > workaround?
> >
>
> Writing to LVM2 could freeze up CFS under high IO. The workaround involve=
s
> hacking the Linux kernel code to detect and circumvent a potential deadlo=
ck.
>
> Roger
>

If you go into kdb when the livelock happens, you will see a lot of
processes that are writing to LVM2 stuck at cfs_writepages() and
wait_on_congestion(). Your cluster will slow down a lot and eventually
freeze up until you reboot the node(s) that are in this state.

[Attachment #3 (text/html)]

On 10/6/05, <b class="gmail_sendername">Roger Tsang</b> &lt;<a \
href="mailto:roger.tsang@gmail.com">roger.tsang@gmail.com</a>&gt; wrote:<div><span \
class="gmail_quote"></span><blockquote class="gmail_quote" style="border-left: 1px \
solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div><span \
class="e" id="q_106c6f008667ae88_0">On 10/6/05, <b class="gmail_sendername">John \
Hughes</b> &lt;<a href="mailto:john@calva.com" target="_blank" onclick="return \
top.js.OpenExtLink(window,event,this)">john@calva.com </a>&gt; wrote:<div><span \
class="gmail_quote"></span><blockquote class="gmail_quote" style="border-left: 1px \
solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Roger Tsang \
wrote:<br><br>&gt;<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt;&nbsp;&nbsp;&nbsp;&nbsp; * \
workaround (not fix) for rare cfs_writepages() deadlock \
on<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; LVM2<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; \
&gt;&nbsp;&nbsp;&nbsp;&nbsp; backed ext3 fs under \
load.<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt;<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; I would \
like to use LVM2 on my  1.9 cluster, could you give \
more<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp; details?<br>&gt;<br>&gt;<br>&gt; If I remember \
right besides the usual steps to setup LVM, all I had to<br>&gt; do was tweak the \
rc.sysinit* scripts to setup LVM2 on condition that <br>&gt; the LVM2 config \
directory is a CDSL.<br><br>Interesting.&nbsp;&nbsp;Any details of what the deadlock \
problem was?&nbsp;&nbsp;What's \
the<br>workaround?<br></blockquote></div><br></span></div> Writing to LVM2 could \
freeze up CFS under high IO.&nbsp; The workaround involves hacking the Linux kernel \
code to detect and circumvent a potential deadlock.<br><span class="sg">
<br>
Roger<br>

</span></blockquote></div><br>
If you go into kdb when the livelock happens, you will see a lot of
processes that are writing to LVM2 stuck at cfs_writepages() and
wait_on_congestion().&nbsp; Your cluster will slow down a lot and
eventually freeze up until you reboot the node(s) that are in this
state.<br>


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
ssic-linux-devel mailing list
ssic-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ssic-linux-devel

[prev in list] [next in list] [prev in thread] [next in thread]