[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-ha
Subject:    Re: [Linux-HA] crm_resource -P strange behavior.
From:       Andrew Beekhof <abeekhof () suse ! de>
Date:       2006-05-26 8:29:42
Message-ID: 54727881-BA41-4E00-8DF5-13BBA915A44B () suse ! de
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


On May 26, 2006, at 10:24 AM, Simone Gotti wrote:

> Hi Andrew,
>
> On Fri, 2006-05-26 at 09:22 +0200, Andrew Beekhof wrote:
>>
>> On May 25, 2006, at 8:07 PM, Simone Gotti wrote:
>>
>>> Hi, (I wrongly sent the same mail to linux-ha-dev, sorry...)
>>
>>
>> np
>>
>>
>>> I was testing the heartbeat behavoir with some tests.
>>>
>>>
>>> When I launched the crm_resource -P to probe for some resources
>>> started
>>> outside of the CRM (also if no one was started out of crm) I noticed
>>> that the cluster failed the probe trying it for an infinite number
>>> of
>>> times. Looks like the OCF IPaddr monitor function returned with a
>>> ret code of 0
>>> (OCF_SUCCESS, right as the resource is up) but tengine doesn't like
>>> it
>>> as it's expecting a value of 7 (OCF_NOT_RUNNING):
>>>
>>>
>>> tengine[10558]: 2006/05/25_16:32:38 ERROR:
>>> mask(events.c:match_graph_event): Action ipaddr01_monitor_0 on
>>> nodo01
>>> failed (target: 7 vs. rc: 0): Error
>>
>>
>> thats the probe detecting an active resource.
>
> But why is it expecting a target of 7 also if heartbeat knows that is
> should be up on this node (see also below)?

it doesn't... the fact that you're telling it to re-check means that  
all bets are off and what it thinks the cluster looks like is likely  
to be wrong.

>
>
>>> This happened also with other resource types not only with IPaddr
>>>
>>>
>>> I attached the ha-debug log file, with the cluster just started only
>>> on
>>> one node (this happens also at least with 2 nodes), and then at
>>> 2006/05/25_17:08:19 I launched the crm_resource -P command; the
>>> cibadmin
>>> -Q after the command, the ondisk cib.xml and the ha.cf.
>>>
>>>
>>> The unique way I found to stop this to happen is an heartbeat
>>> restart.
>>>
>>>
>>> Is this the right behavior, or am I doing something wrong?
>>
>>
>> if you're expecting resources to be detected as active then its  
>> normal
>>
>>
>> one could argue that this should be a warning but the most common  
>> case
>> for this log is when resources are active at startup which is more
>> important.
>
> Probably I'm missing something, but what I don't understand is why  
> it's
> going in an infinite and unstoppable loop when it detects the resource
> up.

oooooh, yes you're right.  it will do that.  sigh, thats a pretty  
serious bug too.
i'll work on it today (with the help of your comprehensive initial  
email)

> My assumption was that it detects the resource up, like it should be
> as the cluster started it on this node, and then he's happy and the
> probe is successfull.
>
> On the other side, if, for example, someone started by hand a resource
> on another node, than it should detect it up, but it should be down  
> and
> then heartbeat handle this situation stopping it. (I know that this
> should never happen, but this is happening in the real world as there
> are operators that don't know how the environment is configured, and
> other clusters suites, handle this possibility).
>
>>
>>
>> we could assume that the resources are already active, but then you'd
>> see similar log messages for the resources that weren't running.
>
>
>>
>>
>> hope that helps
>
> Thanks!
>
> Bye!
>>
>> --
>>
>> Andrew Beekhof
>>
>>
>> "Would the last person to leave please turn out the enlightenment?" -
>> TISM
>>
>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>
>
>
>  --
>  Email.it, the professional e-mail, gratis per te: http:// 
> www.email.it/f
>
>  Sponsor:
>  Corso multimediale sul Controllo di Gestione: impara facilmente  
> come ridurre i costi e aumentare gli utili della tua attivita'
>  Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=5055&d=26-5
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems

--
Andrew Beekhof

"Would the last person to leave please turn out the enlightenment?" -  
TISM


[Attachment #5 (unknown)]

<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: \
after-white-space; "><BR><DIV><DIV>On May 26, 2006, at 10:24 AM, Simone Gotti \
wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi \
Andrew,</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Fri, 2006-05-26 at \
09:22 +0200, Andrew Beekhof wrote:</DIV> <BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">On May 25, 2006, at 8:07 PM, Simone Gotti \
wrote:</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">Hi, (I wrongly sent the same mail to linux-ha-dev, sorry...)</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">np</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I was testing the \
heartbeat behavoir with some tests.</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">When I launched the crm_resource -P to probe \
for some resources</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">started</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">outside of the CRM (also \
if no one was started out of crm) I noticed</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">that the cluster failed \
the probe trying it for an infinite number</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">of</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">times. Looks like the OCF IPaddr monitor function returned with a</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">ret code of 0</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">(OCF_SUCCESS, right as the resource is up) but tengine \
doesn't like</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">it</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">as it's expecting a value of 7 \
(OCF_NOT_RUNNING):</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">tengine[10558]: 2006/05/25_16:32:38 \
ERROR:</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">mask(events.c:match_graph_event): Action ipaddr01_monitor_0 \
on</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">nodo01</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">failed (target: 7 vs. rc: 0): Error</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">thats the probe detecting an active resource.</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">But why is it expecting a \
target of 7 also if heartbeat knows that is</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">should be up on this node \
(see also below)?</DIV></BLOCKQUOTE><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>it doesn't... the fact that you're telling \
it to re-check means that all bets are off and what it thinks the cluster looks like \
is likely to be wrong.</DIV><BR><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><BLOCKQUOTE \
type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">This happened also with other resource types not only with \
IPaddr</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">I attached the ha-debug log file, with the cluster just started \
only</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">on</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">one node (this happens also at least with 2 \
nodes), and then at</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">2006/05/25_17:08:19 I launched the \
crm_resource -P command; the</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">cibadmin</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-Q after the command, the \
ondisk cib.xml and the ha.cf.</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">The unique way I found to stop this to happen \
is an heartbeat</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">restart.</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">Is this the right behavior, or am I doing \
something wrong?</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">if you're expecting resources to be detected \
as active then its normal</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">one could argue that this should be a warning \
but the most common case</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">for this log is when resources are active at \
startup which is more</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">important.</DIV> </BLOCKQUOTE><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">Probably I'm missing something, but what I \
don't understand is why it's</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">going in an infinite and unstoppable loop \
when it detects the resource</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">up.<BR></DIV></BLOCKQUOTE><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>oooooh, yes you're right.  it will do \
that.  sigh, thats a pretty serious bug too.</DIV><DIV>i'll work on it today (with \
the help of your comprehensive initial email)</DIV><BR><BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> \
My assumption was that it detects the resource up, like it should be</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">as \
the cluster started it on this node, and then he's happy and the</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">probe is successfull.</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On \
the other side, if, for example, someone started by hand a resource</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">on \
another node, than it should detect it up, but it should be down and</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">then heartbeat handle this situation stopping it. (I know that this</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">should never happen, but this is happening in the real world as there</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">are operators that don't know how the environment is configured, and</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">other clusters suites, handle this possibility).</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">we could assume that the resources are \
already active, but then you'd</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">see similar log messages for the resources \
that weren't running.</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">hope that helps</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks!</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">Bye!</DIV> <BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">--</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">Andrew Beekhof</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">"Would the last person to leave please turn \
out the enlightenment?" -</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">TISM</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">_______________________________________________</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">Linux-HA mailing list</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; "><A \
href="mailto:Linux-HA@lists.linux-ha.org">Linux-HA@lists.linux-ha.org</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><A \
href="http://lists.linux-ha.org/mailman/listinfo/linux-ha">http://lists.linux-ha.org/mailman/listinfo/linux-ha</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">See also: <A href="http://linux-ha.org/ReportingProblems">http://linux-ha.org/ReportingProblems</A></DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN \
class="Apple-converted-space"> </SPAN>--</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN \
class="Apple-converted-space"> </SPAN>Email.it, the professional e-mail, gratis per \
te: <A href="http://www.email.it/f">http://www.email.it/f</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; "><SPAN class="Apple-converted-space"> \
</SPAN>Sponsor:</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: \



_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

[prev in list] [next in list] [prev in thread] [next in thread]