[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-ha
Subject:    Re: [Linux-HA] crm_resource -P strange behavior.
From:       Andrew Beekhof <abeekhof () suse ! de>
Date:       2006-05-26 15:02:21
Message-ID: CE283DA6-53D1-47E2-AF14-FE5E0F77101C () suse ! de
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


On May 26, 2006, at 10:29 AM, Andrew Beekhof wrote:

>
> On May 26, 2006, at 10:24 AM, Simone Gotti wrote:
>
>> Hi Andrew,
>>
>> On Fri, 2006-05-26 at 09:22 +0200, Andrew Beekhof wrote:
>>>
>>> On May 25, 2006, at 8:07 PM, Simone Gotti wrote:
>>>
>>>> Hi, (I wrongly sent the same mail to linux-ha-dev, sorry...)
>>>
>>>
>>> np
>>>
>>>
>>>> I was testing the heartbeat behavoir with some tests.
>>>>
>>>>
>>>> When I launched the crm_resource -P to probe for some resources
>>>> started
>>>> outside of the CRM (also if no one was started out of crm) I  
>>>> noticed
>>>> that the cluster failed the probe trying it for an infinite number
>>>> of
>>>> times. Looks like the OCF IPaddr monitor function returned with a
>>>> ret code of 0
>>>> (OCF_SUCCESS, right as the resource is up) but tengine doesn't like
>>>> it
>>>> as it's expecting a value of 7 (OCF_NOT_RUNNING):
>>>>
>>>>
>>>> tengine[10558]: 2006/05/25_16:32:38 ERROR:
>>>> mask(events.c:match_graph_event): Action ipaddr01_monitor_0 on
>>>> nodo01
>>>> failed (target: 7 vs. rc: 0): Error
>>>
>>>
>>> thats the probe detecting an active resource.
>>
>> But why is it expecting a target of 7 also if heartbeat knows that is
>> should be up on this node (see also below)?
>
> it doesn't... the fact that you're telling it to re-check means  
> that all bets are off and what it thinks the cluster looks like is  
> likely to be wrong.
>

This has now been addressed in CVS

To avoid infinite loops we now make sure *all* scheduled probes are  
run and the "this node needs to be rechecked" indicator is reset.

I have also made a change such that probes for resources that we  
think are active expect 0 (running) instead of 7 (not running).

People would be advised NOT to use "crm_resource -P" until either  
grabbing the latest from CVS or 2.0.6 comes out (no i dont know when  
that will be but "soonish" seems like a good idea).

A big thank-you to Simone for finding and reporting this! (especially  
with SLES10 on its way)

>>
>>
>>>> This happened also with other resource types not only with IPaddr
>>>>
>>>>
>>>> I attached the ha-debug log file, with the cluster just started  
>>>> only
>>>> on
>>>> one node (this happens also at least with 2 nodes), and then at
>>>> 2006/05/25_17:08:19 I launched the crm_resource -P command; the
>>>> cibadmin
>>>> -Q after the command, the ondisk cib.xml and the ha.cf.
>>>>
>>>>
>>>> The unique way I found to stop this to happen is an heartbeat
>>>> restart.
>>>>
>>>>
>>>> Is this the right behavior, or am I doing something wrong?
>>>
>>>
>>> if you're expecting resources to be detected as active then its  
>>> normal
>>>
>>>
>>> one could argue that this should be a warning but the most common  
>>> case
>>> for this log is when resources are active at startup which is more
>>> important.
>>
>> Probably I'm missing something, but what I don't understand is why  
>> it's
>> going in an infinite and unstoppable loop when it detects the  
>> resource
>> up.
>
> oooooh, yes you're right.  it will do that.  sigh, thats a pretty  
> serious bug too.
> i'll work on it today (with the help of your comprehensive initial  
> email)
>
>> My assumption was that it detects the resource up, like it should be
>> as the cluster started it on this node, and then he's happy and the
>> probe is successfull.
>>
>> On the other side, if, for example, someone started by hand a  
>> resource
>> on another node, than it should detect it up, but it should be  
>> down and
>> then heartbeat handle this situation stopping it. (I know that this
>> should never happen, but this is happening in the real world as there
>> are operators that don't know how the environment is configured, and
>> other clusters suites, handle this possibility).
>>
>>>
>>>
>>> we could assume that the resources are already active, but then  
>>> you'd
>>> see similar log messages for the resources that weren't running.
>>
>>
>>>
>>>
>>> hope that helps
>>
>> Thanks!
>>
>> Bye!
>>>
>>> --
>>>
>>> Andrew Beekhof
>>>
>>>
>>> "Would the last person to leave please turn out the  
>>> enlightenment?" -
>>> TISM
>>>
>>>
>>>
>>> _______________________________________________
>>> Linux-HA mailing list
>>> Linux-HA@lists.linux-ha.org
>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>>> See also: http://linux-ha.org/ReportingProblems
>>
>>
>>
>>  --
>>  Email.it, the professional e-mail, gratis per te: http:// 
>> www.email.it/f
>>
>>  Sponsor:
>>  Corso multimediale sul Controllo di Gestione: impara facilmente  
>> come ridurre i costi e aumentare gli utili della tua attivita'
>>  Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=5055&d=26-5
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>
> --
> Andrew Beekhof
>
> "Would the last person to leave please turn out the enlightenment?"  
> - TISM
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems

--
Andrew Beekhof

"Would the last person to leave please turn out the enlightenment?" -  
TISM


[Attachment #5 (unknown)]

<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: \
after-white-space; "><BR><DIV><DIV>On May 26, 2006, at 10:29 AM, Andrew Beekhof \
wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE \
type="cite"><BR><DIV><DIV>On May 26, 2006, at 10:24 AM, Simone Gotti wrote:</DIV><BR \
class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><DIV style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi Andrew,</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">On Fri, 2006-05-26 at 09:22 +0200, Andrew \
Beekhof wrote:</DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">On May 25, 2006, at 8:07 PM, Simone Gotti wrote:</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi, (I wrongly sent the \
same mail to linux-ha-dev, sorry...)</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">np</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE \
type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">I was testing the heartbeat behavoir with some tests.</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">When I launched the crm_resource -P to probe for some resources</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">started</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">outside of the CRM (also if no one was started out of crm) I \
noticed</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">that the cluster failed the probe trying it for an infinite \
number</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">of</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">times. Looks like the OCF IPaddr monitor \
function returned with a</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">ret code of 0</DIV><DIV style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">(OCF_SUCCESS, right \
as the resource is up) but tengine doesn't like</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">it</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">as \
it's expecting a value of 7 (OCF_NOT_RUNNING):</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">tengine[10558]: \
2006/05/25_16:32:38 ERROR:</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">mask(events.c:match_graph_event): Action \
ipaddr01_monitor_0 on</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">nodo01</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">failed (target: 7 vs. rc: \
0): Error</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">thats the probe detecting an active \
resource.</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">But why is it expecting a target of 7 also if heartbeat knows that is</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">should be up on this node (see also below)?</DIV></BLOCKQUOTE><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>it doesn't... the fact that you're telling \
it to re-check means that all bets are off and what it thinks the cluster looks like \
is likely to be wrong.</DIV><BR></DIV></BLOCKQUOTE><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>This has now been addressed in \
CVS</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>To avoid infinite loops \
we now make sure *all* scheduled probes are run and the "this node needs to be \
rechecked" indicator is reset.</DIV><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>I have also made a change such that probes \
for resources that we think are active expect 0 (running) instead of 7 (not \
running).</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>People would be \
advised NOT to use "crm_resource -P" until either grabbing the latest from CVS or \
2.0.6 comes out (no i dont know when that will be but "soonish" seems like a good \
idea).</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>A big thank-you to \
Simone for finding and reporting this! (especially with SLES10 on its \
way)</DIV><BR><BLOCKQUOTE type="cite"><DIV><BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE \
type="cite"><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">This happened also with other resource types \
not only with IPaddr</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">I attached the ha-debug log file, with the \
cluster just started only</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">on</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">one node (this happens \
also at least with 2 nodes), and then at</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">2006/05/25_17:08:19 I \
launched the crm_resource -P command; the</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">cibadmin</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-Q \
after the command, the ondisk cib.xml and the ha.cf.</DIV><DIV style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">The unique way I found to \
stop this to happen is an heartbeat</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; ">restart.</DIV><DIV style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Is this the right \
behavior, or am I doing something wrong?</DIV> </BLOCKQUOTE><DIV style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">if you're expecting \
resources to be detected as active then its normal</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">one could argue that this \
should be a warning but the most common case</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">for this log is when \
resources are active at startup which is more</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">important.</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Probably I'm missing \
something, but what I don't understand is why it's</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">going in an infinite and \
unstoppable loop when it detects the resource</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">up.<BR></DIV></BLOCKQUOTE><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>oooooh, yes you're right.  it will do \
that.  sigh, thats a pretty serious bug too.</DIV><DIV>i'll work on it today (with \
the help of your comprehensive initial email)</DIV><BR><BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> \
My assumption was that it detects the resource up, like it should be</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">as \
the cluster started it on this node, and then he's happy and the</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">probe is successfull.</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On \
the other side, if, for example, someone started by hand a resource</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">on \
another node, than it should detect it up, but it should be down and</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">then heartbeat handle this situation stopping it. (I know that this</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">should never happen, but this is happening in the real world as there</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">are operators that don't know how the environment is configured, and</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">other clusters suites, handle this possibility).</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">we could assume that the resources are \
already active, but then you'd</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">see similar log messages for the resources \
that weren't running.</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">hope that helps</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks!</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">Bye!</DIV> <BLOCKQUOTE type="cite"><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">--</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">Andrew Beekhof</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">"Would the last person to leave please turn \
out the enlightenment?" -</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">TISM</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">_______________________________________________</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">Linux-HA mailing list</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; "><A \
href="mailto:Linux-HA@lists.linux-ha.org">Linux-HA@lists.linux-ha.org</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><A \
href="http://lists.linux-ha.org/mailman/listinfo/linux-ha">http://lists.linux-ha.org/mailman/listinfo/linux-ha</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">See also: <A href="http://linux-ha.org/ReportingProblems">http://linux-ha.org/ReportingProblems</A></DIV> \



_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

[prev in list] [next in list] [prev in thread] [next in thread]