[prev in list] [next in list] [prev in thread] [next in thread]
List: openais
Subject: [Openais] token->token_seq wrap around.
From: "Muni Bajpai" <muniba () nortel ! com>
Date: 2005-10-18 17:45:01
Message-ID: CFCE7C3BDB79204092974B5B50AD719401FC4530 () zrc2hxm0 ! corp ! nortel ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hey Steve,
Our verification team was running traffic on 5 nodes this last week till
Monday morning (after the cluster had been up for 10 days) all 5 nodes
stopped receiving multicast messages (all 5 of them). The way we know
this is because we have a timer that pops if an EVT message is not
received after 1 second and we know the publisher of the EVT message was
definitely publishing messages. Unfortunately there were no logs being
sent to Stderr so we couldn't really see what the executive was doing.
The gdb trace revealed that the library was waiting indefinitely on ALL
nodes for a response to the request which was a Checkpoint Write. The
gdb trace for the executive showed that it was behaving normally i.e
aispoll was running and that it wasn't stuck like the library.
So looking through the failure paths It was possible that one of the
following happened
1.) The library couldn't send the request to the executive
2.) The executive didn't receive the request
3.) The executive couldn't mcast the request (or dropped it)
4.) The executive couldn't receive the mcast. (or ignored it)
5.) The executive couldn't send the response to the library.
So after considerable thought Kristen and I decided that 1,2,3,5 were
highly unlikely looking through the failure paths in the code, which
left us with 4.
4 was likely as there are decisions based on sequence numbers (cluster
wide) that the executive uses to discard or process messages and as this
happened on all the nodes that was the likely culprit.
So in the end we think it's the fact that the token->token_seq that is
incremented on outgoing mcasts is rolling over after it reaches
0x7fffffff (2147483647). The next increment is (-2147483648).
We ran a test where in totemsrp.c:2717 (trunk version) we set the
token->token_seq to a negative number when the token->token_seq reached
a value of 10000 (just to give us some time to bring up ckpt-wr) and
right when we reached that point the library hung and all mcasts were
ignored from there on .
I'm not sure how to address this other than checking for overflow before
we increment and reversing the state in all the sorted queues. The
former is easy as in the patch below but reversing the stored info keyed
off the sequence number is something I'm not sure.
This is a big issue as it immediately sends our WHOLE cluster out of
service defeating the purpose.
Thanks In advance
Muni
--- openais/trunk/exec/totemsrp.c 2005-10-18 11:24:28 -05:00
+++ trunk/exec/totemsrp.c 2005-10-18 12:13:46 -05:00
@@ -100,6 +100,7 @@
* do not change
*/
#define ENDIAN_LOCAL 0xff22
+#define MAX_INT_VALUE_BEFORE_ROLL_OVER 0x7fffffff
enum message_type {
MESSAGE_TYPE_ORF_TOKEN = 0, /* Ordering,
Reliability, Flow (ORF) control Token */
@@ -535,6 +536,18 @@
}
};
+int incrementSequenceCheckingRollOver(int *seq)
+{
+ assert(seq);
+ if (*seq >= MAX_INT_VALUE_BEFORE_ROLL_OVER)
+ {
+ *seq = 0;
+ return -1;
+ }
+ (*seq)++;
+ return 0;
+}
+
void totemsrp_instance_initialize (struct totemsrp_instance *instance)
{
memset (instance, 0, sizeof (struct totemsrp_instance));
@@ -2231,7 +2244,7 @@
int memb_index_this;
int memb_index_next;
- memb_commit_token->token_seq++;
+
incrementSequenceCheckingRollOver(&memb_commit_token->token_seq);
memb_index_this = (memb_commit_token->memb_index + 1) %
memb_commit_token->addr_entries;
memb_index_next = (memb_index_this + 1) %
memb_commit_token->addr_entries;
memb_commit_token->memb_index = memb_index_this;
@@ -2714,7 +2727,11 @@
memb_state_gather_enter (instance);
} else {
instance->my_token_seq = token->token_seq;
- token->token_seq += 1;
+ if
(incrementSequenceCheckingRollOver(&token->token_seq) < 0)
+ {
+ instance->my_token_seq =
token->token_seq;
+
incrementSequenceCheckingRollOver(&token->token_seq);
+ }
if (instance->memb_state == MEMB_STATE_RECOVERY)
{
/*
[Attachment #5 (text/html)]
<html DEFANGED_xmlns:o="urn:schemas-microsoft-com:office:office" \
DEFANGED_xmlns:w="urn:schemas-microsoft-com:office:word" \
DEFANGED_xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<DEFANGED_style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1568302633;
mso-list-type:hybrid;
mso-list-template-ids:985140944 -352260710 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l0:level1
{mso-level-text:"%1\.\)";
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</DEFANGED_style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">Hey Steve,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">Our verification team was running traffic on 5 nodes this
last week till Monday morning (after the cluster had been up for 10 days) all 5
nodes stopped receiving multicast messages (all 5 of them). The way we know
this is because we have a timer that pops if an EVT message is not received after
1 second and we know the publisher of the EVT message was definitely publishing
messages. Unfortunately there were no logs being sent to Stderr so we couldn’t
really see what the executive was doing.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">The gdb trace revealed that the library was waiting
indefinitely on ALL nodes for a response to the request which was a Checkpoint
Write. The gdb trace for the executive showed that it was behaving normally i.e
aispoll was running and that it wasn’t stuck like the \
library.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">So looking through the failure paths It was possible that
one of the following happened<o:p></o:p></span></font></p>
<p class="MsoNormal" DEFANGED_style="margin-left:.5in;text-indent:-.25in;mso-list:l0 \
level1 lfo1"><![if !supportLists]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial"><span \
DEFANGED_style="mso-list:Ignore">1.)<font size=1 face="Times New Roman"><span \
DEFANGED_style='font:7.0pt "Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial">The library couldn’t send \
the request to the executive<o:p></o:p></span></font></p>
<p class="MsoNormal" DEFANGED_style="margin-left:.5in;text-indent:-.25in;mso-list:l0 \
level1 lfo1"><![if !supportLists]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial"><span \
DEFANGED_style="mso-list:Ignore">2.)<font size=1 face="Times New Roman"><span \
DEFANGED_style='font:7.0pt "Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial">The executive didn’t \
receive the request<o:p></o:p></span></font></p>
<p class="MsoNormal" DEFANGED_style="margin-left:.5in;text-indent:-.25in;mso-list:l0 \
level1 lfo1"><![if !supportLists]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial"><span \
DEFANGED_style="mso-list:Ignore">3.)<font size=1 face="Times New Roman"><span \
DEFANGED_style='font:7.0pt "Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial">The executive couldn’t \
mcast the request (or dropped it)<o:p></o:p></span></font></p>
<p class="MsoNormal" DEFANGED_style="margin-left:.5in;text-indent:-.25in;mso-list:l0 \
level1 lfo1"><![if !supportLists]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial"><span \
DEFANGED_style="mso-list:Ignore">4.)<font size=1 face="Times New Roman"><span \
DEFANGED_style='font:7.0pt "Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial">The executive couldn’t \
receive the mcast. (or ignored it)<o:p></o:p></span></font></p>
<p class="MsoNormal" DEFANGED_style="margin-left:.5in;text-indent:-.25in;mso-list:l0 \
level1 lfo1"><![if !supportLists]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial"><span \
DEFANGED_style="mso-list:Ignore">5.)<font size=1 face="Times New Roman"><span \
DEFANGED_style='font:7.0pt "Times New Roman"'> \
</span></font></span></span></font><![endif]><font size=2 face=Arial><span \
DEFANGED_style="font-size:10.0pt;font-family:Arial">The executive couldn’t send \
the response to the library.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">So after considerable thought Kristen and I decided that 1,2,3,5
were highly unlikely looking through the failure paths in the code, which left
us with 4.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">4 was likely as there are decisions based on sequence numbers
(cluster wide) that the executive uses to discard or process messages and as
this happened on all the nodes that was the likely \
culprit.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">So in the end we think it’s the fact that the \
token->token_seq that is incremented on outgoing mcasts is rolling over \
after it reaches 0x7fffffff (2147483647). The next increment is (-2147483648). \
<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">We ran a test where in totemsrp.c:2717 (trunk version) we
set the token->token_seq to a negative number when the token->token_seq
reached a value of 10000 (just to give us some time to bring up ckpt-wr) and right
when we reached that point the library hung and all mcasts were ignored from
there on .<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">I’m not sure how to address this other than checking
for overflow before we increment and reversing the state in all the sorted
queues. The former is easy as in the patch below but reversing the stored info
keyed off the sequence number is something I’m not \
sure.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">This is a big issue as it immediately sends our WHOLE
cluster out of service defeating the purpose.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">Thanks In advance<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">Muni<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">---
openais/trunk/exec/totemsrp.c 2005-10-18
11:24:28 -05:00<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+++
trunk/exec/totemsrp.c 2005-10-18 12:13:46
-05:00<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">@@ -100,6 +100,7 @@<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> * do not change<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> */<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> #define
ENDIAN_LOCAL &n \
bsp;
0xff22<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+#define
MAX_INT_VALUE_BEFORE_ROLL_OVER
0x7fffffff<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> enum message_type {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
MESSAGE_TYPE_ORF_TOKEN =
0,
/* Ordering, Reliability, Flow (ORF) control Token */<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">@@ -535,6 +536,18 @@<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> \
}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> };<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+int incrementSequenceCheckingRollOver(int \
*seq)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+{<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
assert(seq);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ if (*seq >=
MAX_INT_VALUE_BEFORE_ROLL_OVER)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
{<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
*seq = 0;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
return -1;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
(*seq)++;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ return \
0;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> void totemsrp_instance_initialize (struct
totemsrp_instance *instance)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> memset (instance,
0, sizeof (struct totemsrp_instance));<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">@@ -2231,7 +2244,7 @@<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> int
memb_index_this;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> int
memb_index_next;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">-
memb_commit_token->token_seq++;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
incrementSequenceCheckingRollOver(&memb_commit_token->token_seq);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> memb_index_this =
(memb_commit_token->memb_index + 1) % \
memb_commit_token->addr_entries;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> memb_index_next =
(memb_index_this + 1) % \
memb_commit_token->addr_entries;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
memb_commit_token->memb_index = memb_index_this;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">@@ -2714,7 +2727,11 @@<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
memb_state_gather_enter (instance);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
} else {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
instance->my_token_seq = token->token_seq;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">-
token->token_seq += 1;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
if (incrementSequenceCheckingRollOver(&token->token_seq) < \
0)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
{<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
instance->my_token_seq = token->token_seq;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+ \
incrementSequenceCheckingRollOver(&token->token_seq);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">+
}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial">
if (instance->memb_state == MEMB_STATE_RECOVERY) {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"> & \
nbsp;
/*<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span DEFANGED_style="font-size:10.0pt;
font-family:Arial"><o:p> </o:p></span></font></p>
</div>
</body>
</html>
_______________________________________________
Openais mailing list
Openais@lists.osdl.org
https://lists.osdl.org/mailman/listinfo/openais
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic