[prev in list] [next in list] [prev in thread] [next in thread]
List: openais
Subject: RE: defect 932 fix rollover - WAS -> RE: [Openais]
From: Steven Dake <sdake () mvista ! com>
Date: 2005-10-31 19:56:57
Message-ID: 1130788617.13810.6.camel () unnamed ! az ! mvista ! com
[Download RAW message or body]
Muni,
There are several places where a value must be iterated from a low point
to a high point. In the case that the low point and high point are
greater then 0x7fffffff, we subtract 0x7fffffff. This then ensures that
the low value is always lower then the high value. Consider an example:
low is 0xfffffffe high is 0x00000002. This makes sense; what it says is
we iterate for 4 times for positions 0xfffffffe-0x00000002. But if we
iterated this in a for loop, the for loop would stop immeidately because
the start is greater then the end. There could be different ways to
itereate the for loop, for example determine how many entries to iterate
and then keep start and end points the same without adjustment.
The cutoff (0x7fffffff) could be any larger value as long as the range
would never wrap.
Finally all of the signed ints were changed to unsigned so the rollover
works properly.
Regards
-steve
On Mon, 2005-10-31 at 11:39 -0600, Muni Bajpai wrote:
> Steve,
>
> The patch ran fine with my adjusted values too. Could you give me a
> quick heads up on the algorithm for the change.
>
> Thanks
>
> Muni
>
> -----Original Message-----
> From: Steven Dake [mailto:sdake@mvista.com]
> Sent: Thursday, October 27, 2005 4:59 PM
> To: Bajpai, Muni [RICH1:B670:EXCH]
> Cc: scd@broked.org; openais@lists.osdl.org
> Subject: defect 932 fix rollover - WAS -> RE: [Openais]
> RE:token->token_seq wrap around.
>
> Muni
>
> Try this patch in your test environment. I have tested it and it seems
> to continue to deliver messages, and also not assert during rollover.
> This is an improvement from the past in which case both would occur.
>
> you can change START_SEQNO in totemsrp.c to test the rollover.
>
> If this change is too disturbing for you, you can always use the other
> workaround which should be sufficient.
>
> Regards
> -steve
>
> On Thu, 2005-10-27 at 13:57 -0500, Muni Bajpai wrote:
> > Steve,
> >
> > Thanks, will do. I had tested the overflow mechanism by overflowing at
> a
> > much lower number, didn't realize there were interactions. Will do
> some
> > research
> >
> > Thanks
> >
> > Muni
> >
> > -----Original Message-----
> > From: Steven Dake [mailto:scd@broked.org]
> > Sent: Tuesday, October 25, 2005 6:36 PM
> > To: Bajpai, Muni [RICH1:B670:EXCH]
> > Cc: Steven Dake; openais@lists.osdl.org; Smith, Kristen
> > [RICH1:B670:EXCH]
> > Subject: Re: [Openais] RE: token->token_seq wrap around.
> >
> >
> > Muni
> > I'm not sure this is your problem. I simulated setting the token->seq
> > to 0x7FFFFF000 and executed some traffic. Finally I got this
> assertion:
> >
> > Oct 25 16:28:26 [DEBUG ] [GMI ] Delivering 7fffff00 to 7fffff00
> > Oct 25 16:28:26 [DEBUG ] [GMI ] Delivering MCAST message with seq
> > 2147483392 to pending delivery queue
> > Oct 25 16:28:26 [DEBUG ] [GMI ] mcasted message added to pending
> > queue
> > token aru 2147483392
> > last aru 2147483391
> > high delivered 2147483391
> > Oct 25 16:28:26 [DEBUG ] [GMI ] releasing messages up to and
> > including 2147483391
> > aisexec: ../include/sq.h:181: sq_item_get: Assertion `seq_id <
> > (sq->head_seqid + sq->size)' failed. Aborted
> >
> > The reason it asserts at 7fffff00 is because the size of the message
> > queue is 256 bytes (7ffffff00 + ff rolls over and asserts in sq.h).
> >
> > I'm going to fix this problem and ensure rollovers happen correctly
> > without a reconfig if it can be done, or atleast increase it to
> > 0xffffffff.
> >
> > I will let you know of my progress. But you might find it helpful to
> > continue looking. If you receive a stop and can attach with gdb, look
> > at the values of instance->my_high_seq_recieved and my_aru. and
> > token->token_seq if possible. This should tell you if your
> assumptions
> > about rollover are correct.
> >
> > Regards
> > -steve
> >
> > On Thu, 2005-10-20 at 08:23, Muni Bajpai wrote:
> > > Thanks Steve,
> > >
> > > The fix works. I set the overflow test to a low number and saw a
> > > reconfig happen after that resetting all the data
> > >
> > > Thanks
> > >
> > > Muni
> > >
> > > -----Original Message-----
> > > From: Steven Dake [mailto:steven.dake@gmail.com]
> > > Sent: Tuesday, October 18, 2005 6:36 PM
> > > To: Bajpai, Muni [RICH1:B670:EXCH]
> > > Cc: sdake@mvista.com; openais@lists.osdl.org; Smith, Kristen
> > > [RICH1:B670:EXCH]
> > > Subject: Re: token->token_seq wrap around.
> > >
> > >
> > > Muni
> > > I am on the road in China so I wont be able to immediately address
> > > this issue.
> > >
> > > The variable token->token_seq is used to determine if a duplicate
> > > token is transmitted. I doubt this is your problem. The scenario
> you
> >
> > > describe could happen to token->seq which is the value used to
> > > indicate the global sequence number used on mcast packets.
> > >
> > > A simple and immediate fix is to drop the token is token->seq >
> > > 0x7f000000. This will cause a new configuration which will reset
> the
> > > sequence numbers appropriately.
> > >
> > > A more proper fix will take me a few days of looking at my test
> > > cluster. Unfortunately my link is so slow now I am unable to run any
>
> > > sorts of tests on my test cluster back home.
> > >
> > > I'll be back Sunday.
> > >
> > > Regards
> > > -steve
> > >
> > > On 10/18/05, Muni Bajpai <muniba@nortel.com> wrote:
> > > > Hey Steve,
> > > >
> > > >
> > > >
> > > > Our verification team was running traffic on 5 nodes this last
> week
> > > > till Monday morning (after the cluster had been up for 10 days)
> all
> > 5
> > > > nodes stopped receiving multicast messages (all 5 of them). The
> way
> > we
> > >
> > > > know this is because we have a timer that pops if an EVT message
> is
> > > > not received after 1 second and we know the publisher of the EVT
> > > > message was definitely publishing messages. Unfortunately there
> were
> >
> > > > no logs being sent to Stderr so we couldn't really see what the
> > > > executive was doing.
> > > >
> > > >
> > > >
> > > > The gdb trace revealed that the library was waiting indefinitely
> on
> > > > ALL nodes for a response to the request which was a Checkpoint
> > Write.
> > > > The gdb trace for the executive showed that it was behaving
> normally
> >
> > > > i.e aispoll was running and that it wasn't stuck like the library.
> > > >
> > > >
> > > >
> > > > So looking through the failure paths It was possible that one of
> the
> > > > following happened
> > > >
> > > > 1.) The library couldn't send the request to the executive
> > > >
> > > > 2.) The executive didn't receive the request
> > > >
> > > > 3.) The executive couldn't mcast the request (or dropped it)
> > > >
> > > > 4.) The executive couldn't receive the mcast. (or ignored it)
> > > >
> > > > 5.) The executive couldn't send the response to the library.
> > > >
> > > >
> > > >
> > > > So after considerable thought Kristen and I decided that 1,2,3,5
> > > > were
> > > > highly unlikely looking through the failure paths in the code,
> which
> >
> > > > left us with 4.
> > > >
> > > >
> > > >
> > > > 4 was likely as there are decisions based on sequence numbers
> > > > (cluster
> > > > wide) that the executive uses to discard or process messages and
> as
> > > > this happened on all the nodes that was the likely culprit.
> > > >
> > > >
> > > >
> > > > So in the end we think it's the fact that the token->token_seq
> that
> > > > is incremented on outgoing mcasts is rolling over after it reaches
>
> > > > 0x7fffffff (2147483647). The next increment is (-2147483648).
> > > >
> > > >
> > > >
> > > > We ran a test where in totemsrp.c:2717 (trunk version) we set the
> > > > token->token_seq to a negative number when the token->token_seq
> > > > token->reached
> > > > a value of 10000 (just to give us some time to bring up ckpt-wr)
> and
> > > > right when we reached that point the library hung and all mcasts
> > were
> > > > ignored from there on .
> > > >
> > > >
> > > >
> > > > I'm not sure how to address this other than checking for overflow
> > > > before we increment and reversing the state in all the sorted
> > queues.
> > > > The former is easy as in the patch below but reversing the stored
> > info
> > >
> > > > keyed off the sequence number is something I'm not sure.
> > > >
> > > >
> > > >
> > > > This is a big issue as it immediately sends our WHOLE cluster out
> of
> > > > service defeating the purpose.
> > > >
> > > >
> > > >
> > > > Thanks In advance
> > > >
> > > >
> > > >
> > > > Muni
> > > >
> > > > --- openais/trunk/exec/totemsrp.c 2005-10-18 11:24:28 -05:00
> > > >
> > > > +++ trunk/exec/totemsrp.c 2005-10-18 12:13:46 -05:00
> > > >
> > > > @@ -100,6 +100,7 @@
> > > >
> > > > * do not change
> > > >
> > > > */
> > > >
> > > > #define ENDIAN_LOCAL 0xff22
> > > >
> > > > +#define MAX_INT_VALUE_BEFORE_ROLL_OVER
> 0x7fffffff
> > > >
> > > >
> > > >
> > > > enum message_type {
> > > >
> > > > MESSAGE_TYPE_ORF_TOKEN = 0, /*
> Ordering,
> > > > Reliability, Flow (ORF) control Token */
> > > >
> > > > @@ -535,6 +536,18 @@
> > > >
> > > > }
> > > >
> > > > };
> > > >
> > > >
> > > >
> > > > +int incrementSequenceCheckingRollOver(int *seq)
> > > >
> > > > +{
> > > >
> > > > + assert(seq);
> > > >
> > > > + if (*seq >= MAX_INT_VALUE_BEFORE_ROLL_OVER)
> > > >
> > > > + {
> > > >
> > > > + *seq = 0;
> > > >
> > > > + return -1;
> > > >
> > > > + }
> > > >
> > > > + (*seq)++;
> > > >
> > > > + return 0;
> > > >
> > > > +}
> > > >
> > > > +
> > > >
> > > > void totemsrp_instance_initialize (struct totemsrp_instance
> > > > *instance)
> > > >
> > > > {
> > > >
> > > > memset (instance, 0, sizeof (struct totemsrp_instance));
> > > >
> > > > @@ -2231,7 +2244,7 @@
> > > >
> > > > int memb_index_this;
> > > >
> > > > int memb_index_next;
> > > >
> > > >
> > > >
> > > > - memb_commit_token->token_seq++;
> > > >
> > > > +
> > > > incrementSequenceCheckingRollOver(&memb_commit_token->token_seq);
> > > >
> > > > memb_index_this = (memb_commit_token->memb_index + 1) %
> > > > memb_commit_token->addr_entries;
> > > >
> > > > memb_index_next = (memb_index_this + 1) %
> > > > memb_commit_token->addr_entries;
> > > >
> > > > memb_commit_token->memb_index = memb_index_this;
> > > >
> > > > @@ -2714,7 +2727,11 @@
> > > >
> > > > memb_state_gather_enter (instance);
> > > >
> > > > } else {
> > > >
> > > > instance->my_token_seq = token->token_seq;
> > > >
> > > > - token->token_seq += 1;
> > > >
> > > > + if
> > > > (incrementSequenceCheckingRollOver(&token->token_seq) < 0)
> > > >
> > > > + {
> > > >
> > > > + instance->my_token_seq =
> > > > token->token_seq;
> > > >
> > > > +
> > > > incrementSequenceCheckingRollOver(&token->token_seq);
> > > >
> > > > + }
> > > >
> > > >
> > > >
> > > > if (instance->memb_state ==
> > > > MEMB_STATE_RECOVERY) {
> > > >
> > > > /*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> ______________________________________________________________________
> > > _______________________________________________
> > > Openais mailing list
> > > Openais@lists.osdl.org
> https://lists.osdl.org/mailman/listinfo/openais
> >
> >
> >
> > _______________________________________________
> > Openais mailing list
> > Openais@lists.osdl.org
> > https://lists.osdl.org/mailman/listinfo/openais
_______________________________________________
Openais mailing list
Openais@lists.osdl.org
https://lists.osdl.org/mailman/listinfo/openais
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic