[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openais
Subject:    Re: [Openais] Cluster messaging broken after split cluster reforms
From:       Steven Dake <sdake () redhat ! com>
Date:       2009-01-26 14:51:12
Message-ID: 1232981472.3518.23.camel () balance
[Download RAW message or body]


On Mon, 2009-01-26 at 09:19 +0100, Andrew Beekhof wrote:
> I'm seeing the following behavior somewhat regularly in my (whitetank)  
> cluster
> 
> At the beginning of the test, all nodes are up and in the same cluster  
> partition.
> We then create three partitions using iptables:
> Jan 26 08:01:51 c001n16 CTS: debug: Partition[1]:	['c001n02', 'c001n04']
> Jan 26 08:01:51 c001n16 CTS: debug: Partition[2]:	['c001n01']
> Jan 26 08:01:51 c001n16 CTS: debug: Partition[3]:	['c001n03']
> 
> Node uname/id table:
>   c001n02 156
>   c001n04 158
>   c001n01 155
>   c001n03 157
> 
> This works fine.  The problem occurs when we try to heal them.
> We remove the iptables rules (I verified the rules were removed) and  
> all the nodes reform into the same partition.
> 
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] CLM CONFIGURATION CHANGE
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] New Configuration:
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.155)
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.156)
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.157)
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.158)
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] Members Left:
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] Members Joined:
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.155)
> Jan 26 08:00:58 c001n04 openais[10345]: [CLM  ] 	r(0) ip(10.10.222.157)
> 
> However cluster messaging is permanently non-functional (I've waited  
> hours and it doesn't resolve itself).
> 
> I've attached the openais logs in the hope that someone can figure out  
> why.
> 

What errors do you get back from the api or totem functions?  Does
testevs work in this case?:

> _______________________________________________
> Openais mailing list
> Openais@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/openais

_______________________________________________
Openais mailing list
Openais@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/openais
[prev in list] [next in list] [prev in thread] [next in thread]