[prev in list] [next in list] [prev in thread] [next in thread]
List: gluster-users
Subject: Re: [Gluster-users] Is it required for a node to meet quorum over all the nodes in storage pool?
From: Atin Mukherjee <amukherj () redhat ! com>
Date: 2019-01-31 3:45:06
Message-ID: CAGNCGH1etKS0=iz41vUk+=OmCDQX4L5WCVoA-SBk56amP_Y-bg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Fri, Jan 25, 2019 at 1:41 PM Jeevan Patnaik <g1patnaik@gmail.com> wrote:
> Hi,
>
> I'm just going through the concepts of quorum and split-brains with a
> cluster in general, and trying to understand GlusterFS quorums again which
> I previously found difficult to accurately understand.
>
> When we talk about server quorums, what I understand is that the concept
> is similar to STONITH in cluster i.e., we shoot the node that probably have
> issues/ make the bricks down preventing access at all. But I don't get how
> it calculates quorum.
>
> My understanding:
> In a distributed replicated volume,
> 1. All bricks in a replica set should have same data writes and hence, it
> is required to meet atleast 51% quorum on those replica sets. Now
> considering following 3x replica configuration:
> ServerA,B,C,D,E,F-> brickA,B,C,D,E,F respectively and serverG without any
> brick in storage pool.
>
Please note server quorum isn't calculated based on number of active bricks
rather number of active nodes in the cluster. So in this case even if
server G doesn't host any bricks in the storage pool, the quorum will be
decided based on total number of servers/peers in the cluster vs total
number of active peers in the cluster. If you're interested to know about
it more, please refer
https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/server-quorum/
or
https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-server-quorum.c#L281
(in case you are happy to browse the source code and understand the logic).
> Scenario:
> ServerA,B,F formed a partition i.e., they are isolated with other nodes in
> storage pool.
>
> But serverA,B,C bricks are of same sub-volume, Hence if we consider quorum
> over sub-volumes, A and B meets quorum for it's only participating
> sub-volume and can serve the corresponding bricks. And the corresponding
> bricks on C should go down.
>
> But when we consider quorum over storage pool, C,D,E,G meets quorum
> whereas A,B,F is not. Hence, bricks on A,B,F should fail. And for C, the
> quorum still will not me met for it's sub-volume. So, it will go to read
> only mode. Sub-volume on D and E should work normally.
>
> So, with assumption that only sub-volume quorum is considered, we don't
> have any downtime on sub-volumes, but we have two partitions and if clients
> can access both, clients can still write and read on both the partitions
> separately and without data conflict. The split-brain problem arrives when
> some clients can access one partition and some other.
>
> If quorum is considered for entire storage pool, then this split-brain
> will not be seen as the problem nodes will be dead.
>
> And so why is it's not mandatory to enable server quorum to avoid this
> split-brain issue?
>
> And I also assume that quorum percentage should be greater than 50%.
> There's any option to set custom percentage. Why is it required?
> If all that is required is to kill the problem node partition (group) by
> identifying if it has the largest possible share (i.e. greater than 50),
> does the percentage really matter?
>
> Thanks in advance!
>
> Regards,
> Jeevan.
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 25, 2019 at 1:41 PM \
Jeevan Patnaik <<a href="mailto:g1patnaik@gmail.com">g1patnaik@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">I'm just going through \
the concepts of quorum and split-brains with a cluster in general, and trying to \
understand GlusterFS quorums again which I previously found difficult to accurately \
understand. </div><div dir="auto"><br></div><div dir="auto">When we talk about \
server quorums, what I understand is that the concept is similar to STONITH in \
cluster i.e., we shoot the node that probably have issues/ make the bricks down \
preventing access at all. But I don't get how it calculates quorum. </div><div \
dir="auto"><br></div><div dir="auto">My understanding:</div><div dir="auto">In a \
distributed replicated volume, </div><div dir="auto">1. All bricks in a replica set \
should have same data writes and hence, it is required to meet atleast 51% quorum on \
those replica sets. Now considering following 3x replica configuration:</div><div \
dir="auto">ServerA,B,C,D,E,F-> brickA,B,C,D,E,F respectively and serverG without \
any brick in storage pool.</div></div></blockquote><div><br></div><div>Please note \
server quorum isn't calculated based on number of active bricks rather number of \
active nodes in the cluster. So in this case even if server G doesn't host any \
bricks in the storage pool, the quorum will be decided based on total number of \
servers/peers in the cluster vs total number of active peers in the cluster. If \
you're interested to know about it more, please refer <a \
href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/server- \
quorum/">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/server-quorum/</a> \
or <a href="https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src \
/glusterd-server-quorum.c#L281">https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-server-quorum.c#L281</a> \
(in case you are happy to browse the source code and understand the \
logic).</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div \
dir="auto"><br></div><div dir="auto">Scenario:</div><div dir="auto">ServerA,B,F \
formed a partition i.e., they are isolated with other nodes in storage \
pool.</div><div dir="auto"><br></div><div dir="auto">But serverA,B,C bricks are of \
same sub-volume, Hence if we consider quorum over sub-volumes, A and B meets quorum \
for it's only participating sub-volume and can serve the corresponding bricks. \
And the corresponding bricks on C should go down. </div><div \
dir="auto"><br></div><div dir="auto">But when we consider quorum over storage pool, \
C,D,E,G meets quorum whereas A,B,F is not. Hence, bricks on A,B,F should fail. And \
for C, the quorum still will not me met for it's sub-volume. So, it will go to \
read only mode. Sub-volume on D and E should work normally.</div><div \
dir="auto"><br></div><div dir="auto">So, with assumption that only sub-volume quorum \
is considered, we don't have any downtime on sub-volumes, but we have two \
partitions and if clients can access both, clients can still write and read on both \
the partitions separately and without data conflict. The split-brain problem arrives \
when some clients can access one partition and some other.</div><div \
dir="auto"><br></div><div dir="auto">If quorum is considered for entire storage pool, \
then this split-brain will not be seen as the problem nodes will be dead.</div><div \
dir="auto"><br></div><div dir="auto">And so why is it's not mandatory to enable \
server quorum to avoid this split-brain issue?</div><div dir="auto"><br></div><div \
dir="auto">And I also assume that quorum percentage should be greater than 50%. \
There's any option to set custom percentage. Why is it required?</div><div \
dir="auto">If all that is required is to kill the problem node partition (group) by \
identifying if it has the largest possible share (i.e. greater than 50), does the \
percentage really matter?</div><div dir="auto"><br></div><div dir="auto">Thanks in \
advance!</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div \
dir="auto">Jeevan.</div><div dir="auto"><br></div><div dir="auto"><br></div><div \
dir="auto"><br></div><div dir="auto"><br></div></div> \
_______________________________________________<br> Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" \
target="_blank">Gluster-users@gluster.org</a><br> <a \
href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" \
target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div></div></div>
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic