'[DRBD-user] Is it normal that we can't directly remove the secondary node when fencing is set?'

[prev in list] [next in list] [prev in thread] [next in thread] 

List: drbd-user
Subject: [DRBD-user] Is it normal that we can't directly remove the secondary node when fencing is set?
From: <mzlld1988 () 163 ! com>
Date: 2016-09-10 6:46:01
Message-ID: 7976b4af.38c6.15712d88101.Coremail.mzlld1988 () 163 ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

[Attachment #4 (text/plain)]

Hi everyone,
I have a question about removing the secondary node of DRBD9.
When fencing is set, is it normal that we can't remove the secondary node of DRBD9, \
but the operation is successful of DRBD8.4.6?

Version of DRBD kernel source is the newest version(9.0.4-1).Version of DRBD utils is \
8.9.6. Description:
    3 nodes, one of the nodes is primary,disk state is UpToDate.Fencing is set.
    I got an error message 'State change failed: (-7) State change was refused by \
peer node' when executing the command 'drbdadm down <res-name>' on any of the \
secondary nodes.

Analysis:
    When executing the down command on one of the secondary nodes.
    The secondary node will execute the methods 'change_cluster_wide_state' of \
drbd_state.c.  change_cluster_wide_state()
    {
        ...
        if (have_peers) {
                if (wait_event_timeout(resource->state_wait,
                               cluster_wide_reply_ready(resource),
                               twopc_timeout(resource))){-------------①Waiting for \
                peer node to reply, the thread will sleep until the peer node \
                replies.
                    rv = get_cluster_wide_reply(resource);------------②Get the reply \
info.          }else{
                }
        ...
    }

    Process ①
        Primary node will execute the following methods.
            ..->try_state_change->is_valid_soft_transition->__is_valid_soft_transition

            Finally,__is_valid_soft_transition will return error code \
SS_PRIMARY_NOP。

            if (peer_device->connection->fencing_policy >= FP_RESOURCE &&
                !(role[OLD] == R_PRIMARY && repl_state[OLD] < L_ESTABLISHED && \
                !(peer_disk_state[OLD] <= D_OUTDATED)) &&
                 (role[NEW] == R_PRIMARY && repl_state[NEW] < L_ESTABLISHED && \
!(peer_disk_state[NEW] <= D_OUTDATED)))

                   return SS_PRIMARY_NOP;

            Primary node will set drbd_packet to P_TWOPC_NO, seconday node will get \
the reply to set connection status to TWOPC_NO。  At this time,Process ① will \
finish.

    Process ②
           rv will be set to SS_CW_FAILED_BY_PEER

    ====8.4.6版====
        One is primary, the next one is secondary.
        When executing 'drbdadm down <res-name>' on seconday node, the same error \
message will be recorded in the log file for the first time to change the peer disk \
                to D_UNKNOWN。
        But the command will succeed by changing peer disk to D_OUTDATED for the \
second time.  
        The following code that report the error.
        is_valid_state()
        {
            ...
            if (fp >= FP_RESOURCE &&
                     ns.role == R_PRIMARY && ns.conn < C_CONNECTED && ns.pdsk >= \
D_UNKNOWN①){  rv = SS_PRIMARY_NOP;
                     }
            ...
        }

        After executing the command 'drbdadm down <res-name>' on secondary node, the \
status of the primary node is:  [root@drbd846 drbd-8.4.6]# cat /proc/drbd
        version: 8.4.6 (api:1/proto:86-101)
        GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by \
root@drbd846.node1, 2016-09-08 08:51:45  0: cs:StandAlone ro:Primary/Unknown \
                ds:UpToDate/Outdated   r-----
            ns:1048508 nr:0 dw:0 dr:1049236 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f \
oos:0

        The peer disk state is OutDated, not DUnknown.

[Attachment #5 (text/html)]

<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Hi everyone, I have a question about removing the secondary node of DRBD9. When \
fencing is set, is it normal that we can't remove the secondary node of DRBD9, but \
the operation is successful of DRBD8.4.6? <ul><li>Version of DRBD kernel source is the newest \
version(9.0.4-1).Version of DRBD utils is 8.9.6.</li></ul></div><div>Description:</div>&nbsp;&nbsp;&nbsp; 3 nodes, one of \
the nodes is primary,disk state is UpToDate.Fencing is set. &nbsp;&nbsp;&nbsp; I \
got an error message 'State change failed: (-7) State change was refused by peer node' when executing the command 'drbdadm down &lt;res-name&gt;' \
on any of the secondary nodes. Analysis: &nbsp;&nbsp;&nbsp; \
When executing the down command on one of the secondary nodes. &nbsp;&nbsp;&nbsp; \
The secondary node will execute the methods 'change_cluster_wide_state' of \
drbd_state.c. &nbsp;&nbsp;&nbsp; change_cluster_wide_state() &nbsp;&nbsp;&nbsp; \
{ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
... &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (have_peers) \
{ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
if (wait_event_timeout(resource-&gt;state_wait, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp \
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
cluster_wide_reply_ready(resource), &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp \
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
twopc_timeout(resource))){-------------①Waiting for peer node to reply, the thread \
will sleep until the peer node \
replies. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
rv = get_cluster_wide_reply(resource);------------②Get the reply \
info.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
}else{ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
} &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ... &nbsp;&nbsp;&nbsp; \
} &nbsp;&nbsp;&nbsp; Process ① &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
Primary node will execute the following \
methods. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
..-&gt;try_state_change-&gt;is_valid_soft_transition-&gt;__is_valid_soft_transition <div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
Finally,__is_valid_soft_transition will return error code \
SS_PRIMARY_NOP。</div><div> </div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if \
(peer_device-&gt;connection-&gt;fencing_policy &gt;= FP_RESOURCE \
&amp;&amp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
!(role[OLD] == R_PRIMARY &amp;&amp; repl_state[OLD] &lt; L_ESTABLISHED &amp;&amp; \
!(peer_disk_state[OLD] &lt;= D_OUTDATED)) \
&amp;&amp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
(role[NEW] == R_PRIMARY &amp;&amp; repl_state[NEW] &lt; L_ESTABLISHED &amp;&amp; \
!(peer_disk_state[NEW] &lt;= D_OUTDATED))) <div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
return SS_PRIMARY_NOP;</div><div> </div><div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Primary \
node will set drbd_packet to P_TWOPC_NO, seconday node will get the reply to set \
connection status to \
TWOPC_NO。 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; At \
this time,Process ① will finish. </div>&nbsp;&nbsp;&nbsp; Process ② &nbsp;&nbsp; \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; rv will be set to \
SS_CW_FAILED_BY_PEER &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
&nbsp; &nbsp;&nbsp;&nbsp; \
====8.4.6版==== &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; One is primary, the \
next one is secondary. &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; When executing \
'drbdadm down &lt;res-name&gt;' on seconday node, the same error message will be \
recorded in the log file for the first time to change the peer disk to \
D_UNKNOWN。 &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; But the command will succeed by \
changing peer disk to D_OUTDATED for the second \
time. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp;&nbsp; \
&nbsp;&nbsp;&nbsp;&nbsp; The following code that report the error. &nbsp;&nbsp; \
&nbsp; &nbsp;&nbsp; is_valid_state() &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
{ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
... &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (fp \
&gt;= FP_RESOURCE &amp;&amp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
ns.role == R_PRIMARY &amp;&amp; ns.conn &lt; C_CONNECTED &amp;&amp; ns.pdsk &gt;= \
D_UNKNOWN①){ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
rv = SS_PRIMARY_NOP; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
} &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
... &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; } &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
&nbsp; &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; After executing the command 'drbdadm \
down &lt;res-name&gt;' on secondary node, the status of the primary node \
is: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [root@drbd846 drbd-8.4.6]# cat \
/proc/drbd &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; version: 8.4.6 \
(api:1/proto:86-101) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GIT-hash: \
833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@drbd846.node1, 2016-09-08 \
08:51:45 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0: cs:StandAlone \
ro:Primary/Unknown ds:UpToDate/Outdated&nbsp;&nbsp; \
r----- &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \
ns:1048508 nr:0 dw:0 dr:1049236 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f \
oos:0 &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; The peer disk state is OutDated, not \
DUnknown.</div> &nbsp;

_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[prev in list] [next in list] [prev in thread] [next in thread] 