[prev in list] [next in list] [prev in thread] [next in thread] 

List:       drbd-user
Subject:    [DRBD-user] Recover Data from inconsistent nodes
From:       Manuthu M <mike () angani ! co>
Date:       2020-07-22 13:16:38
Message-ID: D5719050-AB61-47C2-BDE7-25C3427C60A0 () angani ! co
[Download RAW message or body]

Hi all,

Please assist on recovering data from a DRBD MySQL node that failed to come up after \
a maintenance window. Currently we have a setup of 2 mysql nodes (DB01 and DB02) \
using pacemaker and DRBD.

Earlier today, we needed to put on standby the primary node - DB02 in this case and \
use DB01. Pacemaker succeeded in promoting the secondary node (DB01) to primary and \
all other services under it.  After about ~ 45 minutes of usage, we noticed \
inconsistencies in the data. Further investigations revealed that the data was \
missing from about 2 weeks ago (07th July). Which means that the sync had somehow \
stopped between the two nodes. 

Worth noting is that during this time the nodes had started syncing before we \
noticed. [root@db01 ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2020-04-05 \
02:58:18  0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:29240140 nr:0 dw:21665384 dr:10156909 al:1027 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 \
wo:f oos:6974196  [=========>..........] sync'ed: 52.5% (6808/14316)M
	finish: 5:19:38 speed: 360 (1,480) K/sec


We attempted to revert to the previous setup by putting the current node on standby \
and promoting the secondary to primary but failed to come back online. 

We now have a old copy of the database from 2 weeks ago running on the primary. While \
we have nightly backups we have lost a significant amount of data from between last \
night backup to the time the issue arose.

Is it possible to recover this and what is the best approach to doing this. 

Here is the DRBD config file 

[root@db02 ~]# cat /etc/drbd.d/mysql01.res
resource mysql01 {
 protocol C;
 meta-disk internal;
 device /dev/drbd0;
 disk  /dev/centos_db-02/db02;
 handlers {
  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
 }
 net {
  allow-two-primaries no;
  after-sb-0pri discard-zero-changes;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }
 disk {
  on-io-error detach;
 }
 syncer {
  verify-alg sha1;
 }
 on db01 {
  address 172.19.5.8:7789;
 }
 on db02 {
  address 172.19.5.9:7789;
 }
}


Thanks in advance,
Mike(null)
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user


[prev in list] [next in list] [prev in thread] [next in thread]