[prev in list] [next in list] [prev in thread] [next in thread]
List: drbd-user
Subject: Re: [DRBD-user] Increasing a DRBD array
From: Gerald Brandt <gbr () majentis ! com>
Date: 2011-10-24 13:07:20
Message-ID: 47304f06-d982-4aef-a185-3c423bfda611 () thinkpad
[Download RAW message or body]
Hi,
Just a couple of questions... (I'm calling the default primary 'primary' throughout \
and the default secondary 'secondary' throughout, just to name machines. That may be \
where I'm getting confused with what you're saying.)
Why would a full 2Tb resync start at step 4, when the primary still doesn't have \
meta-data space for it?
Why would I switch the primary nodes after setting up the secondary with the new \
meta-data? My thought is:
1. when the secondary comes up with the new flexible-meta-data, it will probably \
resync 2. when the primary gets taken down, the secondary will become active (become \
primary), and server uptime will continue. 3. when the primary comes up with new \
flexible-meta-data, is will become the active node (become primary), and re-sync with \
the secondary.
From what I see (I think), I shouldn't have to switch to the secondary manually, and \
re-syncing should fix everything. At least, that's what I'm hoping.
Heartbeat does the switchover automatically, with no 'user-discernible' downtime.
Reading it all over again, it sounds like we're both saying the same thing, so this \
should work just fine.
Gerald
----- Original Message -----
> From: "Andreas Kurz" <andreas@hastexo.com>
> To: drbd-user@lists.linbit.com
> Sent: Monday, October 24, 2011 4:39:56 AM
> Subject: Re: [DRBD-user] Increasing a DRBD array
>
> On 10/22/2011 05:47 PM, Gerald Brandt wrote:
> > Okay, this is the plan for changing an primary/secondary drbd from
> > 'meta-disk /dev/sda6[1] to 'flexible-meta-disk /dev/sda6'
> >
> > sda6 is already 512 MB in size, which in theory will let me go to
> > 16 TB storage (128 MB is 4 TB storage).
>
> yes, 512MB is fine
>
> >
> > on secondary:
> > 1. drbdadm down iscsi.target.0
> > 2. edit drbd.conf to reflect meta-disk change (also on primary?)
>
> no, do it on the secondary only for now
>
> > 3. drbdadm create-md iscsi.target.0
> > 4. drbdadm up iscsi.target.0
> > 5. wait for full sync to take place
> >
> > on primary
>
> you mean on the previous primary ... as you might want to do a
> switchover to the already prepared secondary ....
>
> > 6-10 same as 1 - 5 above
> >
> > At with point I should be able to do drbdadm resize iscsi.target.0,
> > and I should see 6TB of storage.
>
> after point 4 where the previous primary connects to the already
> updated
> previous secondary the resize starts automatically as the two nodes
> detect they both have more diskspace and metadata that can handle
> this
> ... so you will see a resync of the "missing" 2TB
>
> >
> > That that sound right to everyone? It should give me no downtime
> > and the ability to have more than 4TB storage.
>
> No downtime? ... except the time you need to switch Roles and
> therefore
> migrate services ... but I assume you let your cluster manager do
> this job.
>
> Regards,
> Andreas
>
> --
> Need help with DRBD?
> http://www.hastexo.com/now
>
> >
> >
> > Gerald
> >
> > ----- Original Message -----
> > > From: "Gerald Brandt" <gbr@majentis.com>
> > > To: "Andreas Kurz" <andreas@hastexo.com>
> > > Cc: "drbd-user" <drbd-user@lists.linbit.com>
> > > Sent: Saturday, October 22, 2011 8:33:15 AM
> > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > >
> > > Hi,
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Andreas Kurz" <andreas@hastexo.com>
> > > > To: "drbd-user" <drbd-user@lists.linbit.com>
> > > > Sent: Friday, October 21, 2011 6:12:11 PM
> > > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > > >
> > > > On 10/21/2011 11:39 PM, Gerald Brandt wrote:
> > > > > Hi,
> > > > >
> > > > > I just saw that (google is my friend). Can I change that on a
> > > > > running drbd system?
> > > >
> > > > hmm ... never tried changing meta-date that way ... shutdown,
> > > > dump-md,
> > > > reconfigure, create-md, restore-md might work ... maybe Lars has
> > > > a
> > > > hint ...
> > > >
> > > > I would bring DRBD down on both nodes, stop it when all is in
> > > > sync
> > > > and
> > > > recreate the meta data after changing the config and then skip
> > > > the
> > > > initial sync when bringing them up.
> > > >
> > >
> > > I really can't bring the nodes down. I can bring down one at a
> > > time,
> > > but the systems have to stay running.
> > >
> > > > >
> > > > > ie:
> > > > >
> > > > > original:
> > > > >
> > > > > on iscsi-filer-1 {simply use "meta-disk internal;"
> > > > > device /dev/drbd1;
> > > > > disk /dev/md0;
> > > > > address 192.168.95.1:7789;
> > > > > meta-disk /dev/sda6[1];
> > > > > }
> > > > >
> > > > > on iscsi-filer-2 {
> > > > > device /dev/drbd1;
> > > > > disk /dev/md0;
> > > > > address 192.168.95.2:7789;
> > > > > meta-disk /dev/sda6[1];
> > > > > }
> > > > >
> > > > > new:
> > > > >
> > > > > on iscsi-filer-1 {
> > > > > device /dev/drbd1;
> > > > > disk /dev/md0;
> > > > > address 192.168.95.1:7789;
> > > > > flexible-meta-disk /dev/sda6[1];
> > > >
> > > > no ... that index thing only works for "static" meta-disk ...
> > > > remove
> > > > the
> > > > [1] and resize /dev/sda6 if its not bigger than 196MB.
> > >
> > > I'm not sure I understand. /dev/sda6 is already 512 MB (I think).
> > > Should I change to:
> > >
> > > on iscsi-filer-1 {
> > > device /dev/drbd1;
> > > disk /dev/md0;
> > > address 192.168.95.1:7789;
> > > flexible-meta-disk /dev/sda6;
> > > }
> > >
> > > on iscsi-filer-2 {
> > > device /dev/drbd1;
> > > disk /dev/md0;
> > > address 192.168.95.2:7789;
> > > flexible-meta-disk /dev/sda6;
> > > }
> > >
> > > or would this be better:
> > >
> > > on iscsi-filer-1 {
> > > device /dev/drbd1;
> > > disk /dev/md0;
> > > address 192.168.95.1:7789;
> > > meta-disk internal;
> > > }
> > >
> > > on iscsi-filer-2 {
> > > device /dev/drbd1;
> > > disk /dev/md0;
> > > address 192.168.95.2:7789;
> > > meta-disk internal;
> > > }
> > >
> > >
> > > I'll go back to my lists to see if I'm doing things right.
> > >
> > > 1. bring down the secondary
> > > 2. change the secondary to 'flexible-meta-data /dev/sda6' in
> > > drbd.conf on primary and secondary.
> > > 3. bring secondary back up (may re-sync entire disk, not a serious
> > > issue, just time)
> > > 4. repeat process for primary after re-sync (may cause another
> > > complete resync).
> > >
> > > Gerald
> > >
> > >
> > > >
> > > > Regards,
> > > > Andreas
> > > >
> > > > --
> > > > Need help with DRBD?
> > > > http://www.hastexo.com/now
> > > >
> > > > > }
> > > > >
> > > > > on iscsi-filer-2 {
> > > > > device /dev/drbd1;
> > > > > disk /dev/md0;
> > > > > address 192.168.95.2:7789;
> > > > > flexible-meta-disk /dev/sda6[1];
> > > > > }
> > > > >
> > > > > Then reboot primary, followed by reboot secondary (after sync),
> > > > > and
> > > > > all will be well?
> > > > >
> > > > > Sorry if these seem to be noob questions. I just want to be
> > > > > 100%
> > > > > sure, as the file servers have live data on them.
> > > > >
> > > > > Gerald
> > > > >
> > > > >
> > > > > ----- Original Message -----
> > > > > > From: "Andreas Kurz" <andreas@hastexo.com>
> > > > > > To: drbd-user@lists.linbit.com
> > > > > > Sent: Friday, October 21, 2011 4:26:13 PM
> > > > > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > > > > >
> > > > > > On 10/21/2011 10:48 PM, Gerald Brandt wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > DRBD is running directly on md0. /dev/drbd1 is then exported
> > > > > > > via
> > > > > > > iSCSI.
> > > > > > >
> > > > > > > The logs show:
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.332010] drbd:
> > > > > > > initialized.
> > > > > > > Version: 8.3.7 (api:88/proto:86-91)
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.332012] drbd:
> > > > > > > GIT-hash:
> > > > > > > ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
> > > > > > > root@filer-1,
> > > > > > > 2011-03-05 08:29:38
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.332014] drbd:
> > > > > > > registered as
> > > > > > > block device major 147
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.332015] drbd:
> > > > > > > minor_table @
> > > > > > > 0xffff88021dbaf300
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.334144] block
> > > > > > > drbd1:
> > > > > > > Starting worker thread (from cqueue [1489])
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.334177] block
> > > > > > > drbd1:
> > > > > > > ==>
> > > > > > > truncating very big lower level device to currently maximum
> > > > > > > possible
> > > > > > > 8587575296 sectors <==
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.334179] block
> > > > > > > drbd1:
> > > > > > > ==>>
> > > > > > > using internal or flexible meta data may help <<==
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.334188] block
> > > > > > > drbd1:
> > > > > > > disk(
> > > > > > > Diskless -> Attaching )
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.353306] Loading
> > > > > > > iSCSI
> > > > > > > transport class v2.0-870.
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.360582] skge
> > > > > > > eth0:
> > > > > > > disabling interface
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 iscsid: iSCSI logger with
> > > > > > > pid=1515
> > > > > > > started!
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 kernel: [ 7.381956] iscsi:
> > > > > > > registered
> > > > > > > transport (iser)
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 init: ssh main process (1162)
> > > > > > > terminated
> > > > > > > with status 255
> > > > > > > Oct 21 15:34:53 iscsi-filer-1 postfix/master[1411]: reload --
> > > > > > > version
> > > > > > > 2.7.0, configuration /etc/postfix
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584678] block
> > > > > > > drbd1:
> > > > > > > Found
> > > > > > > 57 transactions (3507 active extents) in activity log.
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584684] block
> > > > > > > drbd1:
> > > > > > > Method
> > > > > > > to ensure write ordering: barrier
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584691] block
> > > > > > > drbd1:
> > > > > > > Backing device's merge_bvec_fn() = ffffffffa00c0100
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584694] block
> > > > > > > drbd1:
> > > > > > > max_segment_size ( = BIO size ) = 4096
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584698] block
> > > > > > > drbd1:
> > > > > > > Adjusting my ra_pages to backing device's (32 -> 96)
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.584704] block
> > > > > > > drbd1:
> > > > > > > drbd_bm_resize called with capacity == 8587575296
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.603470] block
> > > > > > > drbd1:
> > > > > > > resync
> > > > > > > bitmap: bits=1073446912 words=16772608
> > > > > > > Oct 21 15:34:54 iscsi-filer-1 kernel: [ 7.603474] block
> > > > > > > drbd1:
> > > > > > > size =
> > > > > > > 4095 GB (4293787648 KB)
> > > > > > >
> > > > > > > The lines in yellow bug me. I don't recall see them before.
> > > > > > >
> > > > > > > I had a 4 disk RAID-6 md0 (4x2TB = 4 TB RAID-6). I added a
> > > > > > > single
> > > > > > > drive
> > > > > > > (5x2TB = 6 TB array).
> > > > > >
> > > > > > Not using "meta-disk internal" or "flexible-meta-disk" limits
> > > > > > the
> > > > > > device
> > > > > > size to 4TB (=128MB meta data size) ... change your metadata
> > > > > > config
> > > > > > ...
> > > > > > as the logs suggest ... if you want to use all 6TB
> > > > > >
> > > > > > Regards,
> > > > > > Andreas
> > > > > >
> > > > > > --
> > > > > > Need help with DRBD?
> > > > > > http://www.hastexo.com/now
> > > > > >
> > > > > > >
> > > > > > > Any ideas?
> > > > > > >
> > > > > > > Gerald
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------------------------------------------------
> > > > > > >
> > > > > > > From: "Andreas Kurz" <andreas@hastexo.com>
> > > > > > > To: drbd-user@lists.linbit.com
> > > > > > > Sent: Friday, October 21, 2011 2:55:06 PM
> > > > > > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > > > > > >
> > > > > > > On 10/21/2011 09:30 PM, Gerald Brandt wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I've successfully resize the lower level RAID-6 array,
> > > > > > > > and
> > > > > > > > grown
> > > > > > > > it. I'm now attempting to resize drbd, and nothing
> > > > > > > > seems
> > > > > > > > to
> > > > > > > > happen.
> > > > > > > >
> > > > > > > > /dev/md0 is definitely bigger.
> > > > > > > >
> > > > > > > > What should I see during a drbd resize?
> > > > > > >
> > > > > > > You should see a DRBD resync of the newly added space.
> > > > > > >
> > > > > > > What is the lower level device of your DRBD resource? The
> > > > > > > whole md0,
> > > > > > > a
> > > > > > > partition on md0, a lv on a vg on a pv on md0?
> > > > > > >
> > > > > > > ... so if the lower level device has been resized on both
> > > > > > > nodes, DRBD
> > > > > > > should definitely grow on a "drbdadm resize".
> > > > > > >
> > > > > > > Did I mention that on starting DRBD it is resized
> > > > > > > automatically if a
> > > > > > > bigger lower level device is detected? ... have a look at
> > > > > > > the
> > > > > > > kernel
> > > > > > > logs...
> > > > > > >
> > > > > > > Regards,
> > > > > > > Andreas
> > > > > > >
> > > > > > > --
> > > > > > > Need help with DRBD?
> > > > > > > http://www.hastexo.com/now
> > > > > > >
> > > > > > > >
> > > > > > > > Gerald
> > > > > > > >
> > > > > > > >
> > > > > > > > ----- Original Message -----
> > > > > > > > > From: "Gerald Brandt" <gbr@majentis.com>
> > > > > > > > > To: drbd-user@lists.linbit.com
> > > > > > > > > Sent: Tuesday, October 18, 2011 7:08:55 AM
> > > > > > > > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Okay, this is my list of what to do, and in what
> > > > > > > > > order:
> > > > > > > > >
> > > > > > > > > 1. remove the primary from DRBD
> > > > > > > > > 2. add the physical disk to the primary
> > > > > > > > > 3. add the primary back to DRBD and allow resync.
> > > > > > > > > 4. remove the secondary from DRBD
> > > > > > > > > 5. add the physical disk to the secondary
> > > > > > > > > 6. add the secondary back to DRBD and allow resync.
> > > > > > > > > 7. fdisk and add the disk to the RAID array on primary
> > > > > > > > > and
> > > > > > > > > secondary
> > > > > > > > > 8. grow the RAID array on the primary and secondary
> > > > > > > > > (while
> > > > > > > > > mounted
> > > > > > > > > or
> > > > > > > > > not mounted?)
> > > > > > > > > 9. drbdadm resize on the primary (can this be done
> > > > > > > > > while
> > > > > > > > > drbd is
> > > > > > > > > in
> > > > > > > > > use?)
> > > > > > > > > 10. expand the file system on the primary only (via
> > > > > > > > > Citrix
> > > > > > > > > XenServer
> > > > > > > > > (iSCSI))
> > > > > > > > >
> > > > > > > > > That sounds about right. As long as I don't lose
> > > > > > > > > data.
> > > > > > > > > (yeah,
> > > > > > > > > backups. That's left up to my client, but becomes my
> > > > > > > > > problem if
> > > > > > > > > they failed.)
> > > > > > > > >
> > > > > > > > > Gerald
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ----- Original Message -----
> > > > > > > > > > From: "Florian Haas" <florian@hastexo.com>
> > > > > > > > > > To: drbd-user@lists.linbit.com
> > > > > > > > > > Sent: Tuesday, October 18, 2011 3:53:11 AM
> > > > > > > > > > Subject: Re: [DRBD-user] Increasing a DRBD array
> > > > > > > > > >
> > > > > > > > > > On 2011-10-17 22:19, Matt Graham wrote:
> > > > > > > > > > > > > Your DRBD uses the MD RAID as its backing device,
> > > > > > > > > > > > > doesn't it?
> > > > > > > > > > > > > That means you'll have add the drive and resize
> > > > > > > > > > > > > your
> > > > > > > > > > > > > RAID
> > > > > > > > > > > > > first,
> > > > > > > > > > > > > then
> > > > > > > > > > > > > you can resize your DRBD, and then expand the file
> > > > > > > > > > > > > system.
> > > > > > > > > > > > Definitely built on top of the RAID. I'm a bit
> > > > > > > > > > > > confused
> > > > > > > > > > > > by
> > > > > > > > > > > > your
> > > > > > > > > > > > answer though. Step 8 grows the RAID array, and I
> > > > > > > > > > > > don't
> > > > > > > > > > > > have a
> > > > > > > > > > > > step
> > > > > > > > > > > > to grow DRBD. Does that mean I'm missing
> > > > > > > > > > > > something?
> > > > > > > > > > >
> > > > > > > > > > > 8. mdadm --grow /dev/md0 (other mdadm options)
> > > > > > > > > > > Do that on both machines.
> > > > > > > > > > > 8.1 (on secondary) drbdsetup /dev/drbd0 resize
> > > > > > > > > > > 8.2 (on primary) drbdsetup /dev/drbd0 resize
> > > > > > > > > >
> > > > > > > > > > There's a "resize" subcommand for drbdadm too; no
> > > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > drbdsetup.
> > > > > > > > > > And it needs to be done on one node only. Just do
> > > > > > > > > > "drbdadm
> > > > > > > > > > resize
> > > > > > > > > > <resourcename>" on the primary while the two nodes
> > > > > > > > > > are
> > > > > > > > > > connected,
> > > > > > > > > > and
> > > > > > > > > > you're good to go.
> > > > > > > > > >
> > > > > > > > > > > The softRAID is below the DRBD device. Once the
> > > > > > > > > > > softRAID is
> > > > > > > > > > > larger, the DRBD device needs to be informed of
> > > > > > > > > > > this
> > > > > > > > > > > fact.
> > > > > > > > > > > When making arrays/devices larger, you go
> > > > > > > > > > > bottom-up;
> > > > > > > > > > > you make
> > > > > > > > > > > the hardware changes first, then softRAID
> > > > > > > > > > > changes,
> > > > > > > > > > > then DRBD
> > > > > > > > > > > changes, then filesystem changes.
> > > > > > > > > > > 9. (on primary) resize2fs /dev/drbd0
> > > > > > > > > >
> > > > > > > > > > If it's ext3 and if it supports on-line resizing, of
> > > > > > > > > > course. But
> > > > > > > > > > that
> > > > > > > > > > should be obvious.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Florian
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Need help with DRBD?
> > > > > > > > > > http://www.hastexo.com/knowledge/drbd
> > > > > > > > > > _______________________________________________
> > > > > > > > > > drbd-user mailing list
> > > > > > > > > > drbd-user@lists.linbit.com
> > > > > > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > drbd-user mailing list
> > > > > > > > > drbd-user@lists.linbit.com
> > > > > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > drbd-user mailing list
> > > > > > > > drbd-user@lists.linbit.com
> > > > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > drbd-user mailing list
> > > > > > > drbd-user@lists.linbit.com
> > > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > drbd-user mailing list
> > > > > > > drbd-user@lists.linbit.com
> > > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > drbd-user mailing list
> > > > > > drbd-user@lists.linbit.com
> > > > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > > > >
> > > >
> > > >
> > > > --
> > > > Need help with DRBD?
> > > > http://www.hastexo.com/now
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > drbd-user mailing list
> > > > drbd-user@lists.linbit.com
> > > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > > >
> > > _______________________________________________
> > > drbd-user mailing list
> > > drbd-user@lists.linbit.com
> > > http://lists.linbit.com/mailman/listinfo/drbd-user
> > >
> > _______________________________________________
> > drbd-user mailing list
> > drbd-user@lists.linbit.com
> > http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic