[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lustre-devel
Subject:    [Lustre-devel] changelog for whole filesystem?
From:       andreas.dilger () oracle ! com (Andreas Dilger)
Date:       2010-11-12 23:58:17
Message-ID: 2D5AA299-5AB3-4BE9-B917-4390704D8E65 () oracle ! com
[Download RAW message or body]

On 2010-11-12, at 16:41, Robert Read wrote:
> It seems simpler to do the iteration on  a snapshot, instead of a live filesystem, \
> and allow post-snapshot changes to accumulate on the regular changelog for \
> processing once the "from empty" iteration was complete.  

Just need to implement snapshot support for Lustre. :-)

> On Nov 11, 2010, at 10:10 AM, Eric Barton wrote:
> > Nathan wrote...
> > > This same "initial population of a database from objects" problem
> > > occurs when trying to replicate a Lustre filesystem using the
> > > changelog.
> > 
> > That must be a great test case.  If we can reliably reconstruct an
> > exact replica using a "from empty" changelog, we must have got it
> > right :)
> > 
> > > The problem is actually more complicated even for a single changelog
> > > consumer: since iterating through the virtual changelog takes non-0
> > > time, you're not sure if a virtual record was created before or
> > > after an actual changelog entry.  E.g. you might try to resolve the
> > > name of a file for a virtual record either before or after a rename,
> > > and then later you would see the rename in the changelog, leading to
> > > an inconsistent view of the namespace.
> > > 
> > > If you don't care about having the exact right name, then it's easy
> > > enough to ignore inapplicable changelog records.
> > 
> > I was trying to hint at the need to filter changes for individual
> > consumers depending on how far through the initial "from empty"
> > iteration they are in my original post.  Andreas stated it more
> > explicitly.  I think you just need to enumerate the cases - e.g. for
> > rename...
> > 
> > Iterated over yet?               Record
> > Source Inode   Target Inode      emitted
> > no             no                none
> > no             yes               delete target
> > yes            yes               rename source + delete target
> > yes            no                rename source
> > 
> > The final case just looks like a rename where the target name didn't
> > exist already.  In any case, the only real requirement on the stream
> > of changelog records constructed in the initial iteration is that
> > consistent filesystem state be reconstructed after the last record
> > is consumed.
> > 
> > -- 
> > 
> > Cheers,
> > Eric
> > 
> > Eric Barton
> > CTO Whamcloud, Inc.
> > Tel: +44 117 330 1575
> > Mob: +44 7920 797 273
> > 
> > _______________________________________________
> > Lustre-devel mailing list
> > Lustre-devel at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-devel
> 


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.


[prev in list] [next in list] [prev in thread] [next in thread]