[prev in list] [next in list] [prev in thread] [next in thread] 

List:       rsync
Subject:    Re: Reducing rsync memory usage for file lists
From:       Martin Pool <mbp () valinux ! com>
Date:       2001-06-26 13:48:02
[Download RAW message or body]

Yes, that sounds like a pretty good plan for (say) rsync 3.0.  We all
seem to be more or less on the same track as to how the protocol
should look.

Here are my feelings about the way to get there.  I would be happy to
have holes picked in them:

 * rsync 2.x works well, but is too crufty to be a good base for
   future development.   If we want to do anything other than bug
   fixes and small feature additions we should start with a clean
   slate.

 * Running over ssh is an excellent way of pushing the work of
   security and authentication onto somebody else who can do it well. 
   Modules over raw TCP is also a very good way of exporting public
   or high-speed resources.  Unfortunately people get a little
   confused at first about the differences between the two mechanisms.
 
 * The protocol should not assume the structure of the program.  For
   example, it should be possible to send and receive files at
   different times across the same socket, or to hold a socket open
   and idle for a while waiting for human commands.

 * The protocol must be pipelined (or streaming, or call it what you
   will).  Basically we must not block waiting for responses if there
   is something else useful we could be doing, including issuing more
   responses.

 * (This is the most controversial.)  All other things being equal, it
   is better not to invent a new network protocol if there is an
   existing protocol that will do the job well.  The asynchronous RPC
   mechanism in NFSv4 is a good foundation, and has most of what we
   need except for actually computing and applying deltas, which could
   straightforwardly be added.

   I am playing with a userspace NFSv4 client and server, which can
   already retrieve files and list directories.  In a little while I
   might try a (very simple-minded) mirroring program built on top, to
   see how it works out.

-- 
Martin 
VA Linux Systems                        GnuPG encrypted email preferred

[prev in list] [next in list] [prev in thread] [next in thread]