[prev in list] [next in list] [prev in thread] [next in thread] 

List:       beowulf
Subject:    Re: [Beowulf] Rsync - checksums
From:       Bill Wichser <bill () princeton ! edu>
Date:       2019-09-30 17:09:29
Message-ID: 8f8b0035-4f6e-a9eb-a7c7-138af7f099ec () princeton ! edu
[Download RAW message or body]

Just wanted to circle back on my orginal question.  I changed the rsync 
code adding xxhash and we see about a 3x speedup.  Good enough since it 
is very close to not using any checksum speedups.

Bill

On 6/17/19 9:43 AM, Bill Wichser wrote:
> We have moved to a rsync disk backup system, from TSM tape, in order to 
> have a DR for our 10 PB GPFS filesystem.  We looked at a lot of options 
> but here we are.
> 
> md5 checksums take a lot of compute time with huge files and even with 
> millions of smaller ones.  The bulk of the time for running rsync is 
> spent in computing the source and destination checksums and we'd like to 
> alleviate that pain of a cryptographic algorithm.
> 
> Googling around, I found no mention of using a technique like this to 
> improve rsync performance.  I did find reference to a few hashing 
> algorithms though which could certainly work here (xxhash, murmurhash, 
> sbox, cityhash64).
> 
> Rsync has certainly been around for a few years!  We are going to pursue 
> changing the current checksum algorithm and using something much faster. 
> If anyone has done this already and would like to share their 
> experiences that would be wonderful. Ideally this could be some optional 
> plugin for rsync where users could choose which checksummer to use.
> 
> Bill
> _______________________________________________
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit \
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf


[prev in list] [next in list] [prev in thread] [next in thread]