[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fossil-users
Subject:    Re: [fossil-users] fossil too slow on large repository
From:       Gé_Weijers <ge () weijers ! org>
Date:       2009-02-26 18:31:57
Message-ID: 49A6E01D.4030408 () weijers ! org
[Download RAW message or body]

It 's fixed. My test corpus was an rsync'ed directory containing all the 
RFCs and Internet Drafts from ref-editor.org.

It takes about 20 minutes to do the initial checkin, on a FreeBSD 
machine (1GB RAM, 2GHz Celeron)

Thanks!

Ge'

D. Richard Hipp wrote:
> On Feb 25, 2009, at 3:42 PM, Gé Weijers wrote:
>
>   
>> On a related note: I have a large collection of files (13000), some of
>> which are large PDFs, most are source code etc. I have been unable to
>> check this tree in in fossil, it runs out of memory (RAM) after a few
>> minutes, after the working set size goes above half a gigabyte. I can
>> add the files just fine, but the final commit is painful.
>>     
>
>
> http://www.fossil-scm.org/fossil/ci/f6790b7c3cc237fac3d0cc064cb262d50fa12265
>
> I tested with the Enron email corpus.  That's over half a million  
> separate files totaling 1.5 GB in size.  Took about an hour, but the  
> commit did go through (after the fix above), and memory usage never  
> went above 50MB or so.
>
> I'll continue to do performance testing on this enron repository and  
> see if I can't get commits to run a little faster.  Bear in mind,  
> though, that each fossil check-in stores an MD5 checksum over every  
> file in the check-in (both those that changes and those that have not)  
> and regardless of what you do, it does take some time to run MD5 over  
> 1.5 GB in 517431 separate files.
>
>
> D. Richard Hipp
> drh@hwaci.com
>
>
>
> _______________________________________________
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>   


[prev in list] [next in list] [prev in thread] [next in thread]