[prev in list] [next in list] [prev in thread] [next in thread] 

List:       gentoo-dev
Subject:    Re: [gentoo-dev] Large files still in files/
From:       "Malte S. Stretz" <msquadrat.nospamplease () gmx ! net>
Date:       2005-03-07 17:38:37
Message-ID: 200503071838.41942 () malte ! stretz ! eu ! org
[Download RAW message or body]

On Monday 07 March 2005 17:57 CET Martin Schlemmer wrote:
> On Sun, 2005-03-06 at 20:25 +0000, Cory Visi wrote:
> > On Sun, Mar 06, 2005 at 01:10:24PM -0500, Anthony de Boer wrote:
> > > Mark Loeser wrote:
> > > > There's also quite a large amount of binary files still in the
> > > > tree.  A lot of them seem to be compressed patches.  I'm not sure
> > > > what should be done with those, but I thought putting binary files
> > > > into the tree was discouraged unless absolutely necessary.  Lots of
> > > > 4k compressed patches doesn't seem to be something absolutely
> > > > necessary.
> > >
> > > Tying this to the Portage-tree collection-copyright issue, it might
> > > be a good idea for all third-party-sourced patches, with e-mail
> > > headers or other such authorship/source/copyright information still
> > > intact at the start (and happily skipped by the patch command), to be
> > > gzipped and put in distfiles, and the tree itself to be reserved for
> > > stuff written specifically for the Gentoo project.
> > >
> > > This does still leave large Gentoo-supplied patches in question; I'm
> > > uncomfortable with the idea of us getting *that* far from the
> > > upstream sources, though.
> >
> > I kind of like this idea, however, I think it's idealistic. Patches
> > need to be modified very frequently. Especially when we combine
> > multiple patches and make them all work with USE flags.
> >
> > A great deal of our patches really are written specifically work with
> > our ebuilds.
> >
> > What is the real percentage of space usage from compressed or
> > uncompressed patches? How big of a problem is it?
>
> Also the problem is that especially if you have a rapid changing
> package, where the patches changes a lot, putting it in distfiles is a
> pita, as you have to scp it, then wait an hour to 3 depending on how
> lucky you are, and then only commit.  And especially if you forgetful
> like me, you tend to forget to either come back and commit the new
> version/revision, or to copy the new tarball to distfiles ...

Didn't this discussion already happen about half a year ago and one of the 
conclusions was that a possible solution could be a special set of patches 
server from which those files are pulled on demand?  Those servers could 
even be (some of the) the rsync mirrors.

From a user point of view, this would hopefully have the advantage that the 
tedious 'emerge sync' runs (which often happen to time out for some reason, 
might be my internet connection) could be a lot (?) faster as only a 
percentage of the current tree would me mirrored locally -- the longest 
taking part of the rsync process is the "generating file list" one.

For the servers it could have the advantage that there's less load from 
rsync -- I guess even if the patches etc were pulled from the same servers 
via HTTP it would be less load as the files are (a) fetched on demand and 
(b) it wouldn't use the rsync protocol but plain HTTP.  But I don't have 
any numbers, maybe I'm wrong, and I don't know what additional traffic this 
would produce.

Just 2 cents from the peanut gallery.

Cheers,
Malte

--
gentoo-dev@gentoo.org mailing list

[prev in list] [next in list] [prev in thread] [next in thread]