[prev in list] [next in list] [prev in thread] [next in thread] 

List:       gentoo-dev
Subject:    Re: [gentoo-dev] RFC about another *DEPEND variable
From:       Brian Harring <ferringb () gmail ! com>
Date:       2006-09-30 19:34:31
Message-ID: 20060930193430.GA9496 () seldon
[Download RAW message or body]

On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote:
> On Wednesday 27 September 2006 03:54, Brian Harring wrote:
> > > > Bleh, this is getting back to exactly my point that it's unbounded
> > > > resolution.  To support this, every step of execution would require
> > > > scanning for dangling nodes to punt; aside from invalidating -p's
> > > > output it *still* is a collision-protect hit.
> > >
> > > when the package upgrade detects an ABI change you recalculate the
> > > packages that need to be rebuilt ...
> >
> > Reiterating, this screws over any form of up front calculation;
> > dialup users/per minute connections can't rely on emerge -f to
> > actually fetch all involved sources, -p results ain't guranteed
> > valid, parallelization must now lock at each threads merge on the off
> > chance a recalc is forced.
> 
> it does indeed prevent full up front calculation.  we can always make the 
> behavior configurable; revdep on the fly or allow people to break it up.  the 
> key being that their system will still continue to function as the ABI lib 
> has been preserved automagically

Folks aren't that daft; you make it an optional component, it means a 
*proper* solution will never be pushed because the duct tape is in 
place already.

If that's what folks want, sure, but what you're proposing is just 
sliding NEEDED in as the defacto solution without labeling it as such.


> > > dangling nodes get recorded in the new package and considering these old
> > > files are not detrimental to the health of the system, you can do such a
> > > cleanup once at the end of the emerge
> >
> > It's not orphaning files, but your scheme doesn't work for any form of
> > interruption; failed builds, user intervention, etc, they all can
> > leave the lib stuck in the new contents.
> 
> huh ?  i outlined in a previous e-mail how by simply ordering the operations 
> sanely, there is no race condition

Re-read your emails, and mine please.  The scenario I pointed out was 
ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2.

Now how does portage know that it needs to complete that upgrade for 
the next emerge action?  How does it know to even scan for lib1, let 
alone punting?

It's *not* simple, and I keep pointing out this issue (and you're 
missing it every damn time).  Please address it, or point to where you 
have (gone over the thread and still not seeing anything even 
remotely touching this flaw).


> > Dealing with that possibility means the manager has to maintain on
> > disk a list of the refcount of each dangling libs to decrement,
> > unmerge has to modify said list, etc.
> 
> which is already being done in the NEEDED file ... funny how unpainless it is 
> to generate that file

First of all, unpainless is painful.  Which is apt, actually.

Familiar with old style virtuals?  That whole, "you have to walk the 
whole vdb to collect all old style virtuals" ?

This is the same damn thing.

There is no refcount maintained via NEEDED, just a list of libs a 
binary uses; you _still_ have to walk the damn vdb to build the 
refcount list.

Now either you're forcing a fairly huge mapping in memory, or you're 
forcing a scan of the vdb for every pkg operation that has NEEDED 
entries it will install.

So no, NEEDED doesn't cover this, and my point still stands.


> > > > It also involves changing vdb nodes from "installed and usable" to
> > > > "installed/usable" or "lingering"
> > >
> > > no ... the old versions get merged into the new one as their existence is
> > > purely hidden
> >
> > How exactly are they going to be hidden?  A new file?  If so,
> > backwards compatibility for vdb for transitioning in such a support
> > has to be addressed.
> 
> purely hidden from the standpoint of any new package trying to use it ... 
> since you're only installing the runtime ABI library, you cannot link against 
> it or utilize it any way other than existing files.

Except for dlopen, but thats again besides my original point; how do 
you note to portage that the lib is one to watch so it can be punted?

You're suggesting it get shoved into contents, and for portage to 
identify it, it has to do a revdep mapping on the fly to find it.  

This *sucks* massively, both from a speed and complexity standpoint; 
further, the lib isn't hidden from pkg ops (say quickpkging, or 
binpkging) in any way so the cruft spreads.  That's surprisingly minor 
in comparison to implications of relying on refcounting to identify 
the lib to punt.

If the lib to punt isn't tracked in some fashion, the only algo 
you're left with is attempting to find all libs that have a refcount 
of zero, and punt those- obviously, this is going to screw up just 
about every single dlopen based linkage, literally, suck an algo 
won't spot that python dlopen's it's modules, and would think the 
refcount was zero (thus the so could get booted).

The response there is to add blacklists, directories to *not* inspect, 
which is a further hack to try and make this tank fly.

*OR*, you're stuck maintaining a seperated list of libs to punt, 
rather then just trying to slide the lib into existing CONTENTS.


> > > > Tracking BINCOMPAT should *not* be that hard.
> > >
> > > it's one more thing to keep track of and considering all of the
> > > possibilities i outlined, a single maintainer can easily lose his sanity
> > > ... or you force wasted rebuilds on people when it's only needed in some
> > > circumstances
> >
> > How exactly is this forcing wasted rebuilds?  You're stating that
> > maintainers are going to bump it willy nilly.  As I said, it is a key
> > that would be bumped *as needed*, and would only affected pkgs that
> > specified that node as a binding dep (specially marked atom).
> 
> no, i mean for example scenarios where a package provides more than one 
> library (say libfoo and libbar) and only one of those changes ABI values (say 
> libfoo.0 -> libfoo.1).  if another package links against just libbar, you've 
> got a pointless rebuild.

If one changes it's version, it's a fair bet that any consumer of that 
pkg is linked to more then just that lib; as such they would be 
rebuilt *anyways*.


> > Seriously, maintainers ought to know *now* when they're forcing a
> > revdep on folks systems, I'm not seeing the massive overhead from
> > pushing that info into a var, nor am I seeing mass forced useless
> > rebuilds.
> 
> there's a ton of things maintainers ought to know ... pretty soon our package 
> maintainers are going to have to be gods if they want to write an ebuild as 
> they're going to have to know every detail about the package inside and out

Sorry, but if a developer is bumping a pkg and doesn't even look to 
see if the damn thing is potentially going to break others via soname 
changes, that maintainer is being an idiot.

Being aware of a high level detail like "hey, my package installs a 
library, and they changed the soname" isn't being god like, it's 
being at least haphazardly concerned about QA of the tree.

Worth noting this exact scenario is already laid out in the quizes 
too; 'sposed to know the affects of stabilizing a library.


> > Realistically, just the same as the NEEDED solution has holes, the
> > maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is
> > that they can do something for BINCOMPAT; hell, QA checks can be
> > automated to catch missing BINCOMPAT bumps.
> 
> what's the difference ?  in my scenario they dont have to do anything because 
> the bump would have been caught automatically ?

Your solution has holes up the ying yang for actual 
implementation/handling of it due to the fact it's forcing the 
resolver to be an effective recalc at each step, and relies on duct 
taping metadata handling to try and exempt bits of lingering data.

You move that data up front, the implementation is actually sane, it's 
deterministic (thus *representative* to the user) and is more 
powerful.


> > KEYWORDed arches bit, bit unlikely that the underlying arch is
> > actually going to screw with the soname, thus I'd want actual examples
> > backing it up.
> 
> how about libc.so from glibc and libgcc.so from gcc ?  or would you like other 
> packages ?

Considering such a change would be internal ABI, NEEDED doesn't 
actually fix anything; if it were a soname change, level playing 
ground again.

Admittedly, BINCOMPAT breaks down there due to not being fine grained 
enough if it's internal changes; that however doesn't make NEEDED a 
good solution, just means that my alternative has a hole.


> > Besides, again, for keywording, the dev *should* be compiling it, so
> > there is a way to catch it :P.  A revbump isn't going to break things
> > unless the dev is introducing something intrusive, minor version bumps
> > (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive;
> > regardless, dev should know the high level affects of their change
> 
> uhh, there is no such requirement at this time for revbumping on different 
> arches ... currently we say the maintainer tests for his arches and leaves 
> all the others as ~arch

Reiterating; devs should know the high level affects of their changes.  

If it's going to change the soname version, they should know from the 
get go- unless it's an arch specific breakage (which I still posit is 
the rare/corner case), they will *see* it for their arch and bump it 
already.

They keywording comment above is specific to moving from ~arch to arch 
also; for new bumps, the ~arch keywords carry forward, but see the 
paragraph above re: it.


> > Inducing a revdep-rebuild is definitely one of those high level
> > changes they should be aware of *prior* to the bump.
> 
> and yet we look at our history of so many packages being bumped with ABI 
> changes and users having broken-as-shit systems ... i'm accounting for the 
> worst; not hoping for the best

ABI changes aren't fixed by NEEDED.  Presume you meant soname changes.

Stating that things are broken doesn't make NEEDED automagically 
better either; *both* enable a way to fix it, so you need to justify 
the "accounting for the worst; not hoping for the best".


> > > or dig
> > > through the source code line by line and hope to catch all such cases by
> > > hand/eye ?
> >
> > Bit of FUD here, although I spose I'll just point out that if so's
> > change as massively as you're implying above, the affects on -p are
> > that much worse.
> 
> awesome, mark something you disagree with as FUD, problem solved.  my point is 
> that you can never know completely for sure the behavior of a package without 
> digging through the entire source code.

It's FUD due to the fact NEEDED suffers the same fucking issue.  The 
irony is that BINCOMPAT would at least give a knob to mark it as a 
breakage, NEEDED cannot.


> > bind the checks in as a QA measure, enabled via FEATURES=stricter,
> > used to maintain a metadata var.
> 
> a lot of things fail already with FEATURES=stricter ... too bad we dont have 
> per-package env var support as then maintainers could just flag all their own 
> packages as stricter without wanting to shoot themselves due to everyone 
> else's package failures

Take it up with your fellow portage devs then.  It was shot down 
internally multiple times due to the affects it has on the python 
side.

If you want to get into per package features, split a thread and I'll 
let marius argue over it.


> > Literally, pkg x version y forced a rebuild.  revdep is annoying, but
> > stats would be useful to actually evaluate the seperate proposals;
> > related, getting stats for how often the various updaters were
> > required to be ran for a particular pkgs upgrade would be useful in
> > evaluating that particular gap NEEDED has.
> 
> any openssl version bump where the numerical value changes ... only the letter 
> updates preserve ABI compat (0.9.7[a-z] are compat)
> 
> expat-1.x -> expat-2.x
> 
> iirc, tcl-8.3.x -> tcl-8.4.x
> 
> readline-4.x -> readline-5.x
> 
> ncurses-4.x -> ncurses-5.x

Was looking for stats comparing breakage to a subset of the tree; iow, 
out of N packages, how many actually induce a rebuild?  Yes it's 
slightly nasty generating those stats (tinderbox?), but it would be 
useful.


Frankly, if you don't believe me or think my points are correct about 
how a NEEDED based solution is going to suck, zac/jason/genone/anyone 
else.  They're going to point out the same flaws.

~harring

[Attachment #3 (application/pgp-signature)]
-- 
gentoo-dev@gentoo.org mailing list


[prev in list] [next in list] [prev in thread] [next in thread]