[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openbsd-ports
Subject:    Re: heads-up: new dpb(1) functionality
From:       Marc Espie <espie () nerim ! net>
Date:       2012-12-30 23:29:57
Message-ID: 20121230232956.GA24402 () lain ! home
[Download RAW message or body]

On Mon, Dec 24, 2012 at 03:48:58PM +0100, Marc Espie wrote:
> First commit happened a few days ago. dpb scans for already built packages
> more aggressively, so that in case of restarts, the I/B information will
> be more accurate faster: before that change, every pkgpath would go through
> the queue, and already built paths would move to I/B later.

There was a bug related to that change, which has been fixed.
> 
> I'm also test-driving a small patch that moves multi-packages directly into
> I/B when a path finishes.
> 
> This should improve performance slightly. Most specifically, keeping already
> built packages in the queue of stuff to build makes the queue larger and 
> slower.  And having more accurate "already built" information will help dpb
> take better decisions.
That patch turned into a MUCH larger change... dpb's engine is now much faster.
But it's still a large part of dpb's cpu consumption... so now, changes to the
queues don't happen all the time: dpb monitors its own runtime consumption and
only updates the queue "when it's reasonable".

> 
> The other change will soon be committed, it's currently undergoing final tests.
> 
> dpb will now maintain "affinity" information, that is, it will record
> on which host each build is taking place.
The basic affinity information was okay.

> So, when restarting an interrupted dpb/hot fixing a problem, dpb will
> try to respect that information, as far as possible: during a first pass, 
> it will skip over build candidates if the host doesn't match. They will
> only be started on the "wrong" host if there is no choice left (that is,
> the queue is mostly empty and the cpus are starved).

This took at few tries to get right... the most annoying bug would be that you
HAD to restart dpb to take affinity into account (hot-fixes wouldn't do it), this
is now fixed.

Apart from starting jobs with the wrong affinity late, dpb will now also try to
see if free cores are available with the right hostname, thus allowing dpb to
obey affinity constraints in "most" practical cases.


> Those of you running dpb on clusters should appreciate that. It should make
> dpb much more resilient to restarts.
> 

There's the theory and then there's testing. Thanks to naddy@ et al for useful
feedback. I think I can now say that dpb supports affinity "as best as it can".

[prev in list] [next in list] [prev in thread] [next in thread]