[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openembedded-core
Subject:    [OE-core] [RFC] One shared state reuse solution
From:       koen () dominion ! thruhere ! net (Koen Kooi)
Date:       2012-03-31 0:10:32
Message-ID: 3D56E063-F4AD-43A7-8E80-724A6B1BE2A6 () dominion ! thruhere ! net
[Download RAW message or body]


Op 30 mrt. 2012, om 16:23 heeft Chris Larson het volgende geschreven:

> On Fri, Mar 30, 2012 at 3:02 PM, Koen Kooi <koen at dominion.thruhere.net> wrote:
> > Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:
> > 
> > > Greetings,
> > > 
> > > Over the past day, I've implemented a solution for the shared state
> > > reuse issues Mentor has seen with our poky-based product. This
> > > solution is similar in concept to what we had in our Mentor Embedded
> > > Linux 4 (non-yocto-based) product.
> > > 
> > > This implementation restructures SSTATE_DIR such that non-target
> > > sstate archives are placed in a directory specific to the host we're
> > > running on, and allows fallback to sstates from compatible hosts. In
> > > addition, there is a hook in place for modifying the returned build
> > > host identifier string. Using these capabilities, configured as you'll
> > > see below at the gist, I can populate sstate-cache from a Centos5
> > > machine and fully reuse that shared state on a u1004 machine, but if I
> > > take the u1004 sstate-cache and pull it over to Centos5, all the
> > > non-target recipes will be rebuilt.
> > > 
> > > This has a list of "compatible hosts", which are used as fallback
> > > regardless of what distro you're running on, so assumes you won't be
> > > running on a host older than the ones you're using as your
> > > compatibility baseline. I think this will satisfy the needs of most,
> > > and as you'll see when you look at the implementation, is entirely
> > > opt-in currently, so does no harm to anyone who chooses not to utilize
> > > it.
> > > 
> > > https://gist.github.com/2253903 - shows an example of how to make use
> > > of this functionality
> > > https://github.com/kergoth/oe-core/compare/sstate-structure - the \
> > > implementation 
> > > Regarding the implementation, I realize it isn't as clean as it could
> > > be, but the only way to resolve it in a cleaner way would be to modify
> > > bitbake, which I wasn't prepared to do at this juncture. The
> > > fundamental issue adding complexity is that SSTATE_DIR is pulled from
> > > the configuration metadata, not cached per-recipe, so one can't
> > > manipulate where individual recipes get their archives stored. As a
> > > workaround, I set the global SSTATE_DIR to the host-bound location,
> > > then when writing the archives for target recipes, moves the archive
> > > up to the parent directory (to the root of the original SSTATE_DIR)
> > > and symlinks it back.
> > > 
> > > Richard had proposed modifying the filename rather than the directory
> > > structure (e.g. via the sstate package arch), but this would make
> > > things far more complex. In order to implement fallback, one would
> > > have to mangle the filename, and one wouldn't be able to simply
> > > leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
> > > would have to attempt to fetch multiple filenames, which would require
> > > invasive changes to sstate.bbclass. I think using the directory
> > > structure is the easiest and cleanest route to the goal without
> > > invasive changes, and given it's opt-in nature, I'd like to see this
> > > go upstream, at a minimum as a temporary measure until/if a longer
> > > term more invasive solution occurs.
> > > 
> > > I'm looking for questions, comments, testers, and in particular,
> > > thoughts on whether this will meet the needs of others with similar
> > > requirements (e.g. others shipping metadata with associated shared
> > > state).
> > 
> > This is not strictly related to your patchset, but has anyone thought about \
> > license based blacklisting of sstate? I can imagine that an autobuilder will \
> > build everything, includes things like evil 3d drivers, but no want anyone to \
> > access the sstate for those builds.
> 
> I'd think that this would best be handled as a part of population of
> the shared state mirror. That is, we could create a class like
> copyleft_compliance, but for population of a shared state repository,
> obeying licensing for distribution constraints.

That was my idea as well, but I worded my question too vague :)

regards,

Koen


[prev in list] [next in list] [prev in thread] [next in thread]