[prev in list] [next in list] [prev in thread] [next in thread] 

List:       qubes-devel
Subject:    Re: [qubes-devel] Deterministic builds for Qubes OS -- the shortcut?
From:       Holger Levsen <holger () layer-acht ! org>
Date:       2015-12-23 13:53:23
Message-ID: 201512231453.37428.holger () layer-acht ! org
[Download RAW message or body]


Hi,

On Dienstag, 22. Dezember 2015, Joanna Rutkowska wrote:
> TL;DR, the ETA for having Debian build
> deterministically is ~1.5 year, which I take to mean it won't happen
> before 2018. 

there are two slightly different things here to look at: getting Debian 100% 
reproducible (=all 23k source packages in main) and getting reproducible 
packages in Debian at all, cause then one can tackle those packages one is 
interested in.

To get Debian 9, aka Stretch, the next release reproducible at all, we need to 
get three fixes into Debian sid/unstable until November 5th 2016, which is the 
announced freeze date for Debian. These three fixes exist, so it's "merely a 
question of getting them into sid…" ;-)

In our test setup, reproducible.debian.net, we are building all 23k packages 
with these 3 fixes (and 8 other toolchain packages) and a reaching 85% 
reproducibility in the current test setup atm. With just those 3 fixes we 
probably would be at 70% or so, but I do think that it should be possible to 
get them all in in the next 10 months.

And then, Debian 9 will probably be released in April or May 2017. (The last 
10 years every release cycle was 22 months on average with a max variation of 
2 months, it's a wrong myth that Debian releases are unpredictable.)

Getting Debian 100% reproducible OTOH I'm not sure whether we'll achieve for 
Debian 10, though I would love to see that happen 8-) Anyhow.

> Rather than trying to make *everything* build deterministically from
> scratch (which is what current efforts try to do, AFAIU) why not prepare
> an environment (as in: a VM image running under Qubes) that would be
> optimized for ensuring that whatever we build using qubes-builder (yes,
> let's focus on Qubes, for now, ok?) it always produces the same binaries?

I think focusing on Qubes here is entirely appropriate and taking shortcuts is 
also a good idea ;-)
 
> Admittedly the image itself would not build deterministically, true, but we
> would publish it (i.e. the binary) for everybody to download and inspect.
> Assuming people find it acceptable to trust this image (after spending many
> nights analysing it with a disassembler, of course ;) then all future
> builds, including future ISOs, images, and packages, should build
> deterministically. I think this would be a big win for everybody,
> especially that I think we might get this working within single weeks...

I'm not sure I understand, you want to create an environment to build all the 
packages Qubes depends on reproducibile? To rebuild all fedora packages 
reproducible, so that those can be used to build a reproducible installation 
image?

Cause there are actually two problems to tackle, I think: a.) getting 
reproducible binary packages and b.) getting a reproducible installation 
image.

To answer b.)

I've basically don't know how the Qubes iso is really made, but I assume it's 
the same or similar as it would be in Debian: packages are used to create a 
chroot which is turned into the iso. In the Debian case this leads to another 
problem: package installation is also not reproducible, thus such images aint 
neither.

In Debian the main source (besides timestamps of files+directories created at 
installation time) for this is that apt installs packages in non deterministic 
order and then the postinst scripts are run undeterministically as well, which 
for example can lead to different uids/gids for the same system users…

And to get back to a.) in case you dont want to rebuild all of fedora / 
debian: then you "just" need a stable package binary repository, so that when 
you rebuild an iso after a month (or more time) updates (which were not 
available at the original creation date) are not included in image. Debian has 
snapsho.debian.org which archives every binary package ever released by 
Debian, I don't know hat Fedora has to offer here.

> So, how such an environment should look like? I think (perhaps naively)
> that the following adjustments should be just enough:
> 
> 1. Patch any date/time-returning syscalls (or lib calls?) in the VM so that
> each call returns time incremented by, say, +1 sec (we really don't want
> to go with the resolution down to anything too small, so that the build
> process do not miss the time differences due to rounding errors). AFAIU
> there is already a tool for that, although, I've been told, it returns
> always 0, which clearly is not gonna work with most build systems. Also,
> there is apparently a tool that returns incremental time readings but on
> nanosec resolution, which, again, seems too little for most build
> processes to notice.

there is faketime which does that and which we found to cause problems when 
building certain software, so we rather suggest dropping timestamps.

But maybe using faketime is the way to go when building such images? It's 
definitly a a nice shortcut if it works ;-)

> 2. We need to ensure the whole build process is single-threaded (think:
> -j1).

actually building in parallel aint a problem per se, those 85% reproducibility 
of Debian packages is achieved with parallel builds.

> 3. Potentially patch /dev/(u)random to return some predictable values,
> similar to how we need to patch the time syscalls (think: return
> sha1(seed++)?)

seeding using SOURCE_DATE_EPOCH is generally a good idea, yes.

https://reproducible-builds.org/specs/source-date-epoch/

> Marek and Wojtek tried to convince this would not work, because... if
> things were that simple, others would already done that by now. That's a
> legit argument, I agree, but because none of them were able to provide an
> actual _technical_ argument why this would not work, I think it's worth
> for somebody to just try that.  Unsurprisingly I won't be able to do that
> myself anytime in the coming weeks, but hope others might be
> interested...?

I agree, somesome should just test this. I'd be happy to run these tests to 
build Qubes on our hardware, we have plenty of ressources to waste^wuse. 


cheers,
	Holger

["signature.asc" (application/pgp-signature)]

[prev in list] [next in list] [prev in thread] [next in thread]