[prev in list] [next in list] [prev in thread] [next in thread]
List: qubes-devel
Subject: Re: [qubes-devel] Deterministic builds for Qubes OS -- the shortcut?
From: Eric Shelton <knockknock () gmail ! com>
Date: 2015-12-22 18:42:59
Message-ID: f7aba36f-fa77-40dd-9609-8efad8cba05b () googlegroups ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Tuesday, December 22, 2015 at 12:57:24 PM UTC-5, HW42 wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Joanna Rutkowska:
> > Hi,
> >
> > I've recently had a chance to discuss with Marek and Wojtek, who
> recently took
> > part in the Reproducible Builds Summit in Athens [1], the current state
> of the
> > these efforts. TL;DR, the ETA for having Debian build deterministically
> is ~1.5
> > year, which I take to mean it won't happen before 2018. Then, at least
> 0.5 year
> > to get Qubes (i.e. the ISO) to build deterministically, which means we
> won't
> > have it before the end of 2018, optimistically counting ;)
> >
> > So, I thought, why not take the following shortcut:
> >
> > Rather than trying to make *everything* build deterministically from
> scratch
> > (which is what current efforts try to do, AFAIU) why not prepare an
> environment
> > (as in: a VM image running under Qubes) that would be optimized for
> ensuring
> > that whatever we build using qubes-builder (yes, let's focus on Qubes,
> for now,
> > ok?) it always produces the same binaries?
> >
> > Admittedly the image itself would not build deterministically, true, but
> we
> > would publish it (i.e. the binary) for everybody to download and
> inspect.
> > Assuming people find it acceptable to trust this image (after spending
> many
> > nights analysing it with a disassembler, of course ;) then all future
> builds,
> > including future ISOs, images, and packages, should build
> deterministically. I
> > think this would be a big win for everybody, especially that I think we
> might
> > get this working within single weeks...
> >
> > So, how such an environment should look like? I think (perhaps naively)
> that the
> > following adjustments should be just enough:
> >
> > 1. Patch any date/time-returning syscalls (or lib calls?) in the VM so
> that each
> > call returns time incremented by, say, +1 sec (we really don't want to
> go with
> > the resolution down to anything too small, so that the build process do
> not miss
> > the time differences due to rounding errors). AFAIU there is already a
> tool for
> > that, although, I've been told, it returns always 0, which clearly is
> not gonna
> > work with most build systems. Also, there is apparently a tool that
> returns
> > incremental time readings but on nanosec resolution, which, again, seems
> too
> > little for most build processes to notice.
> >
> > 2. We need to ensure the whole build process is single-threaded (think:
> -j1).
> >
> > 3. Potentially patch /dev/(u)random to return some predictable values,
> similar
> > to how we need to patch the time syscalls (think: return sha1(seed++)?)
> >
> > I think we need to ensure these patches (and time/seed variables
> initialization)
> > are only for the processes started by the qubes-builder, so that any
> potential
> > interaction with the VM by the user via e.g. Terminal do not interfere
> here
> > (alternatively do not allow starting of any processes other than
> qubes-builder,
> > and use only qvm-run/qrexec for this).
> >
> > And I think this should be pretty much that :) Again, we need to start
> with this
> > image being (nondeterministically) build by one of us, but then
> everything
> > should build deterministically. Perhaps including the next iteration of
> the very
> > image itself...
> >
> > Marek and Wojtek tried to convince this would not work, because... if
> things
> > were that simple, others would already done that by now. That's a legit
> > argument, I agree, but because none of them were able to provide an
> actual
> > _technical_ argument why this would not work, I think it's worth for
> somebody to
> > just try that. Unsurprisingly I won't be able to do that myself anytime
> in the
> > coming weeks, but hope others might be interested...?
> >
> > Thanks,
> > joanna.
> >
> > [1] https://reproducible-builds.org/events/athens2015/
>
> I will also focus only on direct Qubes components (i.e. ignore packages
> build by Fedora/Debian).
>
> I think we can do better with a comparable amount of effort. The ~1.5
> years for Debian is in my opinion the time until you can reproduce many
> of the packages in stable. The current state in Debian is that they have
> an "experimental toolchain" (basically a patched dpkg/debhelper plus some
> documentation generators, see [2]). With these patches they already
> produce very remarkable success (see [3]). They have a small group which
> works on fixing all the small remaining problems in the ~20k Debian
> source packages. I expect that they will get the dpkg/debhelper patches
> into unstable in the next two months. Then a significant part of
> unstable will already be reproducible. Until this lands in stable, the
> Debian policy requires reproducibility, etc. a lot of time will pass.
> Also the last 5-10% of packages probably will require a lot of work
> until they reach (nearly) 100%.
>
> But since we focus (for now) only on direct Qubes components this doesn't
> need to bother us. I made some tests and the Qubes specific debian
> packages are already reproducible when using the patched dpkg+debhelper.
> On similar patches for Fedora/rpm are already be worked on.
>
> So what we can do is simply modify qubes-builder to use a patched
> version of dpkg/rpm. We also need to make some changes to qubes-builder
> to record the required infos from the build-env. and be able to
> reproduce these (mainly the installed versions of build dependecies).
> With this we should get quickly reproducible .debs/.rpms.
>
> A little bit more work will be needed to get the installer ISO and the
> template images reproducible. But here we can use a pragmatic approach
> (i.e. instead of waiting to get all things fixed upstream use
> postprocessing and local patches) and also get this with manageable
> effort/time.
>
> Some notes on the proposed approach:
>
> Partially this is already done. Gitian [4] uses a predefined VM image to
> build the software in it. For example this is used for the Tor Browser
> (see [5]).
>
> I expect that there will be many small problems (which as as a whole
> will also be a lot of work) until software build successfully and
> reproducible with this setup. For example I can image that there will be
> programs which call the gettime syscall a different number of times
> depending on your CPU speed. Also stuff will break due to the
> manipulated syscalls (can be fixed - but needs work).
>
>
> [2]: https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
> [3]: https://reproducible.debian.net/reproducible.html
> [4]: https://gitian.org/
> [5]:
> https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/README.build \
>
Given the discussion of piggybacking off of Debian's efforts at
implementing reproducible builds (however soon they may be available for
use in the Qubes build process), does that mean there is a plan to
transition away from Fedora to Debian as the base on which Qubes is built?
I have no complaints with going in that direction (Fedora's release cycles
seem to introduce unnecessary friction) - just interested where things are
headed.
Eric
--
You received this message because you are subscribed to the Google Groups \
"qubes-devel" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to qubes-devel+unsubscribe@googlegroups.com. To post to this group, \
send email to qubes-devel@googlegroups.com. To view this discussion on the web visit \
https://groups.google.com/d/msgid/qubes-devel/f7aba36f-fa77-40dd-9609-8efad8cba05b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[Attachment #5 (text/html)]
On Tuesday, December 22, 2015 at 12:57:24 PM UTC-5, HW42 wrote:<blockquote \
class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc \
solid;padding-left: 1ex;">-----BEGIN PGP SIGNED MESSAGE----- <br>Hash: SHA512
<br>
<br>Joanna Rutkowska:
<br>> Hi,
<br>>
<br>> I've recently had a chance to discuss with Marek and Wojtek, who \
recently took <br>> part in the Reproducible Builds Summit in Athens [1], the \
current state of the <br>> these efforts. TL;DR, the ETA for having Debian build \
deterministically is ~1.5 <br>> year, which I take to mean it won't happen \
before 2018. Then, at least 0.5 year <br>> to get Qubes (i.e. the ISO) to build \
deterministically, which means we won't <br>> have it before the end of 2018, \
optimistically counting ;) <br>>
<br>> So, I thought, why not take the following shortcut:
<br>>
<br>> Rather than trying to make *everything* build deterministically from scratch
<br>> (which is what current efforts try to do, AFAIU) why not prepare an \
environment <br>> (as in: a VM image running under Qubes) that would be optimized \
for ensuring <br>> that whatever we build using qubes-builder (yes, let's \
focus on Qubes, for now, <br>> ok?) it always produces the same binaries?
<br>>
<br>> Admittedly the image itself would not build deterministically, true, but we
<br>> would publish it (i.e. the binary) for everybody to download and inspect.
<br>> Assuming people find it acceptable to trust this image (after spending many
<br>> nights analysing it with a disassembler, of course ;) then all future \
builds, <br>> including future ISOs, images, and packages, should build \
deterministically. I <br>> think this would be a big win for everybody, especially \
that I think we might <br>> get this working within single weeks...
<br>>
<br>> So, how such an environment should look like? I think (perhaps naively) that \
the <br>> following adjustments should be just enough:
<br>>
<br>> 1. Patch any date/time-returning syscalls (or lib calls?) in the VM so that \
each <br>> call returns time incremented by, say, +1 sec (we really don't want \
to go with <br>> the resolution down to anything too small, so that the build \
process do not miss <br>> the time differences due to rounding errors). AFAIU \
there is already a tool for <br>> that, although, I've been told, it returns \
always 0, which clearly is not gonna <br>> work with most build systems. Also, \
there is apparently a tool that returns <br>> incremental time readings but on \
nanosec resolution, which, again, seems too <br>> little for most build processes \
to notice. <br>>
<br>> 2. We need to ensure the whole build process is single-threaded (think: \
-j1). <br>>
<br>> 3. Potentially patch /dev/(u)random to return some predictable values, \
similar <br>> to how we need to patch the time syscalls (think: return \
sha1(seed++)?) <br>>
<br>> I think we need to ensure these patches (and time/seed variables \
initialization) <br>> are only for the processes started by the qubes-builder, so \
that any potential <br>> interaction with the VM by the user via e.g. Terminal do \
not interfere here <br>> (alternatively do not allow starting of any processes \
other than qubes-builder, <br>> and use only qvm-run/qrexec for this).
<br>>
<br>> And I think this should be pretty much that :) Again, we need to start with \
this <br>> image being (nondeterministically) build by one of us, but then \
everything <br>> should build deterministically. Perhaps including the next \
iteration of the very <br>> image itself...
<br>>
<br>> Marek and Wojtek tried to convince this would not work, because... if things
<br>> were that simple, others would already done that by now. That's a legit
<br>> argument, I agree, but because none of them were able to provide an actual
<br>> _technical_ argument why this would not work, I think it's worth for \
somebody to <br>> just try that. Unsurprisingly I won't be able to do that \
myself anytime in the <br>> coming weeks, but hope others might be interested...?
<br>>
<br>> Thanks,
<br>> joanna.
<br>>
<br>> [1] <a href="https://reproducible-builds.org/events/athens2015/" \
target="_blank" rel="nofollow" \
onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Freproducible- \
builds.org%2Fevents%2Fathens2015%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHswGVuyxOtWihCdxq3xTR-fmMzEg';return \
true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Freproducib \
le-builds.org%2Fevents%2Fathens2015%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHswGVuyxOtWihCdxq3xTR-fmMzEg';return \
true;">https://reproducible-builds.<wbr>org/events/athens2015/</a> <br>
<br>I will also focus only on direct Qubes components (i.e. ignore packages
<br>build by Fedora/Debian).
<br>
<br>I think we can do better with a comparable amount of effort. The ~1.5
<br>years for Debian is in my opinion the time until you can reproduce many
<br>of the packages in stable. The current state in Debian is that they have
<br>an "experimental toolchain" (basically a patched dpkg/debhelper plus \
some <br>documentation generators, see [2]). With these patches they already
<br>produce very remarkable success (see [3]). They have a small group which
<br>works on fixing all the small remaining problems in the ~20k Debian
<br>source packages. I expect that they will get the dpkg/debhelper patches
<br>into unstable in the next two months. Then a significant part of
<br>unstable will already be reproducible. Until this lands in stable, the
<br>Debian policy requires reproducibility, etc. a lot of time will pass.
<br>Also the last 5-10% of packages probably will require a lot of work
<br>until they reach (nearly) 100%.
<br>
<br>But since we focus (for now) only on direct Qubes components this doesn't
<br>need to bother us. I made some tests and the Qubes specific debian
<br>packages are already reproducible when using the patched dpkg+debhelper.
<br>On similar patches for Fedora/rpm are already be worked on.
<br>
<br>So what we can do is simply modify qubes-builder to use a patched
<br>version of dpkg/rpm. We also need to make some changes to qubes-builder
<br>to record the required infos from the build-env. and be able to
<br>reproduce these (mainly the installed versions of build dependecies).
<br>With this we should get quickly reproducible .debs/.rpms.
<br>
<br>A little bit more work will be needed to get the installer ISO and the
<br>template images reproducible. But here we can use a pragmatic approach
<br>(i.e. instead of waiting to get all things fixed upstream use
<br>postprocessing and local patches) and also get this with manageable
<br>effort/time.
<br>
<br>Some notes on the proposed approach:
<br>
<br>Partially this is already done. Gitian [4] uses a predefined VM image to
<br>build the software in it. For example this is used for the Tor Browser
<br>(see [5]).
<br>
<br>I expect that there will be many small problems (which as as a whole
<br>will also be a lot of work) until software build successfully and
<br>reproducible with this setup. For example I can image that there will be
<br>programs which call the gettime syscall a different number of times
<br>depending on your CPU speed. Also stuff will break due to the
<br>manipulated syscalls (can be fixed - but needs work).
<br>
<br>
<br>[2]: <a href="https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain" \
target="_blank" rel="nofollow" \
onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fwiki.debian.o \
rg%2FReproducibleBuilds%2FExperimentalToolchain\46sa\75D\46sntz\0751\46usg\75AFQjCNFJM_y8iDy_rnRcYzpH-iOl-PzgiQ';return \
true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fwiki.debia \
n.org%2FReproducibleBuilds%2FExperimentalToolchain\46sa\75D\46sntz\0751\46usg\75AFQjCNFJM_y8iDy_rnRcYzpH-iOl-PzgiQ';return \
true;">https://wiki.debian.org/<wbr>ReproducibleBuilds/<wbr>ExperimentalToolchain</a> \
<br>[3]: <a href="https://reproducible.debian.net/reproducible.html" target="_blank" \
rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F% \
2Freproducible.debian.net%2Freproducible.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHT3OlEogn9YPEIi_PNTy00EgF8AQ';return \
true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Freproducib \
le.debian.net%2Freproducible.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHT3OlEogn9YPEIi_PNTy00EgF8AQ';return \
true;">https://reproducible.debian.<wbr>net/reproducible.html</a> <br>[4]: <a \
href="https://gitian.org/" target="_blank" rel="nofollow" \
onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgitian.org%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHbxSq_WDD33hrVEemhF9ctaW8MeA';return \
true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgitian.org%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHbxSq_WDD33hrVEemhF9ctaW8MeA';return \
true;">https://gitian.org/</a> <br>[5]: <a \
href="https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/README.build" \
target="_blank" rel="nofollow" \
onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgitweb.torpro \
ject.org%2Fbuilders%2Ftor-browser-bundle.git%2Ftree%2Fgitian%2FREADME.build\46sa\75D\46sntz\0751\46usg\75AFQjCNEuenYKsm4Aih_qzty4wp0wYzf25g';return \
true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgitweb.tor \
project.org%2Fbuilders%2Ftor-browser-bundle.git%2Ftree%2Fgitian%2FREADME.build\46sa\75D\46sntz\0751\46usg\75AFQjCNEuenYKsm4Aih_qzty4wp0wYzf25g';return \
true;">https://gitweb.torproject.org/<wbr>builders/tor-browser-bundle.<wbr>git/tree/gitian/README.build</a>
<br></blockquote><div> </div><div>Given the discussion of piggybacking off of \
Debian's efforts at implementing reproducible builds (however soon they may be \
available for use in the Qubes build process), does that mean there is a plan to \
transition away from Fedora to Debian as the base on which Qubes is built? I have \
no complaints with going in that direction (Fedora's release cycles seem to \
introduce unnecessary friction) - just interested where things are \
headed.</div><div><br></div><div>Eric</div>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups \
"qubes-devel" group.<br /> To unsubscribe from this group and stop \
receiving emails from it, send an email to <a \
href="mailto:qubes-devel+unsubscribe@googlegroups.com">qubes-devel+unsubscribe@googlegroups.com</a>.<br \
/> To post to this group, send email to <a \
href="mailto:qubes-devel@googlegroups.com">qubes-devel@googlegroups.com</a>.<br /> To \
view this discussion on the web visit <a \
href="https://groups.google.com/d/msgid/qubes-devel/f7aba36f-fa77-40dd-9609-8efad8cba0 \
5b%40googlegroups.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/ \
msgid/qubes-devel/f7aba36f-fa77-40dd-9609-8efad8cba05b%40googlegroups.com</a>.<br /> \
For more options, visit <a \
href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br \
/>
------=_Part_737_1698433385.1450809779973--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic