[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-community
Subject: Re: Post-MegaRelease projects
From: Ben Cooksley <bcooksley () kde ! org>
Date: 2024-02-24 9:23:55
Message-ID: CA+XidOHue+U8Lv4Cf12zb=tHp_aoJ569r66GbiqQ1q903xBimw () mail ! gmail ! com
[Download RAW message or body]
On Sat, Feb 24, 2024 at 9:21 PM Volker Krause <vkrause@kde.org> wrote:
> On Saturday, 24 February 2024 04:31:49 CET Ben Cooksley wrote:
> > On Fri, Feb 23, 2024 at 11:12 PM Sune Vuorela <nospam@vuorela.dk> wrote:
> > > On 2024-02-22, Nate Graham <nate@kde.org> wrote:
> > > > I've started pondering post-megarelease projects. We've spent so
> long on
> > > > porting and bugfixing that I think it might be useful to shift gears
> to
> > > > feature work, and I'd like to brainstorm potential large-scale
> projects
> > > > and gauge the level of interest in putting resources into them soon.
> > >
> > > A bit more from the devops end that I'd love to see people tackle:
> > > - Ensure frameworks and app unit tests interacting with windows can
> run
> > >
> > > on Windows.
> > > More details: The following fails on our windows CI
> > >
> https://invent.kde.org/sune/windows-test-thingie/-/blob/master/main.cpp
> > > I find it weird that we are spending resources on putting things in
> > > the windows store and making apps available on windows, but we can't
> > > actually have passing tests in our CI.
> >
> > This unfortunately will not be easy to solve.
>
> And that's fine, we are in dreaming territory here anyway :)
>
> > One of the key things that we've learned out of doing CI, as has been
> > showcased by FreeBSD in particular, is that the builders need to be
> > ephemeral - that is only around for the build in question that is being
> run.
> > We're currently accomplishing this by using containers - via Podman (for
> > Linux/Android/FreeBSD) and Docker (for Windows).
> >
> > Containers also offer us the advantage of allowing people to easily
> > reproduce the CI environment on their local system without too much
> trouble.
> >
> > For Windows however, Microsoft has limited the container stack to not
> allow
> > anything GUI related to work. The underlying libraries may be there, but
> > the equivalent display server components are not operational.
> >
> > To complicate things further, on Windows certain permissions are
> restricted
> > to the interactive console and are not possible to do as either a
> scheduled
> > task or as a system service.
> > Usage of existing Windows automation frameworks such as Powershell
> Remoting
> > or SSH will therefore not work if we want things to perfectly replicate a
> > end user environment - because those will run the command(s) as part of a
> > non-interactive session (even if the user we connect as is the same one
> > logged in on the desktop console).
> >
> > Resolving this will therefore not only require running full sized Windows
> > VMs (on an ephemeral basis to avoid the recently resolved issues that
> used
> > to plague FreeBSD), but would also need the VM to automatically login and
> > spawn an agent as part of the interactive desktop session that we would
> > connect to in order to run the CI tests.
> >
> > The spawning of a VM would require refactoring the setup of our poor CI
> > workers (again - after they were only just recently completely rebuilt to
> > allow the transition to Podman to fix flatpak-builder) to make use of
> > something like:
> > -
> >
> https://forum.gitlab.com/t/custom-executor-into-windows-vm-using-linux-kvm-l
> > ibvirt/72713/5 -
> > https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html
> >
> > We would still have to write the agent that receives our commands
> > (something like
> > https://gist.github.com/cschwede/3e2c025408ab4af531651098331cce45 maybe)
> >
> > We would still have to solve the issue of how to share disk images among
> > our nodes so they were built (ideally would be able to use Gitlab's
> > Container Registry for this, which is something Tart can do - Tart being
> a
> > virtualisation management utility for ARM based Macs), as well as
> > automating the installation of the various OSes we need to run this way.
> > See
> >
> https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/autom
> > ate-windows-setup?view=windows-11 for some notes on that for Windows.
> >
> > The big downside to all of that of course is that the worker nodes would
> > take longer to startup, and it would make sharing build artifacts much
> more
> > difficult and/or less efficient.
> >
> > > - Find a way to run unit tests on android CI.
> >
> > Likewise, this is very non-trivial - from my understanding our tooling
> > currently for building Android software is centered around it all being
> > cross compiled rather than being built on the native architecture it will
> > be run on.
> > Naturally tests cannot be run (unless you emulate, which I guess we could
> > consider...) if the CPU architectures are not compatible.
> >
> > Even if you emulate though, I imagine we would need to provide a full
> > Android system setup (including all of it's relevant bits of system
> > infrastructure, such as it's display server DisplayFlinger) in order for
> > tests to truly be of use.
> > The path of least resistance is probably by making use of Google's
> existing
> > Android emulator - although i've no idea how you would tie that into CI.
>
> Right, the Android emulator is probably the first thing to look at for
> this.
> That alone isn't enough though, we can't just copy over a unit test
> executable
> and run it there, this needs to be wrapped in an APK (and which
> potentially
> needs to be given permissions, as we don't want system UI dialogs popping
> up,
> etc).
>
> The Qt CI seems to be doing something like this meanwhile, might be worth
> investigating how this is done there.
>
> > We would need to have our build chains ready to build on a native system
> > before we could think about this I think. Building Android x86/x64
> wouldn't
> > cover things off properly due as it won't reflect the actual CPUs being
> > used on end-user devices (and ARM processors can expose issues that don't
> > happen on x86/x64 based systems)
>
> For Android I'd rate that less relevant actually, we'd get by far the most
> things covered with x86_64 as well. I'm only aware of one ARM-specific
> runtime
> issue recently, and that was ARM32 only even. The vast majority of
> problems
> are either related to APK packaging or interaction with (changing) Android
> platform infrastructure, all of which is platform-independent. And the
> majority of ARM-specific issues we actually encounter during building.
>
> ARM64 native builds nevertheless belong on this wishlist, but IMHO more
> importantly for ARM Flatpaks for the mobile apps.
>
I already have a ticket for that - https://phabricator.kde.org/T17120
Hardware will likely be provided via a CAX31 / CAX41 (
https://www.hetzner.com/cloud/) as the number of builds isn't likely to be
huge and therefore cannot justify a RX170 (
https://www.hetzner.com/dedicated-rootserver/matrix-rx/)
(which would make more sense if all our Android builds were ARM native)
At this stage i'd like to get us comfortably through into the post-Binary
Factory world before anything further takes place though (some teething
issues have popped up that weren't immediately evident prior to it's
decommissioning that will need resolving).
From my side, the next big thing to start breaking down and building via CI
would be docs.kde.org.
>
> > > - Make autotests guarding on all our CI's.
> > >
> > > - Clazy and clang-tidy and cppcheck on all our repositories in CI
> > >
> > > /Sune
> >
> > Hopefully the above is food for thought as to why we don't have the
> above -
> > don't mean to say it can't be done, but it certainly is not trivial to
> > accomplish...
>
> Sure, and likewise the wishlist isn't meant as criticism of what we have,
> on
> the contrary, IMHO our CI/CD system has made huge steps forward with the
> migration to Gitlab and to container runners as well as the new signing
> service, but now we are hooked for more ;-)
>
> Yes, those are ambitious ideas, but they would significantly help
> application
> development I think, and assuming Nate's original question was also about
> where the e.V. might want to invest resources into going forward, our
> CI/CD
> infrastructure would IMHO be a good area to (continue to) invest in.
>
> Regards,
> Volker
>
Cheers,
Ben
[Attachment #3 (text/html)]
<div dir="ltr"><div dir="ltr">On Sat, Feb 24, 2024 at 9:21 PM Volker Krause <<a \
href="mailto:vkrause@kde.org">vkrause@kde.org</a>> wrote:<br></div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Saturday, 24 \
February 2024 04:31:49 CET Ben Cooksley wrote:<br> > On Fri, Feb 23, 2024 at \
11:12 PM Sune Vuorela <<a href="mailto:nospam@vuorela.dk" \
target="_blank">nospam@vuorela.dk</a>> wrote:<br> > > On 2024-02-22, Nate \
Graham <<a href="mailto:nate@kde.org" target="_blank">nate@kde.org</a>> \
wrote:<br> > > > I've started pondering post-megarelease projects. \
We've spent so long on<br> > > > porting and bugfixing that I think it \
might be useful to shift gears to<br> > > > feature work, and I'd like \
to brainstorm potential large-scale projects<br> > > > and gauge the level \
of interest in putting resources into them soon.<br> > > <br>
> > A bit more from the devops end that I'd love to see people tackle:<br>
> > - Ensure frameworks and app unit tests interacting with windows can \
run<br> > > <br>
> > on Windows.<br>
> > More details: The following fails on our windows CI<br>
> > <a href="https://invent.kde.org/sune/windows-test-thingie/-/blob/master/main.cpp" \
rel="noreferrer" target="_blank">https://invent.kde.org/sune/windows-test-thingie/-/blob/master/main.cpp</a><br>
> > I find it weird that we are spending resources on putting things \
in<br> > > the windows store and making apps available on windows, but we \
can't<br> > > actually have passing tests in our CI.<br>
> <br>
> This unfortunately will not be easy to solve.<br>
<br>
And that's fine, we are in dreaming territory here anyway :)<br>
<br>
> One of the key things that we've learned out of doing CI, as has been<br>
> showcased by FreeBSD in particular, is that the builders need to be<br>
> ephemeral - that is only around for the build in question that is being run.<br>
> We're currently accomplishing this by using containers - via Podman (for<br>
> Linux/Android/FreeBSD) and Docker (for Windows).<br>
> <br>
> Containers also offer us the advantage of allowing people to easily<br>
> reproduce the CI environment on their local system without too much trouble.<br>
> <br>
> For Windows however, Microsoft has limited the container stack to not allow<br>
> anything GUI related to work. The underlying libraries may be there, but<br>
> the equivalent display server components are not operational.<br>
> <br>
> To complicate things further, on Windows certain permissions are restricted<br>
> to the interactive console and are not possible to do as either a scheduled<br>
> task or as a system service.<br>
> Usage of existing Windows automation frameworks such as Powershell Remoting<br>
> or SSH will therefore not work if we want things to perfectly replicate a<br>
> end user environment - because those will run the command(s) as part of a<br>
> non-interactive session (even if the user we connect as is the same one<br>
> logged in on the desktop console).<br>
> <br>
> Resolving this will therefore not only require running full sized Windows<br>
> VMs (on an ephemeral basis to avoid the recently resolved issues that used<br>
> to plague FreeBSD), but would also need the VM to automatically login and<br>
> spawn an agent as part of the interactive desktop session that we would<br>
> connect to in order to run the CI tests.<br>
> <br>
> The spawning of a VM would require refactoring the setup of our poor CI<br>
> workers (again - after they were only just recently completely rebuilt to<br>
> allow the transition to Podman to fix flatpak-builder) to make use of<br>
> something like:<br>
> -<br>
> <a href="https://forum.gitlab.com/t/custom-executor-into-windows-vm-using-linux-kvm-l" \
rel="noreferrer" target="_blank">https://forum.gitlab.com/t/custom-executor-into-windows-vm-using-linux-kvm-l</a><br>
> ibvirt/72713/5 -<br>
> <a href="https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html" \
rel="noreferrer" target="_blank">https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html</a><br>
> <br>
> We would still have to write the agent that receives our commands<br>
> (something like<br>
> <a href="https://gist.github.com/cschwede/3e2c025408ab4af531651098331cce45" \
rel="noreferrer" target="_blank">https://gist.github.com/cschwede/3e2c025408ab4af531651098331cce45</a> \
maybe)<br> > <br>
> We would still have to solve the issue of how to share disk images among<br>
> our nodes so they were built (ideally would be able to use Gitlab's<br>
> Container Registry for this, which is something Tart can do - Tart being a<br>
> virtualisation management utility for ARM based Macs), as well as<br>
> automating the installation of the various OSes we need to run this way.<br>
> See<br>
> <a href="https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/autom" \
rel="noreferrer" target="_blank">https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/autom</a><br>
> ate-windows-setup?view=windows-11 for some notes on that for Windows.<br>
> <br>
> The big downside to all of that of course is that the worker nodes would<br>
> take longer to startup, and it would make sharing build artifacts much more<br>
> difficult and/or less efficient.<br>
> <br>
> > - Find a way to run unit tests on android CI.<br>
> <br>
> Likewise, this is very non-trivial - from my understanding our tooling<br>
> currently for building Android software is centered around it all being<br>
> cross compiled rather than being built on the native architecture it will<br>
> be run on.<br>
> Naturally tests cannot be run (unless you emulate, which I guess we could<br>
> consider...) if the CPU architectures are not compatible.<br>
> <br>
> Even if you emulate though, I imagine we would need to provide a full<br>
> Android system setup (including all of it's relevant bits of system<br>
> infrastructure, such as it's display server DisplayFlinger) in order for<br>
> tests to truly be of use.<br>
> The path of least resistance is probably by making use of Google's \
existing<br> > Android emulator - although i've no idea how you would tie that \
into CI.<br> <br>
Right, the Android emulator is probably the first thing to look at for this. <br>
That alone isn't enough though, we can't just copy over a unit test \
executable <br> and run it there, this needs to be wrapped in an APK (and which \
potentially <br> needs to be given permissions, as we don't want system UI \
dialogs popping up, <br> etc).<br>
<br>
The Qt CI seems to be doing something like this meanwhile, might be worth <br>
investigating how this is done there.<br>
<br>
> We would need to have our build chains ready to build on a native system<br>
> before we could think about this I think. Building Android x86/x64 \
wouldn't<br> > cover things off properly due as it won't reflect the \
actual CPUs being<br> > used on end-user devices (and ARM processors can expose \
issues that don't<br> > happen on x86/x64 based systems)<br>
<br>
For Android I'd rate that less relevant actually, we'd get by far the most \
<br> things covered with x86_64 as well. I'm only aware of one ARM-specific \
runtime <br> issue recently, and that was ARM32 only even. The vast majority of \
problems <br> are either related to APK packaging or interaction with (changing) \
Android <br> platform infrastructure, all of which is platform-independent. And the \
<br> majority of ARM-specific issues we actually encounter during building.<br>
<br>
ARM64 native builds nevertheless belong on this wishlist, but IMHO more <br>
importantly for ARM Flatpaks for the mobile \
apps.<br></blockquote><div><br></div><div>I already have a ticket for that - <a \
href="https://phabricator.kde.org/T17120">https://phabricator.kde.org/T17120</a></div><div><br></div><div>Hardware \
will likely be provided via a CAX31 / CAX41 (<a \
href="https://www.hetzner.com/cloud/">https://www.hetzner.com/cloud/</a>) as the \
number of builds isn't likely to be huge and therefore cannot justify a RX170 (<a \
href="https://www.hetzner.com/dedicated-rootserver/matrix-rx/">https://www.hetzner.com/dedicated-rootserver/matrix-rx/</a>)</div><div>(which \
would make more sense if all our Android builds were ARM \
native)</div><div><br></div><div>At this stage i'd like to get us comfortably \
through into the post-Binary Factory world before anything further takes place though \
(some teething issues have popped up that weren't immediately evident prior to \
it's decommissioning that will need resolving).</div><div><br></div><div>From my \
side, the next big thing to start breaking down and building via CI would be <a \
href="http://docs.kde.org">docs.kde.org</a>.</div><div> </div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"> <br>
> > - Make autotests guarding on all our CI's.<br>
> > <br>
> > - Clazy and clang-tidy and cppcheck on all our repositories in CI<br>
> > <br>
> > /Sune<br>
> <br>
> Hopefully the above is food for thought as to why we don't have the above \
-<br> > don't mean to say it can't be done, but it certainly is not \
trivial to<br> > accomplish...<br>
<br>
Sure, and likewise the wishlist isn't meant as criticism of what we have, on <br>
the contrary, IMHO our CI/CD system has made huge steps forward with the <br>
migration to Gitlab and to container runners as well as the new signing <br>
service, but now we are hooked for more ;-)<br>
<br>
Yes, those are ambitious ideas, but they would significantly help application <br>
development I think, and assuming Nate's original question was also about <br>
where the e.V. might want to invest resources into going forward, our CI/CD <br>
infrastructure would IMHO be a good area to (continue to) invest in.<br>
<br>
Regards,<br>
Volker<br></blockquote><div><br></div><div>Cheers,</div><div>Ben </div></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic