[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: flatpak CI and stable builds - Re: KDE Gear projects with failing CI (release/23.08) (12 Septemb
From: Ben Cooksley <bcooksley () kde ! org>
Date: 2023-09-21 7:54:59
Message-ID: CA+XidOGmq8OGJ+9mYq8LUFUM+LaZgrV7F8+bRzimwkB47fZ=kg () mail ! gmail ! com
[Download RAW message or body]
On Thu, Sep 21, 2023 at 8:35 AM Albert Astals Cid <aacid@kde.org> wrote:
> El dimecres, 20 de setembre de 2023, a les 13:17:22 (CEST), Ben Cooksley
> va
> escriure:
> > On Wed, Sep 20, 2023 at 9:42 AM Albert Astals Cid <aacid@kde.org> wrote:
> > > El dimarts, 19 de setembre de 2023, a les 22:18:40 (CEST),
> > >
> > > christoph@cullmann.io va escriure:
> > > > On 2023-09-19 21:35, Albert Astals Cid wrote:
> > > > > Please work on fixing them, otherwise i will remove the failing CI
> > > > > jobs on their 4th failing week, it is very important that CI is
> > > > > passing
> > > > > for
> > > > > multiple reasons.
> > > > >
> > > > > Bad news: We have 2 new repositories failing :/
> > > > >
> > > > > = FLATPAK FAILING =
> > > > >
> > > > > kate:
> > > > > * https://invent.kde.org/utilities/kate/-/pipelines/484147
> > > > >
> > > > > * This highlights a design problem, it's building markdown part
> from
> > > > >
> > > > > master
> > > > > instead of from stable branch. We can manually change the branch,
> but
> > > > > i
> > > > > would
> > > > > prefer a solution that doesn't mean changing lots and lots of
> flatpak
> > > > > manifests when we branch.
> > > >
> > > > Hmm, yes that sounds not nice.
> > > >
> > > > But not sure how that would work without that, seems
> > >
> > >
> https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json
> > > ?r>
> > > > ef_type=heads
> > > >
> > > > hard codes what to fetch.
> > > >
> > > > Given one hard codes there the
> > > >
> > > > "runtime-version": "5.15-22.08",
> > >
> > > That one is "fine", the 22.08 here it's related to the "flatpak kde/
> > > freedesktop sdk" not to Gear stuff.
> > >
> > > Yes, we will massively have to update them on master when we decide to
> > > depend
> > > on a new one, but it won't cause problems on the stable branches like
> the
> > > oner
> > > we're experiencing here.
> > >
> > > The problem here is
> > >
> > > {
> > >
> > > "name": "markdownpart",
> > > "buildsystem": "cmake-ninja",
> > > "sources": [
> > >
> > > {
> > >
> > > "type": "git",
> > > "url": "https://invent.kde.org/utilities/markdownpart.git"
> > >
> > > }
> > >
> > > ]
> > >
> > > }
> > >
> > > This unconditionally compiles the master branch of markdownpart repo
> > >
> > > As far as i can see there's three solutions:
> > >
> > > A) If this is just "to make sure it builds CI", we don't need
> markdownpart
> > > nor
> > > konsole on the flatpak since they are just runtime dependencies. This
> is a
> > > sub-optimal solution i'd say since it makes it so that we can't offer
> the
> > > package for testing in the future and makes the diff with the flathub
> > > manifest
> > > bigger than it needs to be
> >
> > The overall intention is for the bundles produced by the Flatpak jobs to
> be
> > useful for people to locally test a project build.
> > In the not too distant future we'll have them available from a Flatpak
> > repository for actual mainline/release branches as well.
> >
> > So the answer certainly isn't (a).
> >
> > > B) Depend on released versions, i.e. a tarball in "sources" instead of
> a
> > > git
> > > repo. This is probably suboptimal too in the sense that will require
> > > constant
> > > updating on master and if we offer the resulting flatpak as "nightly"
> in
> > > the
> > > future for testing it's not "nightly" as it could be.
> >
> > Guess it depends on the nature of the dependency, but in the case of
> > software released together you probably want to build against what you
> will
> > be shipping against yes.
> >
> > > C) Add a marker in the .json like branch: "kde-same-branch" and then
> have
> > > the
> > > code in
> > >
> https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f
> > > latpak.yml replace that "kde-same-branch" either by "master" or by
> > > the appropriate stable branch before actually compiling the flatpak. I
> > > think
> > > this would be the optimal solution but needs work.
> >
> > This is my preferred solution as well. it wouldn't be too difficult given
> > we have a Python script acting as a middle-man already.
> > In the past we did some rewriting of the .flatpak-manifest.yml already.
> >
> > Depending on our requirements it may be worth tying into the same
> > branch-rules.yml logic that the rest of the CI system uses but this is
> > probably best answered by someone who knows the various Flatpak manifests
> > we have better.
> >
> > In #flatpak:kde.org it was mentioned that this would mean that people
> > wouldn't be able to build it as easily themselves, but if we make it well
> > documented (comments in the .flatpak-manifest.yml, etc) then I think this
> > shouldn't be much of an issue.
>
> I had not thought of that and it's indeed not great.
>
> We could rename all the flatpak manifest that need this feature to .
> json.in to
> make it clear "it's not their final form" and that they need to be pre-
> processed.
> This does not help a lot to the "non CI user" if they want to actually use
> the
> manifest since they still need to pre-process it somehow :/
>
At least it would make it explicit that something is required before the
files can be utilised elsewhere.
Ultimately in the long term we are potentially looking at having release
builds we make be submitted into Flathub so this does need a solution in
some form or another.
Realistically I think we only have two choices here:
a) Have our release tooling include logic that bumps/updates the
.flatpak-manifest.yml files to have the right branch in them; or
b) Have a intermediary script like flatpak-build.py do that for us
For now the path of least resistance is probably (b) I think?
>
>
> Solution E)
>
> Since usually/mostly we have our projects be backwards API compatible it's
> usually/mostly not a problem to that kate stable uses markdown master
> (understanding the flatpaks generated by that CI are nightlies), we're
> only
> having this problem right now because of the KF6 transition.
>
> One potentially valid solution is to just disable flatpak CI in stable
> until
> this gets fixed, it's not great but it's just a few months.
>
> Cheers,
> Albert
>
Cheers,
Ben
>
>
> >
> > > D) Something smarter I have not thought about.
> > >
> > > Cheers,
> > >
> > > Albert
> >
> > Cheers,
> > Ben
> >
> > > > I assume one will need to hard code that, too, if one creates no own
> > > > scripting.
> > > >
> > > > But I might be wrong.
> > > >
> > > > Greetings
> > > > Christoph
> > > >
> > > > > = FAILING UNIT TESTS =
> > > > >
> > > > > konsole:
> > > > > * https://invent.kde.org/utilities/konsole/-/pipelines/484148
> > > > >
> > > > > * freebsd_qt515 tests are failing
>
>
>
>
>
[Attachment #3 (text/html)]
<div dir="ltr"><div dir="ltr">On Thu, Sep 21, 2023 at 8:35 AM Albert Astals Cid \
<<a href="mailto:aacid@kde.org">aacid@kde.org</a>> wrote:<br></div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">El dimecres, 20 de \
setembre de 2023, a les 13:17:22 (CEST), Ben Cooksley va <br> escriure:<br>
> On Wed, Sep 20, 2023 at 9:42 AM Albert Astals Cid <<a \
href="mailto:aacid@kde.org" target="_blank">aacid@kde.org</a>> wrote:<br> > \
> El dimarts, 19 de setembre de 2023, a les 22:18:40 (CEST),<br> > > <br>
> > <a href="mailto:christoph@cullmann.io" \
target="_blank">christoph@cullmann.io</a> va escriure:<br> > > > On \
2023-09-19 21:35, Albert Astals Cid wrote:<br> > > > > Please work on \
fixing them, otherwise i will remove the failing CI<br> > > > > jobs on \
their 4th failing week, it is very important that CI is<br> > > > > \
passing<br> > > > > for<br>
> > > > multiple reasons.<br>
> > > > <br>
> > > > Bad news: We have 2 new repositories failing :/<br>
> > > > <br>
> > > > = FLATPAK FAILING =<br>
> > > > <br>
> > > > kate:<br>
> > > > * <a \
href="https://invent.kde.org/utilities/kate/-/pipelines/484147" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/kate/-/pipelines/484147</a><br> > \
> > > <br> > > > > * This highlights a design problem, \
it's building markdown part from<br> > > > > <br>
> > > > master<br>
> > > > instead of from stable branch. We can manually change the branch, \
but<br> > > > > i<br>
> > > > would<br>
> > > > prefer a solution that doesn't mean changing lots and lots of \
flatpak<br> > > > > manifests when we branch.<br>
> > > <br>
> > > Hmm, yes that sounds not nice.<br>
> > > <br>
> > > But not sure how that would work without that, seems<br>
> > <br>
> > <a href="https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json" \
rel="noreferrer" target="_blank">https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json</a><br>
> > ?r> <br>
> > > ef_type=heads<br>
> > > <br>
> > > hard codes what to fetch.<br>
> > > <br>
> > > Given one hard codes there the<br>
> > > <br>
> > > "runtime-version": "5.15-22.08",<br>
> > <br>
> > That one is "fine", the 22.08 here it's related to the \
"flatpak kde/<br> > > freedesktop sdk" not to Gear stuff.<br>
> > <br>
> > Yes, we will massively have to update them on master when we decide to<br>
> > depend<br>
> > on a new one, but it won't cause problems on the stable branches like \
the<br> > > oner<br>
> > we're experiencing here.<br>
> > <br>
> > The problem here is<br>
> > <br>
> > {<br>
> > <br>
> > "name": "markdownpart",<br>
> > "buildsystem": "cmake-ninja",<br>
> > "sources": [<br>
> > <br>
> > {<br>
> > <br>
> > "type": "git",<br>
> > "url": "<a \
href="https://invent.kde.org/utilities/markdownpart.git" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/markdownpart.git</a>"<br> > \
> <br> > > }<br>
> > <br>
> > ]<br>
> > <br>
> > }<br>
> > <br>
> > This unconditionally compiles the master branch of markdownpart repo<br>
> > <br>
> > As far as i can see there's three solutions:<br>
> > <br>
> > A) If this is just "to make sure it builds CI", we don't need \
markdownpart<br> > > nor<br>
> > konsole on the flatpak since they are just runtime dependencies. This is \
a<br> > > sub-optimal solution i'd say since it makes it so that we \
can't offer the<br> > > package for testing in the future and makes the \
diff with the flathub<br> > > manifest<br>
> > bigger than it needs to be<br>
> <br>
> The overall intention is for the bundles produced by the Flatpak jobs to be<br>
> useful for people to locally test a project build.<br>
> In the not too distant future we'll have them available from a Flatpak<br>
> repository for actual mainline/release branches as well.<br>
> <br>
> So the answer certainly isn't (a).<br>
> <br>
> > B) Depend on released versions, i.e. a tarball in "sources" \
instead of a<br> > > git<br>
> > repo. This is probably suboptimal too in the sense that will require<br>
> > constant<br>
> > updating on master and if we offer the resulting flatpak as \
"nightly" in<br> > > the<br>
> > future for testing it's not "nightly" as it could be.<br>
> <br>
> Guess it depends on the nature of the dependency, but in the case of<br>
> software released together you probably want to build against what you will<br>
> be shipping against yes.<br>
> <br>
> > C) Add a marker in the .json like branch: "kde-same-branch" and \
then have<br> > > the<br>
> > code in<br>
> > <a href="https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f" \
rel="noreferrer" target="_blank">https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f</a><br>
> > latpak.yml replace that "kde-same-branch" either by \
"master" or by<br> > > the appropriate stable branch before actually \
compiling the flatpak. I<br> > > think<br>
> > this would be the optimal solution but needs work.<br>
> <br>
> This is my preferred solution as well. it wouldn't be too difficult \
given<br> > we have a Python script acting as a middle-man already.<br>
> In the past we did some rewriting of the .flatpak-manifest.yml already.<br>
> <br>
> Depending on our requirements it may be worth tying into the same<br>
> branch-rules.yml logic that the rest of the CI system uses but this is<br>
> probably best answered by someone who knows the various Flatpak manifests<br>
> we have better.<br>
> <br>
> In #flatpak:<a href="http://kde.org" rel="noreferrer" \
target="_blank">kde.org</a> it was mentioned that this would mean that people<br> \
> wouldn't be able to build it as easily themselves, but if we make it \
well<br> > documented (comments in the .flatpak-manifest.yml, etc) then I think \
this<br> > shouldn't be much of an issue.<br>
<br>
I had not thought of that and it's indeed not great.<br>
<br>
We could rename all the flatpak manifest that need this feature to .<a \
href="http://json.in" rel="noreferrer" target="_blank">json.in</a> to <br> make it \
clear "it's not their final form" and that they need to be pre-<br> \
processed. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
This does not help a lot to the "non CI user" if they want to actually use \
the <br> manifest since they still need to pre-process it somehow \
:/<br></blockquote><div><br></div><div>At least it would make it explicit that \
something is required before the files can be utilised \
elsewhere.</div><div><br></div><div>Ultimately in the long term we are potentially \
looking at having release builds we make be submitted into Flathub so this does need \
a solution in some form or another.<br></div><div><br></div><div>Realistically I \
think we only have two choices here:</div><div>a) Have our release tooling include \
logic that bumps/updates the .flatpak-manifest.yml files to have the right branch in \
them; or</div><div>b) Have a intermediary script like flatpak-build.py do that for \
us</div><div><br></div><div>For now the path of least resistance is probably (b) I \
think?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
<br>
Solution E)<br>
<br>
Since usually/mostly we have our projects be backwards API compatible it's <br>
usually/mostly not a problem to that kate stable uses markdown master <br>
(understanding the flatpaks generated by that CI are nightlies), we're only <br>
having this problem right now because of the KF6 transition.<br>
<br>
One potentially valid solution is to just disable flatpak CI in stable until <br>
this gets fixed, it's not great but it's just a few months.<br>
<br>
Cheers,<br>
Albert<br></blockquote><div><br></div><div>Cheers,</div><div>Ben</div><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> <br>
<br>
> <br>
> > D) Something smarter I have not thought about.<br>
> > <br>
> > Cheers,<br>
> > <br>
> > Albert<br>
> <br>
> Cheers,<br>
> Ben<br>
> <br>
> > > I assume one will need to hard code that, too, if one creates no \
own<br> > > > scripting.<br>
> > > <br>
> > > But I might be wrong.<br>
> > > <br>
> > > Greetings<br>
> > > Christoph<br>
> > > <br>
> > > > = FAILING UNIT TESTS =<br>
> > > > <br>
> > > > konsole:<br>
> > > > * <a \
href="https://invent.kde.org/utilities/konsole/-/pipelines/484148" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/konsole/-/pipelines/484148</a><br> \
> > > > <br> > > > > * freebsd_qt515 tests are \
failing<br> <br>
<br>
<br>
<br>
</blockquote></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic