[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: flatpak CI and stable builds - Re: KDE Gear projects with failing CI (release/23.08) (12 Septemb
From: Ben Cooksley <bcooksley () kde ! org>
Date: 2023-09-25 8:39:21
Message-ID: CA+XidOFw2viz6EZhnfJu7agE7DhPin_+_R+0sPZzzLSJ9dADgw () mail ! gmail ! com
[Download RAW message or body]
On Mon, Sep 25, 2023 at 9:43 AM Albert Astals Cid <aacid@kde.org> wrote:
> El dijous, 21 de setembre de 2023, a les 9:54:59 (CEST), Ben Cooksley va
> escriure:
> > On Thu, Sep 21, 2023 at 8:35 AM Albert Astals Cid <aacid@kde.org> wrote:
> > > El dimecres, 20 de setembre de 2023, a les 13:17:22 (CEST), Ben
> Cooksley
> > > va
> > >
> > > escriure:
> > > > On Wed, Sep 20, 2023 at 9:42 AM Albert Astals Cid <aacid@kde.org>
> wrote:
> > > > > El dimarts, 19 de setembre de 2023, a les 22:18:40 (CEST),
> > > > >
> > > > > christoph@cullmann.io va escriure:
> > > > > > On 2023-09-19 21:35, Albert Astals Cid wrote:
> > > > > > > Please work on fixing them, otherwise i will remove the
> failing CI
> > > > > > > jobs on their 4th failing week, it is very important that CI is
> > > > > > > passing
> > > > > > > for
> > > > > > > multiple reasons.
> > > > > > >
> > > > > > > Bad news: We have 2 new repositories failing :/
> > > > > > >
> > > > > > > = FLATPAK FAILING =
> > > > > > >
> > > > > > > kate:
> > > > > > > * https://invent.kde.org/utilities/kate/-/pipelines/484147
> > > > > > >
> > > > > > > * This highlights a design problem, it's building markdown
> part
> > >
> > > from
> > >
> > > > > > > master
> > > > > > > instead of from stable branch. We can manually change the
> branch,
> > >
> > > but
> > >
> > > > > > > i
> > > > > > > would
> > > > > > > prefer a solution that doesn't mean changing lots and lots of
> > >
> > > flatpak
> > >
> > > > > > > manifests when we branch.
> > > > > >
> > > > > > Hmm, yes that sounds not nice.
> > > > > >
> > > > > > But not sure how that would work without that, seems
> > >
> > >
> https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json
> > >
> > > > > ?r>
> > > > >
> > > > > > ef_type=heads
> > > > > >
> > > > > > hard codes what to fetch.
> > > > > >
> > > > > > Given one hard codes there the
> > > > > >
> > > > > > "runtime-version": "5.15-22.08",
> > > > >
> > > > > That one is "fine", the 22.08 here it's related to the "flatpak
> kde/
> > > > > freedesktop sdk" not to Gear stuff.
> > > > >
> > > > > Yes, we will massively have to update them on master when we
> decide to
> > > > > depend
> > > > > on a new one, but it won't cause problems on the stable branches
> like
> > >
> > > the
> > >
> > > > > oner
> > > > > we're experiencing here.
> > > > >
> > > > > The problem here is
> > > > >
> > > > > {
> > > > >
> > > > > "name": "markdownpart",
> > > > > "buildsystem": "cmake-ninja",
> > > > > "sources": [
> > > > >
> > > > > {
> > > > >
> > > > > "type": "git",
> > > > > "url": "https://invent.kde.org/utilities/markdownpart.git"
> > > > >
> > > > > }
> > > > >
> > > > > ]
> > > > >
> > > > > }
> > > > >
> > > > > This unconditionally compiles the master branch of markdownpart
> repo
> > > > >
> > > > > As far as i can see there's three solutions:
> > > > >
> > > > > A) If this is just "to make sure it builds CI", we don't need
> > >
> > > markdownpart
> > >
> > > > > nor
> > > > > konsole on the flatpak since they are just runtime dependencies.
> This
> > >
> > > is a
> > >
> > > > > sub-optimal solution i'd say since it makes it so that we can't
> offer
> > >
> > > the
> > >
> > > > > package for testing in the future and makes the diff with the
> flathub
> > > > > manifest
> > > > > bigger than it needs to be
> > > >
> > > > The overall intention is for the bundles produced by the Flatpak
> jobs to
> > >
> > > be
> > >
> > > > useful for people to locally test a project build.
> > > > In the not too distant future we'll have them available from a
> Flatpak
> > > > repository for actual mainline/release branches as well.
> > > >
> > > > So the answer certainly isn't (a).
> > > >
> > > > > B) Depend on released versions, i.e. a tarball in "sources"
> instead of
> > >
> > > a
> > >
> > > > > git
> > > > > repo. This is probably suboptimal too in the sense that will
> require
> > > > > constant
> > > > > updating on master and if we offer the resulting flatpak as
> "nightly"
> > >
> > > in
> > >
> > > > > the
> > > > > future for testing it's not "nightly" as it could be.
> > > >
> > > > Guess it depends on the nature of the dependency, but in the case of
> > > > software released together you probably want to build against what
> you
> > >
> > > will
> > >
> > > > be shipping against yes.
> > > >
> > > > > C) Add a marker in the .json like branch: "kde-same-branch" and
> then
> > >
> > > have
> > >
> > > > > the
> > > > > code in
> > >
> > >
> https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f
> > >
> > > > > latpak.yml replace that "kde-same-branch" either by "master" or by
> > > > > the appropriate stable branch before actually compiling the
> flatpak. I
> > > > > think
> > > > > this would be the optimal solution but needs work.
> > > >
> > > > This is my preferred solution as well. it wouldn't be too difficult
> > > > given
> > > > we have a Python script acting as a middle-man already.
> > > > In the past we did some rewriting of the .flatpak-manifest.yml
> already.
> > > >
> > > > Depending on our requirements it may be worth tying into the same
> > > > branch-rules.yml logic that the rest of the CI system uses but this
> is
> > > > probably best answered by someone who knows the various Flatpak
> > > > manifests
> > > > we have better.
> > > >
> > > > In #flatpak:kde.org it was mentioned that this would mean that
> people
> > > > wouldn't be able to build it as easily themselves, but if we make it
> > > > well
> > > > documented (comments in the .flatpak-manifest.yml, etc) then I think
> > > > this
> > > > shouldn't be much of an issue.
> > >
> > > I had not thought of that and it's indeed not great.
> > >
> > > We could rename all the flatpak manifest that need this feature to .
> > > json.in to
> > > make it clear "it's not their final form" and that they need to be pre-
> > > processed.
> > >
> > >
> > > This does not help a lot to the "non CI user" if they want to actually
> use
> > > the
> > > manifest since they still need to pre-process it somehow :/
> >
> > At least it would make it explicit that something is required before the
> > files can be utilised elsewhere.
> >
> > Ultimately in the long term we are potentially looking at having release
> > builds we make be submitted into Flathub so this does need a solution in
> > some form or another.
>
> I don't see how that would work, for flathub we use released tarballs as
> sources for everything, not git branches, i guess we could switch to git
> tags?
Correct, we would need to use Git tags for the Flathub uploads.
The same applies to our Android / Windows builds as well.
>
>
> >
> > Realistically I think we only have two choices here:
> > a) Have our release tooling include logic that bumps/updates the
> > .flatpak-manifest.yml files to have the right branch in them; or
> > b) Have a intermediary script like flatpak-build.py do that for us
> >
> > For now the path of least resistance is probably (b) I think?
>
> Probably
>
> Cheers,
> Albert
>
Thanks,
Ben
>
> >
> > > Solution E)
> > >
> > > Since usually/mostly we have our projects be backwards API compatible
> it's
> > > usually/mostly not a problem to that kate stable uses markdown master
> > > (understanding the flatpaks generated by that CI are nightlies), we're
> > > only
> > > having this problem right now because of the KF6 transition.
> > >
> > > One potentially valid solution is to just disable flatpak CI in stable
> > > until
> > > this gets fixed, it's not great but it's just a few months.
> > >
> > > Cheers,
> > >
> > > Albert
> >
> > Cheers,
> > Ben
> >
> > > > > D) Something smarter I have not thought about.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Albert
> > > >
> > > > Cheers,
> > > > Ben
> > > >
> > > > > > I assume one will need to hard code that, too, if one creates no
> own
> > > > > > scripting.
> > > > > >
> > > > > > But I might be wrong.
> > > > > >
> > > > > > Greetings
> > > > > > Christoph
> > > > > >
> > > > > > > = FAILING UNIT TESTS =
> > > > > > >
> > > > > > > konsole:
> > > > > > > * https://invent.kde.org/utilities/konsole/-/pipelines/484148
> > > > > > >
> > > > > > > * freebsd_qt515 tests are failing
>
>
>
>
>
[Attachment #3 (text/html)]
<div dir="ltr"><div dir="ltr">On Mon, Sep 25, 2023 at 9:43 AM Albert Astals Cid \
<<a href="mailto:aacid@kde.org">aacid@kde.org</a>> wrote:<br></div><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">El dijous, 21 de \
setembre de 2023, a les 9:54:59 (CEST), Ben Cooksley va <br> escriure:<br>
> On Thu, Sep 21, 2023 at 8:35 AM Albert Astals Cid <<a \
href="mailto:aacid@kde.org" target="_blank">aacid@kde.org</a>> wrote:<br> > \
> El dimecres, 20 de setembre de 2023, a les 13:17:22 (CEST), Ben Cooksley<br> \
> > va<br> > > <br>
> > escriure:<br>
> > > On Wed, Sep 20, 2023 at 9:42 AM Albert Astals Cid <<a \
href="mailto:aacid@kde.org" target="_blank">aacid@kde.org</a>> wrote:<br> > \
> > > El dimarts, 19 de setembre de 2023, a les 22:18:40 (CEST),<br> > \
> > > <br> > > > > <a href="mailto:christoph@cullmann.io" \
target="_blank">christoph@cullmann.io</a> va escriure:<br> > > > > > \
On 2023-09-19 21:35, Albert Astals Cid wrote:<br> > > > > > > \
Please work on fixing them, otherwise i will remove the failing CI<br> > > > \
> > > jobs on their 4th failing week, it is very important that CI is<br> \
> > > > > > passing<br> > > > > > > for<br>
> > > > > > multiple reasons.<br>
> > > > > > <br>
> > > > > > Bad news: We have 2 new repositories failing :/<br>
> > > > > > <br>
> > > > > > = FLATPAK FAILING =<br>
> > > > > > <br>
> > > > > > kate:<br>
> > > > > > * <a \
href="https://invent.kde.org/utilities/kate/-/pipelines/484147" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/kate/-/pipelines/484147</a><br> > \
> > > > > <br> > > > > > > * This highlights a \
design problem, it's building markdown part<br> > > <br>
> > from<br>
> > <br>
> > > > > > master<br>
> > > > > > instead of from stable branch. We can manually change \
the branch,<br> > > <br>
> > but<br>
> > <br>
> > > > > > i<br>
> > > > > > would<br>
> > > > > > prefer a solution that doesn't mean changing lots \
and lots of<br> > > <br>
> > flatpak<br>
> > <br>
> > > > > > manifests when we branch.<br>
> > > > > <br>
> > > > > Hmm, yes that sounds not nice.<br>
> > > > > <br>
> > > > > But not sure how that would work without that, seems<br>
> > <br>
> > <a href="https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json" \
rel="noreferrer" target="_blank">https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json</a><br>
> > <br>
> > > > ?r><br>
> > > > <br>
> > > > > ef_type=heads<br>
> > > > > <br>
> > > > > hard codes what to fetch.<br>
> > > > > <br>
> > > > > Given one hard codes there the<br>
> > > > > <br>
> > > > > "runtime-version": "5.15-22.08",<br>
> > > > <br>
> > > > That one is "fine", the 22.08 here it's related to \
the "flatpak kde/<br> > > > > freedesktop sdk" not to Gear \
stuff.<br> > > > > <br>
> > > > Yes, we will massively have to update them on master when we \
decide to<br> > > > > depend<br>
> > > > on a new one, but it won't cause problems on the stable \
branches like<br> > > <br>
> > the<br>
> > <br>
> > > > oner<br>
> > > > we're experiencing here.<br>
> > > > <br>
> > > > The problem here is<br>
> > > > <br>
> > > > {<br>
> > > > <br>
> > > > "name": "markdownpart",<br>
> > > > "buildsystem": "cmake-ninja",<br>
> > > > "sources": [<br>
> > > > <br>
> > > > {<br>
> > > > <br>
> > > > "type": "git",<br>
> > > > "url": "<a \
href="https://invent.kde.org/utilities/markdownpart.git" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/markdownpart.git</a>"<br> > \
> > > <br> > > > > }<br>
> > > > <br>
> > > > ]<br>
> > > > <br>
> > > > }<br>
> > > > <br>
> > > > This unconditionally compiles the master branch of markdownpart \
repo<br> > > > > <br>
> > > > As far as i can see there's three solutions:<br>
> > > > <br>
> > > > A) If this is just "to make sure it builds CI", we \
don't need<br> > > <br>
> > markdownpart<br>
> > <br>
> > > > nor<br>
> > > > konsole on the flatpak since they are just runtime dependencies. \
This<br> > > <br>
> > is a<br>
> > <br>
> > > > sub-optimal solution i'd say since it makes it so that we \
can't offer<br> > > <br>
> > the<br>
> > <br>
> > > > package for testing in the future and makes the diff with the \
flathub<br> > > > > manifest<br>
> > > > bigger than it needs to be<br>
> > > <br>
> > > The overall intention is for the bundles produced by the Flatpak jobs \
to<br> > > <br>
> > be<br>
> > <br>
> > > useful for people to locally test a project build.<br>
> > > In the not too distant future we'll have them available from a \
Flatpak<br> > > > repository for actual mainline/release branches as \
well.<br> > > > <br>
> > > So the answer certainly isn't (a).<br>
> > > <br>
> > > > B) Depend on released versions, i.e. a tarball in \
"sources" instead of<br> > > <br>
> > a<br>
> > <br>
> > > > git<br>
> > > > repo. This is probably suboptimal too in the sense that will \
require<br> > > > > constant<br>
> > > > updating on master and if we offer the resulting flatpak as \
"nightly"<br> > > <br>
> > in<br>
> > <br>
> > > > the<br>
> > > > future for testing it's not "nightly" as it could \
be.<br> > > > <br>
> > > Guess it depends on the nature of the dependency, but in the case \
of<br> > > > software released together you probably want to build against \
what you<br> > > <br>
> > will<br>
> > <br>
> > > be shipping against yes.<br>
> > > <br>
> > > > C) Add a marker in the .json like branch: \
"kde-same-branch" and then<br> > > <br>
> > have<br>
> > <br>
> > > > the<br>
> > > > code in<br>
> > <br>
> > <a href="https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f" \
rel="noreferrer" target="_blank">https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f</a><br>
> > <br>
> > > > latpak.yml replace that "kde-same-branch" either by \
"master" or by<br> > > > > the appropriate stable branch before \
actually compiling the flatpak. I<br> > > > > think<br>
> > > > this would be the optimal solution but needs work.<br>
> > > <br>
> > > This is my preferred solution as well. it wouldn't be too \
difficult<br> > > > given<br>
> > > we have a Python script acting as a middle-man already.<br>
> > > In the past we did some rewriting of the .flatpak-manifest.yml \
already.<br> > > > <br>
> > > Depending on our requirements it may be worth tying into the same<br>
> > > branch-rules.yml logic that the rest of the CI system uses but this \
is<br> > > > probably best answered by someone who knows the various \
Flatpak<br> > > > manifests<br>
> > > we have better.<br>
> > > <br>
> > > In #flatpak:<a href="http://kde.org" rel="noreferrer" \
target="_blank">kde.org</a> it was mentioned that this would mean that people<br> \
> > > wouldn't be able to build it as easily themselves, but if we make \
it<br> > > > well<br>
> > > documented (comments in the .flatpak-manifest.yml, etc) then I \
think<br> > > > this<br>
> > > shouldn't be much of an issue.<br>
> > <br>
> > I had not thought of that and it's indeed not great.<br>
> > <br>
> > We could rename all the flatpak manifest that need this feature to .<br>
> > <a href="http://json.in" rel="noreferrer" target="_blank">json.in</a> \
to<br> > > make it clear "it's not their final form" and that \
they need to be pre-<br> > > processed.<br>
> > <br>
> > <br>
> > This does not help a lot to the "non CI user" if they want to \
actually use<br> > > the<br>
> > manifest since they still need to pre-process it somehow :/<br>
> <br>
> At least it would make it explicit that something is required before the<br>
> files can be utilised elsewhere.<br>
> <br>
> Ultimately in the long term we are potentially looking at having release<br>
> builds we make be submitted into Flathub so this does need a solution in<br>
> some form or another.<br>
<br>
I don't see how that would work, for flathub we use released tarballs as <br>
sources for everything, not git branches, i guess we could switch to git \
tags?</blockquote><div><br></div><div>Correct, we would need to use Git tags for the \
Flathub uploads. </div><div>The same applies to our Android / Windows builds as \
well.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> <br>
> <br>
> Realistically I think we only have two choices here:<br>
> a) Have our release tooling include logic that bumps/updates the<br>
> .flatpak-manifest.yml files to have the right branch in them; or<br>
> b) Have a intermediary script like flatpak-build.py do that for us<br>
> <br>
> For now the path of least resistance is probably (b) I think?<br>
<br>
Probably<br>
<br>
Cheers,<br>
Albert<br></blockquote><div><br></div><div>Thanks,</div><div>Ben</div><div> \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> <br>
> <br>
> > Solution E)<br>
> > <br>
> > Since usually/mostly we have our projects be backwards API compatible \
it's<br> > > usually/mostly not a problem to that kate stable uses markdown \
master<br> > > (understanding the flatpaks generated by that CI are nightlies), \
we're<br> > > only<br>
> > having this problem right now because of the KF6 transition.<br>
> > <br>
> > One potentially valid solution is to just disable flatpak CI in stable<br>
> > until<br>
> > this gets fixed, it's not great but it's just a few months.<br>
> > <br>
> > Cheers,<br>
> > <br>
> > Albert<br>
> <br>
> Cheers,<br>
> Ben<br>
> <br>
> > > > D) Something smarter I have not thought about.<br>
> > > > <br>
> > > > Cheers,<br>
> > > > <br>
> > > > Albert<br>
> > > <br>
> > > Cheers,<br>
> > > Ben<br>
> > > <br>
> > > > > I assume one will need to hard code that, too, if one \
creates no own<br> > > > > > scripting.<br>
> > > > > <br>
> > > > > But I might be wrong.<br>
> > > > > <br>
> > > > > Greetings<br>
> > > > > Christoph<br>
> > > > > <br>
> > > > > > = FAILING UNIT TESTS =<br>
> > > > > > <br>
> > > > > > konsole:<br>
> > > > > > * <a \
href="https://invent.kde.org/utilities/konsole/-/pipelines/484148" rel="noreferrer" \
target="_blank">https://invent.kde.org/utilities/konsole/-/pipelines/484148</a><br> \
> > > > > > <br> > > > > > > * \
freebsd_qt515 tests are failing<br> <br>
<br>
<br>
<br>
</blockquote></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic