[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-release-team
Subject:    Re: [Kde-pim] 4.11.0 release and tests
From:       Ben Cooksley <bcooksley () kde ! org>
Date:       2013-08-07 8:34:32
Message-ID: CA+XidOHY3qbDgqjtiR4_crk9GRzbRPyJO6gQhwa8bU-6dwQA5g () mail ! gmail ! com
[Download RAW message or body]

On Tue, Aug 6, 2013 at 7:47 PM, Volker Krause <vkrause@kde.org> wrote:
> On Monday 05 August 2013 21:54:33 Allen Winter wrote:
> > On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote:
> > > @people, please keep both lists on the loop.
> > > 
> > > One of the things we agreed we wanted for the 4.11 release was no \
> > > failing tests on jenkins.
> > > 
> > > We've almost achieved that.
> > > 
> > > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false
> > > 
> > > We have only 8 failing tests, all of them in kdepim-*.
> 
> Yep, although not entirely surprising considering that's where the \
> majority of all our tests seem to be (>1000). That's no excuse of course, \
> just statistics ;)
> 
> Looking at the fail graphs for kdepim*, you can also see that this plan \
> had some positive effects :)
> 
> > > My question to the kdepim people:
> > > * Why are those tests failing?
> 
> Looking at kdepimlibs-stable:
> 
> > > > TestSuite.kblog-testmetaweblog
> 
> IIRC that's an online test, probably the remote site changed (happened \
> with a few others from kblog as well), deactivate/expected fail IMHO.
> 
> > > > TestSuite.Compat-KOrganizer_3.1.ics
> > > > TestSuite.Compat-KOrganizer_3.2.ics
> 
> IIRC those were fixed with libical 1.0, right Allen?
> 
> 
> Looking at kdepim-runtime-stable:
> 
> > > > TestSuite.kolabconvertertest
> 
> Broken, wasn't adjusted to the kolab resource rewrite I think, should be
> deactivated/expected failed. Christian, can you look into fixing this one
> please?
> 
> 
> Looking at kdepim-stable:
> 
> > > > TestSuite.ktimezonecomboboxtest
> > > > TestSuite.summaryeventtest
> 
> those fail with ktimezoned error messages here, probably test setup \
> errors? Who could look into that?

That is quite an unstable test according to Jenkins - see the below link.
http://build.kde.org/job/kdepim_master/lastCompletedBuild/testReport/(root)/TestSuite/ktimezonecomboboxtest/history/?


> 
> > > > TestSuite.messageviewer-rendertest
> > > > TestSuite.messagecomposer-messagefactorytest
> > > > TestSuite.messagecomposer-maintextjobtest
> 
> those are in theory valid tests, but fail for technical reasons (crypto
> setup), non-trivial to fix correctly I think, so temporarily \
> disable/expected fail until someone has time to fix the test harness \
> there, I'd say.

messageviewer-rendertest seems to be failing due to changes in the
HTML code itself.
messagecomposer-maintextjobtest appears to be related to charset issues.
messagecomposer-messagefactorytest fails due to header changes from
what I can tell.

You can navigate through the Jenkins interface to get the logs from
the test executions if you wish.

> 
> > > My question to release-team and kdepim people:
> > > * Should we delay the release until tests are fixed?
> 
> IMHO that's not necessary.
> 
> > > * Should we simply QFAIL/remove the tests since it seems they are not
> > > that important?
> 
> see above, yes.
> 
> > > * Should we let them fail and still release?
> > > 
> > > Personally i'd like the tests either fixed or QFailed, this way for \
> > > 4.11.1 one can go to jenkins, check if everything is green and have a \
> > > very high level overview that with some luck maybe nothing regressed.
> 
> yes, makes sense
> 
> > I agree with you.  We need 0% failing.
> 
> Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for \
> this one already (and add the rest as soon as we hit 0% there for the \
> first time)?

Both Akonadi master and Akonadi stable have now been configured to
send emails to kde-pim@kde.org when tests begin to fail, or if the
build fails.

> 
> > 2 of the kdepimlibs fails are from kcalcore (the \
> > TestSuite.Compat.KOrganizer ones) I'm surprised at this.  Can someone \
> > tell me which version of libical is installed on the jenkins machines?
> 
> IIRC < 1.0, which should explain this.
> 
> > We should QFAIL the kblog test.  I can't remember the last time that \
> > was working.
> 
> They use online service and test accounts on these, that's not going to \
> keep working long-term without maintenance.
> 
> regards,
> Volker

Thanks,
Ben Cooksley
KDE Sysadmin

> 
> _______________________________________________
> KDE PIM mailing list kde-pim@kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/
_______________________________________________
release-team mailing list
release-team@kde.org
https://mail.kde.org/mailman/listinfo/release-team


[prev in list] [next in list] [prev in thread] [next in thread]