[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openembedded-core
Subject:    [OE-core] [PATCH 0/6] keep reducing regression reports noise
From:       Alexis Lothoré via lists.openembedded.org <alexis.lot
Date:       2023-02-28 18:10:44
Message-ID: 20230228181052.4191521-1-alexis.lothore () bootlin ! com
[Download RAW message or body]

From: Alexis Lothoré <alexis.lothore@bootlin.com>

Hello,
here is another batch of fixes to reduce noise in regression reports. Fixes are
directly linked to main noise sources seen in 4.2_M3 regression report ([1]).
- fix some existing selftests for resulttool
- add more filters for ptests incorrectly logging failures. The root cause of
  those wrongly named test results remained to be fixed to stop saving broken
  test results, but this series fixes at least parsing for existing results
- Stop logging "newly passing" tests in regression reports. Some real
  regressions are "hidden" in big chunks of newly passing tests:
foo: FAIL -> PASS
bar: FAIL -> PASS
moo: FAIL -> PASS
[...]
xxx: PASS -> FAIL
[...]
yyy: FAIL -> PASS
zzz: FAIL -> PASS

We are reaching a point where regression reports are small enough to get posted
on pastebin, so by following [2] you can find a report sample generated with
tooling patched with this series

[1] https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt
 [2] https://pastebin.com/QgfLKhpx

Alexis Lothoré (6):
  scripts/resulttool: call fixup_ptest_names in regression_common
  oeqa/selftest/resulttool: fix ptest filtering tests
  oeqa/selftest/resulttool: fix fake data used for testing
  scripts/resulttool: fix ptests results containing a non reproducible
    path
  oeqa/selftest/resulttool: add test for error propagation in test name
    filtering
  scripts/resulttool: do not count newly passing tests as regressions

 .../oeqa/selftest/cases/resulttooltests.py    | 112 +++++++++++-------
 scripts/lib/resulttool/regression.py          |  66 +++++++----
 2 files changed, 112 insertions(+), 66 deletions(-)

-- 
2.39.1



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#177847): https://lists.openembedded.org/g/openembedded-core/message/177847
Mute This Topic: https://lists.openembedded.org/mt/97296204/4454766
Group Owner: openembedded-core+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [openembedded-core@marc.info]
-=-=-=-=-=-=-=-=-=-=-=-



[prev in list] [next in list] [prev in thread] [next in thread]