[prev in list] [next in list] [prev in thread] [next in thread] 

List:       wine-devel
Subject:    New failures analysis
From:       Francois Gouget <fgouget () codeweavers ! com>
Date:       2023-10-20 23:35:55
Message-ID: 770d94af-52a9-3eb0-3b35-2641ca57cc8a () codeweavers ! com
[Download RAW message or body]

New failures:
 Where do they come from?
 How do they get in?
 All you've ever wanted to know (or not).

So I collected data about the new failure modes (see failure-new.txt), 
mostly from the start of July (older data is incomplete).

By new failure mode I mean any failure group that I can say started 
happening on a specific day in that range. This calls for some caveats:

* A failure mode is a group of failures that happens together in a 
  given test unit and thus likely all have the same origin. Normally 
  each failure mode is described in a bug.

  In the rest of this document I'll just say 'failure' in place of 
  'failure mode'.

* This includes failures that happen on non GitLab / TestBot machines.

* I don't know the origin of all of them so there are certainly some 
  that are not caused by a change in Wine. So not all are Wine 
  regressions.

* But until their cause is known they still need to be investigated so
  I'm still counting them when it seems appropriate. However I did not 
  include those where it was known from the start that they were caused 
  by external factors.


First the date range I used for each month:
  Jul 2023-06-30 - 2023-07-28   28 days
  Aug 2023-07-28 - 2023-08-28   31 days
      (08-28 is about when the TestBot -> GitLab CI bridge broke)
  Sep 2023-09-01 - 2023-10-06   39 days


So how many new failures are there?
  Jul  36 new failures -> 1.3 / day
  Aug   7 new failures -> 0.2 / day (vacation effect!)
  Sep  53 new failures -> 1.4 / day

  Note that a single MR may cause multiple tests to fail but each new 
  failing test would be counted separately here on the premise that they 
  each need to be investigated and reported.


Where do they come from?

                Jul    Aug    Sep
  Unknown         8      4     18
  Commit          4      1      6
  MR              7      2     17
  Total          19      7     41
  Total / day   0.7    0.2    1.1

  * These counts are deduplicated except for the new failures of 
    unknown origin obviously.

  * The commits correspond to direct Wine commits so they don't go 
    through the GitLab CI or the TestBot.

  * About half of the new failures have no known origin. This can be 
    because they happen on machines I don't have access to and thus 
    where I cannot bisect. It can also be because the failure fails to 
    reproduce which is annoyingly common.

  * To keep up 5 to 8 failure would have to be fixed


How do they get in?

                  Jul    Aug    Sep
  Untested          2      1      7
  Bad               2      0      0
  Module            1      0      1
  Extra configs     0      0      2
  Flaky             2      0      2
  WineTest only     1      0      0
  Bridge            0      0      7
  Misc              1      1      0
  Total             9      2     19

  Here's a description of the way the failures got in. Those correspond 
  to deduplicated failure counts since we're not interested in the 
  number of tests impacted but in how the commits got in:

  o 33% Untested
    These correspond to direct Wine commits. As such they bypass the 
    GitLab CI and TestBot so there was no way to detect them beforehand.
    There has been a lot of changes in the lowe level code lately so I 
    think this is usually lower.

  o 23% Bridge
    These failures were detected by the TestBot but not forwarded 
    to the MR page because the mailing list to GitLab bridge broke on or 
    about 2023-08-28.

  o 13% Flaky
    These failures are random and just did not happen when the MR got 
    tested by the GitLab CI and TestBot. One way to minimize the chances 
    of this happening is to run the tests multiple times.

    But the TestBot already runs the tests on up to 24 Windows 
    configurations and 10 Linux ones which means random failures that 
    are not configuration-specific have a very good chance of being 
    caught. It's the other ones that get through, like those that only 
    impact a specific locale or a specific Windows version.

  o 7% Module
    When a patch modifies a module but not the tests it contains the 
    TestBot skips some test configurations which can lead to some 
    failures being undetected.

  o 7% Extra
    This corresponds to failures specific to test configurations that 
    the TestBot only uses for the nightly WineTest runs.

  In the above counts I omitted the following:
  o 18 Unknown
    This corresponds to cases where what is causing the failure has not 
    been identified, and thus how they got in undetected is not known. 
    Note that there is one exception: if the failure does not impact the 
    GitLab and TestBot machines then the reason is automatically "Non 
    TestBot".

  o 17 Non TestBot
    These new failures impact neither the GitLab nor the TestBot 
    machines. This corresponds to macOS-specific failures for instance. 
    Detecting them beforehand is therefore impossible... except by 
    expanding the set of test machines.


Are they fixed yet?

                Jul       Aug        Sep
  Fixed       15 (47%)   2 (33%)   21 (40%)
  Not fixed   17 (53%)   4 (67%)   31 (60%)

  Not there yet.

  * Take with a big grain of salt : the bugs are not always closed 
    when a failure is fixed. I may also have forgotten to mark some 
    entries as fixed in failures-new.txt but I think that part is mostly 
    okay now.

  * Older failures don't seem more likely to be fixed. But then failures 
    are not necessarily reported in the order they appear.

-- 
Francois Gouget <fgouget@codeweavers.com>
["failures-new.txt" (text/plain)]

Test units most frequently getting new failures:
grep -v -P '^(#|  +[0-9])' failures-new.txt |
 sed -n '/week 2023-10-16$/,/week 2023-08-28$/p' |
 grep -E -v '[-]> fixed' | sed -e 's/[^a-z0-9:._]/\n/g' | grep -E '[a-z0-9]:[a-z0-9]' \
|  sort | uniq -c | sort -n -r

# or to filter on days instead of weeks
 sed -n '/^2023-10-13/,/^2023-10-02/p' |
# or to exclude fixed issues
 sed -e 's/[a-z][a-z0-9:._]*:[a-z][a-z0-9:._]* -> fixed.*//g' |
# or to deduplicate origins that cause multiple bugs
 sed -e 's/^\(.. [^ ]*\) .*/\1/' -e t -e d | sort -u |
# or to filter out failures caused by GitLab's switch to Debian 12
 grep -E -v 5799e89f15b5 |
# or to keep only the data lines
 grep -E '^.. ' | sort |

# Origin counts
# <...> main grep, time period and other filters go here
 cat >/tmp/r$$.txt; for t in r c m "."; do echo $t $(grep -E -i "^$t. " /tmp/r$$.txt \
| wc -l); done; rm /tmp/r$$.txt # or reason counts
 cat >/tmp/r$$.txt; for t in "[.]" b c m e o f g h i l w r t "[?]" "."; do echo $t \
$(grep -E -i "^.$t " /tmp/r$$.txt | wc -l); done; rm /tmp/r$$.txt

# All bug numbers
 sed -e 's/\(bug [0-9][0-9]*\)/\n\1\n/g' | sed -e 's/^bug //' -e t -e d | sort -u | \
wc -l # or to filter out fixed bugs prefix with
 grep -E -v 'fixed bug' |

For importing in spreadsheets:
(p=2099-12-31; for d in `sed -e 's/ ---.*//' failures-new.txt | grep -P \
'^[0-9]{4}-[0-9]{2}-[0-9]{2}$'` ; do  echo -n "$p,"; for t in "[.]" b c m e o f g h i \
l w r t "[?]" "."; do count=`sed -n "/^$p/,/^$d/p" failures-new.txt | grep -E -i \
"^.$t " | wc -l`; echo -n "$count,"; done; p=$d; echo; done) | tac


Jul
 grep -v -P '^(#|  +[0-9])' failures-new.txt | sed -n '/^2023-07-28/,/^2023-06-30/p' \
| Aug
 grep -v -P '^(#|  +[0-9])' failures-new.txt | sed -n '/^2023-08-28/,/^2023-07-28/p' \
| Sep
 grep -v -P '^(#|  +[0-9])' failures-new.txt | sed -n '/^2023-10-06/,/^2023-08-28/p' \
|


# One character classification codes:
#
# ORIGINS : Where does the new failure come from?
#
# r Unexplained new failure
#   The reason why this new failure appeared is not known. It could be a
#   change in Wine, making it a regression, or some unidentified environment
#   change or glitch. Even in that case it could mean the test is buggy in
#   that it is not robust enough.
# c Commit causing a regression
#   The new failure is caused by a commit which went neither through the
#   TestBot nor through the GitLab CI.
# m Merge request causing a regression
#   The new failure is caused by the specified merge request.
#
# REASONS : Why did the TestBot not detect it?
#
# . Untested
#   No tests were performed prior to the regression being introduced in Wine
#   (i.e. this can only be paired with m or c origins).
#   This is typically the case for commits that don't go through GitLab but it
#   can also happen if an MR is modified during the merge.
# b Bad merge
#   The failures were detected and reported but the merge request was merged
#   anyway.
# c GitLab CI only
#   Happens in the GitLab CI but not on the TestBot VMs.
# m Module patch
#   The patch does not touch the tests, causing the TestBot to skip running
#   the tests in some configurations, most notably Windows, and thus not
#   detect the regression.
# e Extra configuration
#   The failure could not be detected by the TestBot because it only happens
#   in one of the non-base TestBot configurations.
# o Other configuration
#   The failure only happens in non-TestBot / GitLab test configurations such
#   as running the 32-bit tests in the same wineprefix after the 64-bit tests.
# f Flaky
#   The failure did not happen when the TestBot tested the merge request
#   despite the relevant configuration being tested and the failure happening
#   outside WineTest runs, presumably because it happens randomly and rarely
#   enough.
# g Git skew
#   The patch was meant to be apply on a top newer Git revision but the
#   TestBot received it before it noticed that Git revision was available,
#   leading to a bad patch error.
# h Hidden
#   Some other failure or issue hid the new failure. For instance this can
#   happen when network trouble prevents some tests from running correctly or
#   if a patch fails to apply or to compile. This can also happen when two MRs
#   succeed separately but interfere and cause failures when combined.
# i Minor
#   The new failure is minor and was ignored by the TestBot. This is typically
#   the case of commits that introduce dependencies on missing APIs.
# l Linux test suite
#   The failure only happen when running the full test suite in Wine.
# w Windows test suite
#   The failure only happen when running the full test suite on Windows.
# r Report bug
#   The failure was detected by the TestBot but not reported to GitLab. This
#   should only be used after having ferified that the TestBot did detect the
#   failure.
# t TestBot bug
#   The failure was not detected because of a TestBot bug: it did not detect
#   the failure, thought it was a preexisting one, or it did not run the right
#   tests.
# ? Status to be determined


DONE: 55128 (2023-06-24) -> 55807 (2023-10-18)
TODO: Bugs before 54659 (2023-03-14)


2023-10-20 --- week 2023-10-16

2023-10-13 --- week 2023-10-09

2023-10-10
me 4024         uiautomationcore:uiautomation -> bug 55807
mr 3959         uxtheme:system -> bug 55771
mr 3959         uxtheme:system -> bug 55772

2023-10-09
ro unknown      psapi:psapi_main -> bug 55803 (rb-wayland)

2023-10-06 --- week 2023-10-02

c. de81e2ea41b2 dbghelp:dbghelp -> bug 55747
c. de81e2ea41b2 ntdll:loader -> bug 55746
c. de81e2ea41b2 ntdll:wow64 -> bug 55755
c. de81e2ea41b2 psapi:psapi_main -> bug 55744
c. de81e2ea41b2 tasklist.exe:tasklist -> bug 55745
r? unknown      wbemprox:query -> bug 55751 (rb-macos)

2023-10-05
me 4009         advapi32:eventlog -> bug 55731
c. 4847c1d8e474 kernel32:loader -> fixed bug 55732
me 4020         ntdll:file -> bug 55733

2023-10-03
r? unknown      amstream:amstream -> bug 55725
r? unknown      d2d1:d2d1 -> bug 55765
mc 4004         msxml3:domdoc -> bug 55720

2023-10-02
r? unknown      mfmediaengine:mfmediaengine -> bug 55724

2023-09-29 --- week 2023-09-25
mo 3953         dmime:dmime -> bug 55722 (rb-macos)
ro unknown      rpcrt4:rpc -> bug 55689
ro unknown      winhttp:winhttp -> bug 55690

2023-09-28
c. a81c53504ae3 msvfw32:mciwnd -> fixed bug 55696
r? unknown      ntdll:info -> bug 55721

2023-09-27
ro unknown      user32:win -> bug 55767 (rb-wayland)

2023-09-26
ro unknown      kernel32:file -> bug 55668 (rb-macos)
ro unknown      kernel32:file -> bug 55568 (rb-macos)

2023-09-25
mr 3930         d3d10:effect -> fixed bug 55651
r? unknown      dmime:dmime -> bug 55687
c. 0abd795ff026 kernel32:file -> bug 55667 (rb-macos)

2023-09-22 --- week 2023-09-18

2023-09-19
mc 3882         msvcr120:msvcr120 -> fixed bug 55686
mr 3836         ntdll:file -> fixed bug 55631
mr 3836         ntdll:file -> fixed bug 55632

2023-09-18
ro unknown      dbghelp:dbghelp -> fixed bug 55650 (rb-macos)
mr 3867         mfreadwrite:mfplat -> fixed bug 55635

2023-09-15 --- week 2023-09-11

2023-09-14
ro unknown      ntdll:wow64 -> fixed bug 55648
ro unknown      psapi:psapi_main -> fixed bug 55658 (rb-macos)
mr 3832         user32:dialog -> fixed bug 55656
mr 3832         user32:dialog -> bug 55655

2023-09-12
mr 3792         quartz:vmr9 -> fixed bug 55587
mr 3792         quartz:vmr9 -> bug 55649

2023-09-11
mf 3751         dmime:dmime -> bug 55637
ro unknown      ntdll:exception -> bug 55657 (rb-macos)

2023-09-08 -- week 2023-09-04
mr 3746         d3d8:device -> bug 55583

2023-09-07
c. 425bd55b02e3 dbghelp:dbghelp -> fixed bug 55560

2023-09-06
mf 3623         uiautomationcore:uiautomation -> fixed bug 55580

2023-09-05
m. 3643         combase:roapi -> fixed bug 55531 (AJ merge)
m. 3643         kernel32:actctx -> fixed bug 55532 (AJ merge)
m. 3643         setupapi:devinst -> fixed bug 55533 (AJ merge)
m. 3643         setupapi:install -> fixed bug 55533 (AJ merge)
m. 3643         ucrtbase:thread -> fixed bug 55534 (AJ merge)
mt 3548         windows.applicationmodel:model -> fixed bug 55652 (missing makedep \
testdll support)

2023-09-04
r? unknown      advapi32:registry -> fixed bug 55577
c. d64200805b18 kernel32:loader -> bug 55578

2023-09-01 -- week 2023-08-28
c. d64200805b18 kernel32:loader -> bug 55579

2023-08-31
mr 3691         dinput:hid -> fixed bug 55535

2023-08-30
mm 3681         mshtml:dom -> bug 55569
ro unknown      quartz:mpegaudio -> bug 55552 (rb-macos)
ro unknown      quartz:mpeglayer3 -> bug 55553 (rb-macos)

2023-08-28
c. f6dbcccab9e3 kernel32:virtual -> bug 55561

2023-08-25 --- week 2023-08-21

2023-08-23
m? 3439         mf:mf -> bug 55588

2023-08-18 --- week 2023-08-14

2023-08-11 --- week 2023-08-07

2023-08-08
r? unknown      quartz:filtergraph -> bug 55758
r? unknown      user32:msg -> bug 55464
mh 3510         mfmediaengine:mfmediaengine -> fixed bug 55422 (MR!3351 interference)

2023-08-04 --- week 2023-07-31
ro unknown      secur32:negotiate -> fixed bug 55627
ro unknown      secur32:ntlm -> fixed bug 55627

2023-07-28 -- week 2023-07-24

2023-07-27
mf 3389         build:mac -> fixed bug 55366, -lpcsclite not found (configure bug)
mm 3401         comctl32:listbox -> bug 55394
mg 3398         gdiplus:graphicspath -> fixed bug 55351

2023-07-21 -- week 2023-07-17

2023-07-20
r? unknown      ddraw:ddraw7 -> bug 55349

2023-07-19
r? unknown      ntdll:exception -> bug 55480

2023-07-18
r? unknown      winmm:mci -> bug 55605

2023-07-17
mo 3336         advapi32:registry -> bug 55288 (32-after-64)
mo 3336         ntdll:reg -> bug 55288 (32-after-64)

2023-07-14 --- week 2023-07-10
r? unknown      user32:msg -> bug 55786
r? unknown      d3dx10_39:d3dx10 -> fixed bug 55223
r? unknown      d3dx10_42:d3dx10 -> fixed bug 55223

2023-07-13
cf 5799e89f15b5 itss:protocol -> bug 55460 (deb12)

2023-07-12
mb 3050         mshtml:misc -> fixed bug 55244 (bad for mshtml:dom crash)

2023-07-11
r? unknown      psapi:psapi_main -> bug 55416
c. 7e521b4db353 ntdll:info -> bug 55269
r? unknown      vbscript:createobj -> bug 55330

2023-07-10
cw 5799e89f15b5 amstream:amstream -> fixed bug 55283 (deb12)
cw 5799e89f15b5 gdi32:font -> bug 55267 (deb12)
cw 5799e89f15b5 kernel32:loader -> bug 55228 (deb12)
cw 5799e89f15b5 kernel32:loader -> bug 55229 (deb12)
cw 5799e89f15b5 kernel32:loader -> bug 55266 (deb12)
cw 5799e89f15b5 kernel32:process -> bug 55226 (deb12)
cw 5799e89f15b5 kernel32:thread -> fixed bug 55225 (deb12)
cw 5799e89f15b5 ntdll:om -> fixed bug 55245 (deb12)
cw 5799e89f15b5 ntdll:om -> fixed bug 55246 (deb12)
cw 5799e89f15b5 ntdll:wow64 -> fixed bug 55273 (deb12)
cw 5799e89f15b5 oleaut32:usrmarshal -> fixed bug 55237 (deb12)
cw 5799e89f15b5 scrrun:dictionary -> fixed bug 55236 (deb12)
cw 5799e89f15b5 winmm:capture -> fixed bug 55240 (deb12)
cw 5799e89f15b5 winmm:capture -> fixed bug 55241 (deb12)
cw 5799e89f15b5 winmm:wave -> fixed bug 55240 (deb12)
cw 5799e89f15b5 winmm:wave -> fixed bug 55241 (deb12)
cw 5799e89f15b5 wtsapi32:wtsapi -> fixed bug 55227 (deb12)
mo 3274         gdi32:dc -> bug 55291 (fg-deb64)

2023-07-07 -- week 2023-07-03

2023-07-05
mb 3049         user32:text -> bug 55215

2023-07-04
c. 36963b6d27ab oleaut32:vartest -> fixed bug 55194

2023-06-30 -- week 2023-06-26

2023-06-28
r? unknown      d3dx9_36:effect -> bug 55478

2023-06-27
me 3122         ntdll:file -> fixed bug 55331 (1607 & 1709)

2023-06-23 --- week 2023-06-19
mi 3129         dbghelp:path -> fixed bug 55128 (entry point)

2023-06-22
mf 3141         secur32:schannel -> bug 55114 failure seemed systematic though
r? unknown      user32:msg -> bug 55374
r? unknown      user32:msg -> fixed bug 55354

2023-06-21
c. 2ef4cc48fbce dbghelp:dbghelp -> bug 55109
c. 2ef4cc48fbce kernel32:debugger -> bug 55120
r? unknown      mmdevapi:render -> bug 55727
r? unknown      ntdll:exception -> bug 55111

2023-06-19
mt 3080         xaudio2_8:xaudio2 -> bug 55108 (xaudio2_7)

2023-06-16 --- week 2023-06-12
r? unknown      psapi:psapi_main -> bug 55429

2023-06-15
r?              user32:win -> bug 55778

2023-06-14
r? unknown      d3dx10_34:d3dx10 -> fixed bug 55223
r? unknown      d3dx10_35:d3dx10 -> fixed bug 55223

2023-06-13
mf 3055         quartz:vmr7 -> bug 55110
mf 3055         quartz:vmr9 -> bug 55110

2023-06-12
c. cc2cfb9b792b dbghelp:dbghelp -> bug 55058
r? unknown      user32:msg -> bug 55465
r? unknown      user32:msg -> bug 55463

2023-06-09 --- week 2023-06-05
r? unknown      ntdll:threadpool -> bug 55493

2023-06-07
mt 2918         xaudio2_8:xaudio2 -> bug 55016 (xaudio2_7)

2023-06-05
r? unknown      user32:cursoricon -> bug 55424

2023-06-02 --- week 2023-05-29
c. b337c5b1a175 kernel32:comm -> bug 55014
mb 2950         user32:monitor -> bug 55013

2023-06-01
r? unknown      shell32:systray -> bug 55017

2023-05-31
mf 2901         mf:transform -> bug 55638

2023-05-26 --- week 2023-05-22

2023-05-19 --- week 2023-05-15
r? unknown      user32:msg -> bug 55066

2023-05-15
r? unknown      win32u:win32u -> bug 54975

2023-05-12 --- week 2023-05-08

2023-05-05 --- week 2023-05-01
2023-05-04
r? unknown      imm32:imm32 -> bug 55491

2023-05-01
me 2678         user32:monitor -> bug 55065 (debiant)

2023-04-28 --- week 2023-04-24

2023-04-21 --- week 2023-04-17

2023-04-17
me 2659         imm32:imm32 -> bug 54864 (Korean locale)

2023-04-14 --- week 2023-04-10
2023-04-13
mb 2647         dnsapi:query -> bug 54847
mb 2647         dnsapi:query -> bug 54848

2023-04-07 --- week 2023-04-03

2023-03-31 --- week 2023-03-27

2023-03-29
mw 2531         imm32:imm32 -> bug 54760

2023-03-24 --- week 2023-03-20

2023-03-22
mo 2445         advapi32:registry -> bug 54742 (32-after-64)
mo 2445         msi:action -> bug 54738 (64-bit)
mo 2445         ole32:compobj -> bug 54751 (only in 64-bit full suite)
mo 2445         oleaut32:typelib -> bug 54739 (64-bit)

2023-03-17 --- week 2023-03-13
r? unknown      adsldp:ldap -> bug 54707 (ldap server outage)

2023-03-16
mb 2421         dinput:device8 -> bug 54713
me 2419         imm32:imm32 -> bug 54710 (Korean locale)
me 2419         imm32:imm32 -> bug 54711 (Korean locale)

2023-03-15
mo 1445         reg.exe:copy -> bug 54691 (64-bit)
mo 1445         reg.exe:delete -> bug 54691 (64-bit)
mo 1445         reg.exe:export -> bug 54691 (64-bit)
mo 1445         reg.exe:import -> bug 54691 (64-bit)
mo 1445         reg.exe:query -> bug 54691 (64-bit)
mo 1445         reg.exe:add -> bug 54691 (32-after-64)

2023-03-14
c. c44e58c88bd1 ntdll:info -> bug 54688

2023-03-13
r? unknown      shlwapi:ordinal -> bug 54683

2023-03-10 --- week 2023-03-06

2023-03-09
me 2354         imm32:imm32 -> bug 54669 (Korean & Mixed locales)
mb 2361         kernel32:loader -> bug 54657

2023-03-08
mf 2347         dbghelp:dbghelp -> bug 54717 (rare)
mb 2252         psapi:psapi_main -> bug 55010
mw 2234         user32:input -> bug 55327

2023-03-06

2023-03-03 --- week 2023-02-27
r? unknown      dinput:device8 -> bug 54774

2023-03-02
r? unknown      shell32:progman_dde -> bug 54758
mf 2313         shlwapi:ordinal -> bug 54683 (summertime-specific)

2023-02-28
r? unknown      comdlg32:filedlg -> bug 55153

2023-02-27

2023-02-24 --- week 2023-02-20
2023-02-23
r? unknown      d3d11:d3d11 -> bug 54757

2023-02-22
mo 2250         winscard:winscard -> bug 54661 (missing build dep)

2023-02-17 --- week 2023-02-13

2023-02-10 --- week 2023-02-06
mb 2154         user32:msg -> fixed bug 54477

2023-02-09
r? unknown      httpapi:httpapi -> bug 55127

2023-02-03 --- week 2023-01-30

2023-01-27 --- week 2023-01-23

2023-01-20 --- week 2023-01-16

2023-01-13 --- week 2023-01-09

2023-01-06 --- week 2023-01-02

2022-12-23 --- week 2022-12-19

2022-12-22
mf 1874         user32:msg -> fixed, no bug

2022-12-16 --- week 2022-12-12

2022-12-09 --- week 2022-12-02

2022-12-07
r? unknown      explorerframe:nstc -> bug 54791
mb 1732         taskschd:scheduler -> fixed bug 54105

2022-12-02 --- week 2022-11-28

2022-11-26 --- week 2022-11-21

2022-11-22
mb 1495         vbscript:run -> fixed since

2022-11-18 --- week 2022-11-14

2022-11-17
mb 1399         all:tests -> bug 53963
mb 1398         ntoskrnl.exe:ntoskrnl -> bug 53976

2022-11-11 -- week 2022-11-07





[prev in list] [next in list] [prev in thread] [next in thread]