[prev in list] [next in list] [prev in thread] [next in thread] 

List:       wine-devel
Subject:    Re: A case for policy changes to get to zero test failures
From:       Alexandre Julliard <julliard () winehq ! org>
Date:       2023-10-31 11:36:16
Message-ID: 87bkcfkor3.fsf () wine
[Download RAW message or body]

Francois Gouget <fgouget@codeweavers.com> writes:

> * The other policy changes are still to be determined.

Thank you for raising the issue. I understand your frustration that
after all these years we still can't achieve zero failures, but I'm not
envisioning drastic policy changes at this point, particularly not extra
constraints on developers.

We all know that fixing tests can be hard even for the best developers,
so punishing people for not fixing tests, by reverting their changes or
blocking their MRs, would be counterproductive IMHO.

There's a balance to strike between the efforts needed to fix the tests,
and the efforts needed to develop new features and fix bugs in real
apps. A successful test suite is important, but it's not important
enough to sacrifice progress in other areas.

We also have to look at the positive side: we have millions of tests
that are being executed and succeed on every commit, and they are
catching many potential regressions. While the failures are annoying,
they are only a tiny percentage of the tests, and they don't make the
whole test suite useless. That's also why I don't want to disable an
entire test file when only a couple of tests fail. However, it's OK to
disable specific tests, using flaky() or todo(), if they are too hard to
fix properly.

So I think we have to accept that test failures and regressions are the
price to pay for making progress. We all need to do our best to minimize
them and keep them under control, but given the complexity of what we
are doing with Wine, we can't always achieve perfection. I'm also hoping
that, as the pool of paid developers grows, we will have more room to
allocate some paid developer time to fixing the tests.

-- 
Alexandre Julliard
julliard@winehq.org


[prev in list] [next in list] [prev in thread] [next in thread]