[prev in list] [next in list] [prev in thread] [next in thread] 

List:       webkit-dev
Subject:    Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit
From:       Maciej Stachowiak <mjs () apple ! com>
Date:       2017-11-17 16:23:13
Message-ID: AAA17A02-7426-42F1-9952-3F3E6BA8F46A () apple ! com
[Download RAW message or body]



> On Nov 17, 2017, at 7:53 AM, Frédéric WANG <fred.wang@free.fr> wrote:
> 
> On 17/11/2017 16:26, Maciej Stachowiak wrote:
> > WebKit has a lot of tests that were regression tests for a specific
> > bug fix, not as conformance tests (though they might b useful for that
> > too). In such cases, the association with the bugs.webkit.org bug is
> > more important than the spec URL. That's particularly the case when
> > the test has been changed multiple times to reflect further WebKit
> > behavior changes. I know that I've personally found it very useful to
> > look at revision history of tests, or search for bug numbers or
> > keywords in LayoutTests/ChangeLog to find related tests.
> > Not saying this is the sole purpose of tests, but losing this ability or making \
> > it more awkward would be a downside to deleting tests from the WebKit repo.
> Well, I think we agree here, that's why "conformance" tests was
> specified in my message. For such tests, connecting the history of tests
> with the development of the specs is actually more important. For
> example I recently noticed that some bugs with WOFF2 (on Apple's ports
> only) and WebVTT have been ignored in WebKit because we don't
> import/sync WPT tests for these specs.

I think importing new test suites is a different question than upstreaming/removing \
tests. In general importing more test suites is probably good, but we probably need \
to tackle the WPT performance problem before we pull in too many new WPT suites.

I should also note that pulling in the test suite won't automatically get the bugs \
fixed or prioritized, or even filed in the bug tracker \
necessarily.eratwerATWeRTWAerATWertwAerer

> > Maybe I'm missing something, but isn't that already how it works in WebKit? EWS \
> > and buildbot run all the tests, but on your own machine you can get \
> > run-webkit-tests to run any subset you want.
> Not everybody has the hardware to produce debug builds of WebKit on all
> existing ports and run a big amount of tests :-)
> 
> AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you
> don't have options to select a subset of the tests or the ports.
> Buildbots run all the builds & tests on all platforms *after* a patch
> lands in the main repo. So we do too much for WIP patches (extra runtime
> cost I'm mentioning here) and not enough to check the final patch before
> landing (which causes different issues like extra rollout or gardening
> commits or difficult regression bisecting).

I find it super convenient that EWS runs all the tests even on WIP patches. It often \
catches test failures in tests I didn't think to run myself. I think it would be \
great if EWS could run tests on more platforms. Just running release regression tests \
on more platforms would be a big win, if debug builds are the gating factor.

I guess it would be kind of neat to have a feature of "run the tests across all \
configurations, but only some of them". But I don't think I would prioritize it over \
having more kinds of tests, or finding ways to make the full test suite run faster.

> 
> -- 
> Frédéric Wang - frederic-wang.fr
> 
> 

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


[prev in list] [next in list] [prev in thread] [next in thread]