[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit
From: Alexey Proskuryakov <ap () webkit ! org>
Date: 2017-11-28 18:13:34
Message-ID: 1E29B778-77E4-461C-8B48-EFFDB5FBABDC () webkit ! org
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> 17 нояб. 2017 г., в 23:02, youenn fablet <youennf@gmail.com> \
> написал(а):
> Tests are available at https://w3c-test.org <https://w3c-test.org/> which makes it \
> easy to share through any tool supporting hyperlinks. A webarchive can also be made \
> so that it is easy to share and probably edit such tests. Tools like jsfiddle are \
> also a great way to create/share/edit tests. I received several bug reports on \
> bugzilla using it and this proved to be efficient.
I don't think that these are applicable verbatim, but there may be some solution. \
Let's find it before assuming that we can have it.
In particular,
- editing a WebArchive is not really feasible;
- jsfiddle is cool, but I can't just copy a WPT test to jsfiddle to share it.
> Let me explain why I think that WebKit tests are often more valuable as regression \
> tests than WPT tests are. We add tests as we fix bugs, so we know that the tests \
> are generally for problems that have a high impact on users and developers - that's \
> because someone actually discovered the problem, and someone prioritized it highly \
> enough to fix. We also know that our tests cover code that is error prone, which is \
> why we had bugs in the first place. Of course, anything can be broken, but certain \
> things are less likely to. Compliance tests written for specs are also valuable, \
> but at some point we need to prioritize which tests to investigate and even to run. \
> I don't really see why we should prioritize the tests to run when all of them \
> provide clear value to some WebKit members.
We prioritize which tests to run all the time, there are whole configurations for \
which we don't run any tests. That's largely because it's not enough to simply run \
the tests - keeping them in a state where they produce actionable results requires a \
lot of attention.
The time it takes to run tests definitely matters a lot for WebKit. So far we haven't \
taken a route like the one Robert mentioned, with a huge amount of hardware running \
shards of tests. There are multiple reasons to that, one of those being that we are \
very interested in finding bugs below WebKit too, and having dedicated testers helps \
with that. When the machine gets into a weird state after a few weeks of uptime, it \
is much easier to start isolating the problem when we can see which specific queues \
are hitting it. It is also much easier to reason about data collected by centralized \
systems, such as crash log collection services.
- Alexey
> I agree that we need to prioritize tests we investigate. There can be a solution \
> inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing such \
> tests
> - WebKit contributors would prioritize WPT-WebKit-metadata failing tests
>
> That said, if these tests are so beneficial to WebKit, they are potentially very \
> useful to other teams as well. And vice-versa, we might find really good WPT tests \
> that show useful crashes and failures in WebKit. I am experiencing that nowadays \
> with WPT service worker tests.
[Attachment #5 (text/html)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" \
class=""><div class="">17 нояб. 2017 г., в 23:02, youenn fablet <<a \
href="mailto:youennf@gmail.com" class="">youennf@gmail.com</a>> \
написал(а):</div><br class="Apple-interchange-newline"><div class=""><div \
dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px;" class=""><div class=""><div \
class="gmail_quote"><div class="">Tests are available at<span \
class="Apple-converted-space"> </span><a href="https://w3c-test.org/" \
class="">https://w3c-test.org</a> which makes it easy to share through any tool \
supporting hyperlinks.</div><div class="">A webarchive can also be made so that it is \
easy to share and probably edit such tests.</div><div class="">Tools like jsfiddle \
are also a great way to create/share/edit tests.</div><div class="">I received \
several bug reports on bugzilla using it and this proved to be \
efficient.</div></div></div></div></div></blockquote><div><br class=""></div><div>I \
don't think that these are applicable verbatim, but there may be some solution. Let's \
find it before assuming that we can have it.</div><div><br class=""></div><div>In \
particular, </div><div>- editing a WebArchive is not really \
feasible;</div><div>- jsfiddle is cool, but I can't just copy a WPT test to jsfiddle \
to share it.</div><br class=""><blockquote type="cite" class=""><div class=""><div \
dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px;" class=""><div class=""><div \
class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; \
border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, \
204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: \
after-white-space;" class=""><div class=""></div><div class="">Let me explain why I \
think that WebKit tests are often more valuable as regression tests than WPT tests \
are. We add tests as we fix bugs, so we know that the tests are generally for \
problems that have a high impact on users and developers - that's because someone \
actually discovered the problem, and someone prioritized it highly enough to fix. We \
also know that our tests cover code that is error prone, which is why we had bugs in \
the first place. Of course, anything can be broken, but certain things are less \
likely to. Compliance tests written for specs are also valuable, but at some point we \
need to prioritize which tests to investigate and even to \
run.</div></div></blockquote><div class=""> </div><div class="">I don't really \
see why we should prioritize the tests to run when all of them provide clear value to \
some WebKit members.</div></div></div></div></div></blockquote><div><br \
class=""></div><div>We prioritize which tests to run all the time, there are whole \
configurations for which we don't run any tests. That's largely because it's not \
enough to simply run the tests - keeping them in a state where they produce \
actionable results requires a lot of attention.</div><div><br class=""></div><div>The \
time it takes to run tests definitely matters a lot for WebKit. So far we haven't \
taken a route like the one Robert mentioned, with a huge amount of hardware running \
shards of tests. There are multiple reasons to that, one of those being that we are \
very interested in finding bugs below WebKit too, and having dedicated testers helps \
with that. When the machine gets into a weird state after a few weeks of uptime, it \
is much easier to start isolating the problem when we can see which specific queues \
are hitting it. It is also much easier to reason about data collected by centralized \
systems, such as crash log collection services.</div><div><br class=""></div><div \
class=""> <div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; \
text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" \
class=""><div class="">- Alexey</div><div class=""><br class=""></div></div><br \
class="Apple-interchange-newline">
</div>
<blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: \
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; \
font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px;" class=""><div class=""><div class="gmail_quote"><div \
class="">I agree that we need to prioritize tests we investigate. There can be a \
solution inside WPT, like adding WebKit specific metadata so that:</div><div \
class="">- WPT contributors would communicate with WebKit members whenever changing \
such tests<br class=""></div><div class="">- WebKit contributors would prioritize \
WPT-WebKit-metadata failing tests</div><div class=""><br class=""></div><div \
class="">That said, if these tests are so beneficial to WebKit, they are potentially \
very useful to other teams as well.</div><div class="">And vice-versa, we might find \
really good WPT tests that show useful crashes and failures in WebKit.</div><div \
class="">I am experiencing that nowadays with WPT service worker \
tests.</div></div></div></div></div></blockquote></div><br class=""><div class=""> \
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div \
class=""><br class=""></div></div></div></body></html>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic