[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Filtering results on wpt.fyi, Safari-specific failures
From: Maciej Stachowiak <mjs () apple ! com>
Date: 2019-02-25 23:23:46
Message-ID: 2172E337-1218-4146-B8B7-ACC4FB755CD5 () apple ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> On Feb 25, 2019, at 1:57 PM, Philip Jägenstedt <foolip@chromium.org> wrote:
>
> I think I know what's going on there. When drilling down into tests and subtests, \
> only those matching the filter are shown. Clearing the filter things look a bit \
> different in the directories you mentioned: \
> https://wpt.fyi/results/ambient-light?label=master&label=experimental&product=chrome \
> %5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned \
> <https://wpt.fyi/results/ambient-light?label=master&label=experimental&product=chrom \
> e%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned>
> https://wpt.fyi/results/bluetooth?label=master&label=experimental&product=chrome%5B \
> taskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned \
> <https://wpt.fyi/results/bluetooth?label=master&label=experimental&product=chrome%5B \
> taskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned>
> In particular for idlharness.js tests some subtests will pass because they're \
> preconditions for the real tests.
OK.
> There will also be tests that check that something doesn't work, which will pass \
> even if the feature is entirely unsupported if "not working" results in the same \
> thing, e.g. throwing an exception. Sometimes tests can be tweaked to fail if the \
> feature is unsupported.
It would be helpful for clarity if "feature not supported at all" resulted in zero \
tests passing, but perhaps it is challenging to stick to writing tests that way.
>
> Drilling down into a directory somewhat at random and clearing filters, it does \
> look like this is legit: \
> https://wpt.fyi/results/fetch/api/cors?label=master&label=experimental&product=chrom \
> e%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned \
> <https://wpt.fyi/results/fetch/api/cors?label=master&label=experimental&product=chro \
> me%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned> \
>
There's definitely lots of failures that look legit.
>
> On Mon, Feb 25, 2019 at 8:31 PM Maciej Stachowiak <mjs@apple.com \
> <mailto:mjs@apple.com>> wrote:
> Neat.
>
> I see some obvious areas for focus, where Safari fails lots of tests that the other \
> browser don't.
> For context, I tried looking at this view, which shows all tests that Safari and \
> Firefox pass with Safari results regardless of result: \
> https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskcluste \
> r%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29 \
> <https://wpt.fyi/results/?label=master&label=experimental&product=chrome[taskcluster \
> ]&product=firefox[taskcluster]&product=safari[azure]&aligned&q=(chrome:pass%7Cchrome:ok)+(firefox:pass%7Cfirefox:ok)>
>
> I noticed some puzzling results there: Safari passes all the ambient-light and \
> bluetooth tests that Chrome and Firefox do, despite not supporting these standards \
> at all. (For that matter I'm not sure Firefox supports these specs either.) Not \
> sure if harness problem, or dubious tests that don't actually test the standard.
> Regards,
> Maciej
>
> > On Feb 25, 2019, at 5:48 AM, Philip Jägenstedt <foolip@chromium.org \
> > <mailto:foolip@chromium.org>> wrote:
> > I'd like to point out right away that diagnosing reftest failures is
> > currently cumbersome because we don't store the screenshots. This is
> > also a work in progress:
> > https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing \
> > <https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing>
> >
> > Until that has launched, I would recommend ignoring reftest failures
> > if the cause of failure isn't obvious.
> >
> > On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt <foolip@chromium.org \
> > <mailto:foolip@chromium.org>> wrote:
> > >
> > > Hi all,
> > >
> > > Following the improved Safari results last year [1] and the discussion
> > > that generated, I'm happy to announce that the filtering requested as
> > > now available in the search box. The full syntax is documented [2] but
> > > there's also a new insights view [3] with some useful searches.
> > >
> > > Especially interesting for this list could be this view, of Chrome
> > > Dev, Firefox Nightly and Safari Technology Preview, filtered to the
> > > Safari-specific failures:
> > > https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskcl \
> > > uster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%2 \
> > > 8chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29 \
> > > <https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskc \
> > > luster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=% \
> > > 28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29>
> > >
> > > Both Google and Mozilla have efforts [4][5] to reduce the number of
> > > Chrome/Firefox-specific failures, as this seems like a category of
> > > problems which especially valuable, where changing just one browser
> > > can remove a pain point for web developers.
> > >
> > > No doubt some failures are spurious, but hopefully there is value to
> > > be found by looking into where the largest numbers of failures appear
> > > to be. If something seems to be wrong with the search/filtering,
> > > please file an issue for us! [6]
> > >
> > > Credit to Mark Dittmer and Luke Bjerring who owned this project.
> > >
> > > P.S. We are also working on triage metadata for wpt.fyi, to make it
> > > possible to burn down a list of failures like this and not later have
> > > to re-triage to find the new failures. [7]
> > >
> > > [1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html \
> > > <https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html> [2] \
> > > https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md \
> > > <https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md> \
> > > [3] https://staging.wpt.fyi/insights <https://staging.wpt.fyi/insights> [4] \
> > > https://bugs.chromium.org/p/chromium/issues/detail?id=896242 \
> > > <https://bugs.chromium.org/p/chromium/issues/detail?id=896242> [5] \
> > > https://bugzilla.mozilla.org/show_bug.cgi?id=1498357 \
> > > <https://bugzilla.mozilla.org/show_bug.cgi?id=1498357> [6] \
> > > https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md \
> > > <https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md>
> > > [7] https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing \
> > > <https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing>
> > >
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
> > https://lists.webkit.org/mailman/listinfo/webkit-dev \
> > <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>
[Attachment #5 (text/html)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote \
type="cite" class=""><div class="">On Feb 25, 2019, at 1:57 PM, Philip Jägenstedt \
<<a href="mailto:foolip@chromium.org" class="">foolip@chromium.org</a>> \
wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" \
class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class="">I \
think I know what's going on there. When drilling down into tests and subtests, only \
those matching the filter are shown. Clearing the filter things look a bit different \
in the directories you mentioned:<div class=""><a \
href="https://wpt.fyi/results/ambient-light?label=master&label=experimental&pr \
oduct=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned" \
class="">https://wpt.fyi/results/ambient-light?label=master&label=experimental& \
;product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned</a><br \
class=""></div><div class=""><a \
href="https://wpt.fyi/results/bluetooth?label=master&label=experimental&produc \
t=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned" \
class="">https://wpt.fyi/results/bluetooth?label=master&label=experimental&pro \
duct=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned</a><br \
class=""></div><div class=""><br class=""></div><div class="">In particular for \
idlharness.js tests some subtests will pass because they're preconditions for the \
real tests. </div></div></div></div></div></div></blockquote><div><br \
class=""></div>OK.</div><div><br class=""><blockquote type="cite" class=""><div \
class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div \
dir="ltr" class=""><div class="">There will also be tests that check that something \
doesn't work, which will pass even if the feature is entirely unsupported if "not \
working" results in the same thing, e.g. throwing an exception. Sometimes tests can \
be tweaked to fail if the feature is \
unsupported.</div></div></div></div></div></div></blockquote><div><br \
class=""></div><div>It would be helpful for clarity if "feature not supported at all" \
resulted in zero tests passing, but perhaps it is challenging to stick to writing \
tests that way.</div><br class=""><blockquote type="cite" class=""><div class=""><div \
dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" \
class=""><div class=""><br class=""></div><div class="">Drilling down into a \
directory somewhat at random and clearing filters, it does look like this is \
legit:</div><div class=""><a \
href="https://wpt.fyi/results/fetch/api/cors?label=master&label=experimental&p \
roduct=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned" \
class="">https://wpt.fyi/results/fetch/api/cors?label=master&label=experimental&am \
p;product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=saf \
ari%5Bazure%5D&aligned</a> </div></div></div></div></div></div></blockquote><div><br \
class=""></div><div>There's definitely lots of failures that look legit.</div><br \
class=""><blockquote type="cite" class=""><div class=""><br class=""><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 25, 2019 at 8:31 PM \
Maciej Stachowiak <<a href="mailto:mjs@apple.com" class="">mjs@apple.com</a>> \
wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
style="overflow-wrap: break-word;" class=""><div class=""><br \
class=""></div>Neat.<div class=""><br class=""></div><div class="">I see some obvious \
areas for focus, where Safari fails lots of tests that the other browser \
don't. <br class=""><div class=""><br class=""></div><div class="">For context, \
I tried looking at this view, which shows all tests that Safari and Firefox pass with \
Safari results regardless of result:</div><div class=""><a \
href="https://wpt.fyi/results/?label=master&label=experimental&product=chrome[ \
taskcluster]&product=firefox[taskcluster]&product=safari[azure]&aligned&q=(chrome:pass%7Cchrome:ok)+(firefox:pass%7Cfirefox:ok)" \
target="_blank" class="">https://wpt.fyi/results/?label=master&label=experimental& \
amp;product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=s \
afari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29</a></div><div \
class=""><br class=""></div><div class="">I noticed some puzzling results there: \
Safari passes all the ambient-light and bluetooth tests that Chrome and Firefox do, \
despite not supporting these standards at all. (For that matter I'm not sure Firefox \
supports these specs either.) Not sure if harness problem, or dubious tests that \
don't actually test the standard.<br class=""><div class=""><br class=""></div><div \
class="">Regards,</div><div class="">Maciej</div><div class=""><br \
class=""><blockquote type="cite" class=""><div class="">On Feb 25, 2019, at 5:48 AM, \
Philip Jägenstedt <<a href="mailto:foolip@chromium.org" target="_blank" \
class="">foolip@chromium.org</a>> wrote:</div><br \
class="gmail-m_-2175485949127351989Apple-interchange-newline"><div class=""><div \
class="">I'd like to point out right away that diagnosing reftest failures is<br \
class="">currently cumbersome because we don't store the screenshots. This is<br \
class="">also a work in progress:<br class=""><a \
href="https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing" \
target="_blank" class="">https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing</a><br \
class=""><br class="">Until that has launched, I would recommend ignoring reftest \
failures<br class="">if the cause of failure isn't obvious.<br class=""><br \
class="">On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt <<a \
href="mailto:foolip@chromium.org" target="_blank" \
class="">foolip@chromium.org</a>> wrote:<br class=""><blockquote type="cite" \
class=""><br class="">Hi all,<br class=""><br class="">Following the improved Safari \
results last year [1] and the discussion<br class="">that generated, I'm happy to \
announce that the filtering requested as<br class="">now available in the search box. \
The full syntax is documented [2] but<br class="">there's also a new insights view \
[3] with some useful searches.<br class=""><br class="">Especially interesting for \
this list could be this view, of Chrome<br class="">Dev, Firefox Nightly and Safari \
Technology Preview, filtered to the<br class="">Safari-specific failures:<br \
class=""><a href="https://wpt.fyi/results/?label=master&label=experimental&pro \
duct=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5 \
Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29" \
target="_blank" class="">https://wpt.fyi/results/?label=master&label=experimental& \
amp;product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=s \
afari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29</a><br \
class=""><br class="">Both Google and Mozilla have efforts [4][5] to reduce the \
number of<br class="">Chrome/Firefox-specific failures, as this seems like a category \
of<br class="">problems which especially valuable, where changing just one browser<br \
class="">can remove a pain point for web developers.<br class=""><br class="">No \
doubt some failures are spurious, but hopefully there is value to<br class="">be \
found by looking into where the largest numbers of failures appear<br class="">to be. \
If something seems to be wrong with the search/filtering,<br class="">please file an \
issue for us! [6]<br class=""><br class="">Credit to Mark Dittmer and Luke Bjerring \
who owned this project.<br class=""><br class="">P.S. We are also working on triage \
metadata for wpt.fyi, to make it<br class="">possible to burn down a list of failures \
like this and not later have<br class="">to re-triage to find the new failures. \
[7]<br class=""><br class="">[1] <a \
href="https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html" \
target="_blank" class="">https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html</a><br \
class="">[2] <a href="https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md" \
target="_blank" class="">https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md</a><br \
class="">[3] <a href="https://staging.wpt.fyi/insights" target="_blank" \
class="">https://staging.wpt.fyi/insights</a><br class="">[4] <a \
href="https://bugs.chromium.org/p/chromium/issues/detail?id=896242" target="_blank" \
class="">https://bugs.chromium.org/p/chromium/issues/detail?id=896242</a><br \
class="">[5] <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1498357" \
target="_blank" class="">https://bugzilla.mozilla.org/show_bug.cgi?id=1498357</a><br \
class="">[6] <a href="https://github.com/web-platform-tests/wpt.fyi/issues/new?title=S \
tructured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md" \
target="_blank" class="">https://github.com/web-platform-tests/wpt.fyi/issues/new?titl \
e=Structured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md</a><br \
class="">[7] <a href="https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing" \
target="_blank" class="">https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing</a><br \
class=""></blockquote>_______________________________________________<br \
class="">webkit-dev mailing list<br class=""><a \
href="mailto:webkit-dev@lists.webkit.org" target="_blank" \
class="">webkit-dev@lists.webkit.org</a><br class=""><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank" \
class="">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br \
class=""></div></div></blockquote></div><br \
class=""></div></div></div></blockquote></div> </div></blockquote></div><br \
class=""></body></html>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic