[prev in list] [next in list] [prev in thread] [next in thread]
List: webkit-dev
Subject: Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!
From: Stephen Mcgruer <smcgruer () chromium ! org>
Date: 2019-12-03 2:33:38
Message-ID: CADY3MaeerzsTntNxLPfN2tx9NZSuTTcxS7qrFJc9z+fJ=82sjA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
>
> There's a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn
We have unfortunately struggled historically with getting reliable +
consistent results for Safari, with spurious infrastructure + other
problems. That said, Igalia has been working on improving reliability and
have made good progress, and we have also invested in projects like the
WebkitGTK runs to help with this problem (e.g. if a test fails in both
Safari and WebKitGTK, there's a better chance it's a real failure[0])
We have had many Safari TP 96 runs since my announcement (
https://wpt.fyi/runs?label=master&max-count=100&product=safari); you can
click on a single run (the icon) to view the run itself, or click on two
runs to get a diff view - which can be useful for spotting flakes (as most
SHAs are only changing a few tests, so differences between consecutive runs
are usually flakes). There is also an explicit flake analyzer on our
insights page - https://wpt.fyi/insights (it's a little clunky but can
still be useful).
I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
Improvements would be great! Looking at that PR, it is quite old, so will
need to be rebased and force pushed. Let me know if you need any help on
that :).
STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.
It does look like there are still some harness failures in further STP 96
runs (e.g. https://wpt.fyi/results/webauthn?run_id=361070008); I see a
mention in the failing tests of a "UnexpectedAlertOpenException: unexpected
alert open (500):" - is that what you mean? Is there some flag we could be
setting on STP, or something Safaridriver could be doing to avoid these?
[0]:
https://wpt.fyi/results/?label=master&product=safari%5Bexperimental%5D&product=webkitg \
tk&product=chrome%5Bexperimental%5D&product=firefox%5Bexperimental%5D&aligned&q=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Apass
is
an example search for tests that fail in both Safari + WebKitGTK, but pass
in both Chrome and Firefox.
On Mon, 2 Dec 2019 at 21:04, Jiewen Tan <jiewen_tan@apple.com> wrote:
> Hi Maciej,
>
> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak <mjs@apple.com> wrote:
>
>
> There's a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
>
> The new WebCrypto failures are surprising, but likely real and should be
> investigated:
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013
> <https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013>
>
>
> I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
> https://github.com/web-platform-tests/wpt/pull/6102
>
>
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn/
>
>
> STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.
>
> Best,
> Jiewen
>
>
>
> On Nov 27, 2019, at 7:07 AM, Stephen Mcgruer <smcgruer@chromium.org>
> wrote:
>
> Excited to announce that Safari Tech Preview 96 is now available on
> wpt.fyi!
>
> Example run:
> https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned
>
> Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but
> very close):
> https://wpt.fyi/results/?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013
>
> If you are surprised by the results in the diff view, I would love to hear
> about it. One thing that surprised me is that the release blog post[0]
> mentioned Web Animations being enabled by default, but we see no
> differences in the test diff for web-animations/. It's possible we enable
> some flag that turned on Web Animations already, but I can't see it
> obviously in our safari setup[1].
>
> [0]:
> https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/
> [1]:
> https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">There's a number of \
mysterious timeouts with 96. Not sure if flakiness or real?<br>Many large new chunks \
of not-run tests are caused by a harness error or timeout. E.g. html/ and \
webauthn</blockquote><div><br></div><div>We have unfortunately struggled historically \
with getting reliable + consistent results for Safari, with spurious infrastructure \
+ other problems. That said, Igalia has been working on improving reliability and \
have made good progress, and we have also invested in projects like the WebkitGTK \
runs to help with this problem (e.g. if a test fails in both Safari and WebKitGTK, \
there's a better chance it's a real failure[0])</div><div><br></div><div>We \
have had many Safari TP 96 runs since my announcement (<a \
href="https://wpt.fyi/runs?label=master&max-count=100&product=safari">https://wpt.fyi/runs?label=master&max-count=100&product=safari</a>); \
you can click on a single run (the icon) to view the run itself, or click on two runs \
to get a diff view - which can be useful for spotting flakes (as most SHAs are only \
changing a few tests, so differences between consecutive runs are usually flakes). \
There is also an explicit flake analyzer on our insights page - <a \
href="https://wpt.fyi/insights">https://wpt.fyi/insights</a> (it's a little \
clunky but can still be useful).</div><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">I believe these \
tests are flaky. I have made a PR to improve it a while ago. I should probably get \
those improvement landed \
sometime.</span></blockquote><div><br></div><div>Improvements would be great! Looking \
at that PR, it is quite old, so will need to be rebased and force pushed. Let me know \
if you need any help on that :).</div><div><br></div><div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">STP 96 has enabled user agent UI for WebAuthn, \
which could cause crashes. This could be the reason why webauthn is 0 all the \
time.</blockquote><div><br></div></div><div>It does look like there are still some \
harness failures in further STP 96 runs (e.g. <a \
href="https://wpt.fyi/results/webauthn?run_id=361070008">https://wpt.fyi/results/webauthn?run_id=361070008</a>); \
I see a mention in the failing tests of a "UnexpectedAlertOpenException: \
unexpected alert open (500):" - is that what you mean? Is there some flag we \
could be setting on STP, or something Safaridriver could be doing to avoid \
these?</div><div><br></div><div>[0]: <a \
href="https://wpt.fyi/results/?label=master&product=safari%5Bexperimental%5D&p \
roduct=webkitgtk&product=chrome%5Bexperimental%5D&product=firefox%5Bexperiment \
al%5D&aligned&q=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Ap \
ass">https://wpt.fyi/results/?label=master&product=safari%5Bexperimental%5D&pr \
oduct=webkitgtk&product=chrome%5Bexperimental%5D&product=firefox%5Bexperimenta \
l%5D&aligned&q=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Apass</a> \
is an example search for tests that fail in both Safari + WebKitGTK, but pass in \
both Chrome and Firefox. </div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, 2 Dec 2019 at 21:04, Jiewen Tan <<a \
href="mailto:jiewen_tan@apple.com">jiewen_tan@apple.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
style="overflow-wrap: break-word;">Hi Maciej,<br><div><br><blockquote \
type="cite"><div>On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak <<a \
href="mailto:mjs@apple.com" target="_blank">mjs@apple.com</a>> \
wrote:</div><br><div><div style="overflow-wrap: break-word;"><div><br></div>There's a \
number of mysterious timeouts with 96. Not sure if flakiness or \
real?<div><br></div><div>The new WebCrypto failures are surprising, but likely real \
and should be investigated: <a \
href="https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013" \
target="_blank">https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter \
=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013</a></div></div></div></blockquote><div><br></div><div><span \
style="color:rgb(0,0,0)">I believe these tests are flaky. I have made a PR to improve \
it a while ago. I should probably get those improvement landed sometime.</span><div \
style="color:rgb(0,0,0)"><a \
href="https://github.com/web-platform-tests/wpt/pull/6102" \
target="_blank">https://github.com/web-platform-tests/wpt/pull/6102</a></div></div><br><blockquote \
type="cite"><div><div style="overflow-wrap: break-word;"><div><br></div><div>Many \
large new chunks of not-run tests are caused by a harness error or timeout. E.g. \
html/ and webauthn/</div></div></div></blockquote><div><br></div><div>STP 96 has \
enabled user agent UI for WebAuthn, which could cause crashes. This could be the \
reason why webauthn is 0 all the \
time.</div><div><br></div><div>Best,</div><div>Jiewen</div><br><blockquote \
type="cite"><div><div style="overflow-wrap: \
break-word;"><div><br></div><div><div><br><blockquote type="cite"><div>On Nov 27, \
2019, at 7:07 AM, Stephen Mcgruer <<a href="mailto:smcgruer@chromium.org" \
target="_blank">smcgruer@chromium.org</a>> wrote:</div><br><div><div \
dir="ltr">Excited to announce that Safari Tech Preview 96 is now available on <a \
href="http://wpt.fyi/" target="_blank">wpt.fyi</a>!<div><br>Example run: <a \
href="https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned" \
target="_blank">https://wpt.fyi/results/?label=master&label=experimental&produ \
ct=chrome&product=firefox&product=safari&aligned</a></div><div><br></div><div>Diff \
against Safari Tech Preview 95 (not at exactly the same WPT sha, but very close): <a \
href="https://wpt.fyi/results/?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013" \
target="_blank">https://wpt.fyi/results/?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013</a></div><div><br></div><div>If \
you are surprised by the results in the diff view, I would love to hear about it. One \
thing that surprised me is that the release blog post[0] mentioned Web Animations \
being enabled by default, but we see no differences in the test diff for \
web-animations/. It's possible we enable some flag that turned on Web Animations \
already, but I can't see it obviously in our safari \
setup[1].</div><div><br></div><div>[0]: <a \
href="https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/" \
target="_blank">https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/</a></div><div>[1]: \
<a href="https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml" \
target="_blank">https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml</a></div></div>
_______________________________________________<br>webkit-dev mailing list<br><a \
href="mailto:webkit-dev@lists.webkit.org" \
target="_blank">webkit-dev@lists.webkit.org</a><br><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" \
target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br></div></bl \
ockquote></div><br></div></div>_______________________________________________<br>webkit-dev \
mailing list<br><a href="mailto:webkit-dev@lists.webkit.org" \
target="_blank">webkit-dev@lists.webkit.org</a><br><a \
href="https://lists.webkit.org/mailman/listinfo/webkit-dev" \
target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br></div></blockquote></div><br></div></blockquote></div>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic