'Re: [webkit-dev] Huge improvement in Safari results on wpt.fyi'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       webkit-dev
Subject:    Re: [webkit-dev] Huge improvement in Safari results on wpt.fyi
From:       Philip_Jägenstedt <foolip () chromium ! org>
Date:       2018-10-12 15:25:45
Message-ID: CAARdPYcKpe3XEEKcdED6T8nZJkSbK2paSGNH5aPYhCYy-SY1Jw () mail ! gmail ! com
[Download RAW message or body]

Hi Dean,

On the run of Safari that was used for this report, the infrastructure
test for ahem was actually passing:
https://wpt.fyi/results/infrastructure/assumptions?sha=67152fdecd&product=chrome[stable]&product=edge[stable]&product=firefox[stable]&product=safari[experimental]

Are you sure that Ahem is the explanation for the failures, do you
have a test that you think is actually passing and the wpt.fyi results
are wrong? Clearly, having screenshots would make it easier to
understand a situation like this, and it's something we've discussed a
bit today:
https://github.com/web-platform-tests/wpt.fyi/issues/57
On Fri, Oct 12, 2018 at 3:01 AM Dean Jackson <dino@apple.com> wrote:
> 
> It turns out that many (most?) of the CSS failures are because we no longer expose \
> user-installed fonts, e.g. Ahem. 
> Options:
> 
> - update lots of tests to load Ahem via @font-face (yuck)
> - allow Ahem to be used if installed (weird to special case one font, but probably \
> ok) 
> Dean
> 
> > On 12 Oct 2018, at 03:26, Philip Jägenstedt <foolip@chromium.org> wrote:
> > 
> > Alright, I've written a one-off script [1] to find the Safari-only
> > failures, and here's the output:
> > https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e
> > 
> > Barring bugs, each of linked tests or one of its subtests should be
> > failing in Safari Technology Preview and passing in stable versions of
> > Chrome, Edge and Firefox.
> > 
> > Numerically, most of the failures are in css (622), encoding (135) and
> > html (60). With css, it's mostly css/CSS2.
> > 
> > I hope looking through this may be of use to you!
> > 
> > [1] https://github.com/foolip/ad-hoc-wpt-results-analysis
> > 
> > On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <foolip@chromium.org> wrote:
> > > 
> > > That filtering capability unfortunately does not yet exist on wpt.fyi
> > > but it's a high priority and actively being worked on:
> > > https://github.com/web-platform-tests/wpt.fyi/issues/201
> > > 
> > > FWIW, I suspect that these purposes, comparing to the stable versions
> > > of all *other* browsers might be the most useful:
> > > https://wpt.fyi/results/?product=chrome%5Bstable%5D&product=edge%5Bstable%5D&product=firefox%5Bstable%5D&product=safari%5Bexperimental%5D&aligned
> > >  
> > > Again, no way to filter on wpt.fyi, but I'll see if I can download the
> > > full results and write a quick script.
> > > 
> > > On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rniwa@webkit.org> wrote:
> > > > 
> > > > Thanks for the intriguing data, Philip.
> > > > 
> > > > Is there a way to get a list of tests where all other browsers pass but \
> > > > Safari / WebKit fail? 
> > > > That would allow us to quickly identify the set of tests we can fix to \
> > > > improve the interoperability across browsers right away. 
> > > > - R. Niwa
> > > > 
> > > > On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <foolip@chromium.org> \
> > > > wrote:
> > > > > 
> > > > > Hi WebKittens,
> > > > > 
> > > > > Fresh off the bots, I'm excited to report more robust Safari results,
> > > > > and that Safari WPT pass rates are clearly improving! Thanks to the
> > > > > hard work of Mike Pennisi [1] we now have the first Safari 12 results:
> > > > > https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0
> > > > > 
> > > > > This uses the same setup as for Safari Technology Preview, which has
> > > > > been running for a while [2] and are the results you see on the
> > > > > "experimental" view:
> > > > > https://wpt.fyi/results/?label=experimental
> > > > > 
> > > > > This appears much more robust than the Safari 11 data we've collected
> > > > > from Sauce Labs, and we can see a massive improvement between Safari
> > > > > 11 and 12:
> > > > > https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff
> > > > >  
> > > > > This lumps together infrastructure improvements as well as Safari
> > > > > 11->12 improvements, but improvements in service-workers/ [3] stands
> > > > > out, as well as in webdriver/, referrer-policy/, css/css-align/, and
> > > > > others. (The effect of moving away from Sauce is mainly less
> > > > > timeouts.)
> > > > > 
> > > > > Also very interesting is to compare Safari 12 stable to TP:
> > > > > https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff
> > > > >  
> > > > > One can tell that work is going in canvas-related things,
> > > > > web-animations/, css/css-logical/ and more! \o/
> > > > > 
> > > > > I hope you'll all find these results valuable, and please report bugs
> > > > > or feature requests here:
> > > > > https://github.com/web-platform-tests/wpt.fyi/issues
> > > > > 
> > > > > P.S. We're also trying to use use these diff views to spot
> > > > > regressions. It's a bit hard to use, [4] but a fix in in progress [5]
> > > > > and I might check back here when that works. I'll append to the end of
> > > > > this email a non-exhaustive list of possible regressions already
> > > > > possible to spot.
> > > > > 
> > > > > [1] https://github.com/web-platform-tests/results-collection/issues/604
> > > > > [2] https://wpt.fyi/test-runs?labels=safari,experimental
> > > > > [3] https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
> > > > >  [4] https://github.com/web-platform-tests/wpt.fyi/issues/411
> > > > > [5] https://github.com/web-platform-tests/wpt.fyi/pull/609
> > > > > 
> > > > > P.P.S. Possible regressions in Safari TP:
> > > > > https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> > > > >  https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> > > > >  https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> > > > >  _______________________________________________
> > > > > webkit-dev mailing list
> > > > > webkit-dev@lists.webkit.org
> > > > > https://lists.webkit.org/mailman/listinfo/webkit-dev
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[prev in list] [next in list] [prev in thread] [next in thread] 