From webkit-dev Thu Mar 05 02:24:26 2020 From: Maciej Stachowiak Date: Thu, 05 Mar 2020 02:24:26 +0000 To: webkit-dev Subject: Re: [webkit-dev] Content-DPR: Image resolution optimization at CDN/transport layer Message-Id: <352F4099-2129-491F-8542-F3C3153B34A6 () apple ! com> X-MARC-Message: https://marc.info/?l=webkit-dev&m=158337517226001 MIME-Version: 1 Content-Type: multipart/mixed; boundary="--===============0927812751==" --===============0927812751== Content-type: multipart/alternative; boundary="Apple-Mail=_38AEE2A9-07B4-44FE-A9F4-B62C54EFEDDE" --Apple-Mail=_38AEE2A9-07B4-44FE-A9F4-B62C54EFEDDE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 2, 2020, at 11:22 PM, Noam Rosenthal wrote: >=20 >=20 >=20 > On Mon, Nov 11, 2019 at 10:13 PM Maciej Stachowiak > wrote: >=20 >=20 >> On Nov 11, 2019, at 4:37 AM, Noam Rosenthal > wrote: >>=20 >>=20 >>=20 >> On Mon, Nov 11, 2019 at 12:13 AM Maciej Stachowiak > wrote: >> - Is there a specification for Content-DPR? All I could find in = bugzila and elsewhere is links to = >, which does not = mention Content-DPR at all). >> In a nutshell, the spec is in transition from IETF to HTML/fetch, and = the client-hints aspects of it are still in progress (unlike conent-dpr, = which is much simpler hasn't changed since introduced). It's documented = here: = https://github.com/yoavweiss/client-hints-infrastructure/blob/master/speci= fication_situation.md = . >> I will answer more about this separately if required, waiting for = some info from the people working on the spec originally. >=20 > It looks like these are the relevant Pull Requests: >=20 >=20 > HTML: https://github.com/whatwg/html/pull/3774 = | = https://whatpr.org/fetch/773/6a644c6...5067615.html#concept-response-image= -density = > Fetch: https://github.com/whatwg/fetch/pull/773 = | = https://whatpr.org/html/3774/e32a6f8...ddb0544/images.html#current-pixel-d= ensity = >=20 > It looks like these are in a somewhat messy state. For example, Fetch = places the Content-DPR value in an =E2=80=9Cimage density=E2=80=9D = variable, but it doesn=E2=80=99t look like the Pull Request to HTML = doesn=E2=80=99t use it anywhere. As another example, HTML directly = references Content-DPR as setting the =E2=80=9Ccurrent pixel density=E2=80= =9D, but does not specify that it would take priority over a density = derived from srcset. There are also no diffs to CSS Images or to SVG, = which provide distinct ways to reference images and which presumably = should respect Content-DPR. >=20 > =20 >=20 >> =20 >> - Can you give us some examples of how the CDN would make the = decision of what density of image to serve without knowing the = client=E2=80=99s DPR via client-hints?=20 >> Some examples from https://github.com/eeeps/content-dpr-explainer = : >> - making decisions by user agent, like choosing to cap certain images = for user-agents known to have smaller displays >> - making decisions by traffic/geo, like serving lower-resolution = images when traffic is bogged down >> - A client may ask for "placeholder image" (e.g. ?placeholder=3Dtrue = in the URL), and the CDN would decide whether to serve a lower-quality = JPEG or to scale it down, or do both. >=20 > These seem like reasonable use cases. >=20 >>=20 >> - Is this presuming situations where the CDN serves the images but = not the markup or CSS (which presumably could be rewritten to account = for DPR)? >> Correct. >> =20 >> - What happens if Content-DPR is provided, but the markup or CSS has = conflicting explicit info? For example, , and image-2x.jpg is served with a = Content-DPR header of 3x. Or the analogous thing with CSS. >> When image size is decided, css/markup takes precedence, then content = DPR, and finally srcset-selection DPR. This is actually a bonus value of = this feature - if the server happens to serve an image that differs from = the ratio requested in the srcset tag, the intrinsic size match the = served content and not the request (which is more of a recommendation). >>=20 >> - Does Content-DPR have any effect on images where an explicit size = is provided, either in markup or in CSS? (I=E2=80=99m assuming no.) >> No, it only has effect on intrinsic size.=20 >=20 > Overall, this seems like a feature with valid use cases. But = unfortunately, it=E2=80=99s a (currently) Blink-only feature with no = clear specification. To the extent there is a spec, it=E2=80=99s mixed = in with multiple Pull Requests. These pull requests mix in Client Hints, = a feature unlikely to gain multi implementor support, so they probably = won=E2=80=99t land any time soon. And the language about Content-DPR is = broken and does not not seem to match Chrome=E2=80=99s actual = implementation. >=20 > So in summary, there is no proper spec, and Chrome has shown = willingness to have their implementation change faster than even draft = spec language in PRs can keep up with.=20 >=20 > Although the use case seems valid, I don=E2=80=99t think it=E2=80=99s = a good idea for WebKit to implement the feature in this state. It seems = likely to cause interop headaches and a need to reverse engineer Blink = rather than developing against specs and test cases. >=20 > There has been a lot of work done on the spec front since the comments = above, and I think we're ready for a re-review the state. > * The next pull request: https://github.com/whatwg/html/pull/5112 = has been thumbed up to make = it into the HTML spec, and is separate from client-hints, but requires = second implementor interest, which is what this email thread is for :) I don=E2=80=99t think it has? I=E2=80=99m seeing: "[ ] At least two implementers are interested (and none opposed):=E2=80=9D= And satisfying this is a requirement for adding features to WHATWG = Living Standards. And the review block at the bottom says =E2=80=9CReview required=E2=80=9D = and =E2=80=9CMerging is Blocked with big red X=E2=80=99s. > * Web platform tests are comprehensive, separated from client-hints, = and match both the spec and the Chrome implementation: = https://github.com/web-platform-tests/wpt/tree/master/content-dpr = . We = can always add more if we find more use-cases. >=20 > Would love to hear more thoughts! Sounds like a good time to re-review. Regards, Maciej --Apple-Mail=_38AEE2A9-07B4-44FE-A9F4-B62C54EFEDDE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Mar 2, 2020, at 11:22 PM, Noam Rosenthal <noam@webkit.org> = wrote:



On Mon, Nov 11, 2019 at 10:13 PM Maciej Stachowiak = <mjs@apple.com> = wrote:


On Nov 11, 2019, at 4:37 AM, = Noam Rosenthal <noam@webkit.org> wrote:



On Mon, Nov 11, 2019 at 12:13 AM Maciej = Stachowiak <mjs@apple.com> wrote:
- Is there a specification for Content-DPR? = All I could find in bugzila and elsewhere is links to <https://httpwg.org/http-extensions/client-hints.html>, = which does not mention Content-DPR at = all).
In a nutshell, the = spec is in transition from IETF to HTML/fetch, and the client-hints = aspects of it are still in progress (unlike conent-dpr, which is much = simpler hasn't changed since introduced). It's documented here: https://github.com/yoavweiss/client-hints-infrastructure/blob/m= aster/specification_situation.md.
I will answer more about this separately if required, waiting = for some info from the people working on the spec = originally.

It looks like these are the relevant Pull = Requests:



It looks like these are in a somewhat messy state. For = example, Fetch places the Content-DPR value in an =E2=80=9Cimage = density=E2=80=9D variable, but it doesn=E2=80=99t look like the Pull = Request to HTML doesn=E2=80=99t use it anywhere. As another example, = HTML directly references Content-DPR as setting the =E2=80=9Ccurrent = pixel density=E2=80=9D, but does not specify that it would take priority = over a density derived from srcset. There are also no diffs to CSS = Images or to SVG, which provide distinct ways to reference images and = which presumably should respect = Content-DPR.

 

 
- Can you give us some examples of how the = CDN would make the decision of what density of image to serve without = knowing the client=E2=80=99s DPR via =  client-hints? 
- making decisions by user agent, like choosing to cap = certain images for user-agents known to have smaller displays
- making decisions by traffic/geo, like serving = lower-resolution images when traffic is bogged down
-= A client may ask for "placeholder image" (e.g. ?placeholder=3Dtrue in = the URL), and the CDN would decide whether to serve a lower-quality JPEG = or to scale it down, or do = both.

These seem like reasonable use cases.


- Is = this presuming situations where the CDN serves the images but not the = markup or CSS (which presumably could be rewritten to account for = DPR)?
Correct.
 
- What happens if Content-DPR = is provided, but the markup or CSS has conflicting explicit info? For = example, <img srcset=3D=E2=80=9Cimage-2x.jpg 2x, image.jpg 1x=E2=80=9D&= gt;, and image-2x.jpg is served with a Content-DPR header of 3x. Or the = analogous thing with CSS.
When image size is decided, css/markup takes precedence, then = content DPR, and finally srcset-selection DPR. This is actually a bonus = value of this feature - if the server happens to serve an image that = differs from the ratio requested in the srcset tag, the intrinsic size = match the served content and not the request (which is more of a = recommendation).

- Does Content-DPR have any effect on images = where an explicit size is provided, either in markup or in CSS? (I=E2=80=99= m assuming no.)
No, it = only has effect on intrinsic = size. 

Overall, this seems like a feature with valid use cases. But = unfortunately, it=E2=80=99s a (currently) Blink-only feature with no = clear specification. To the extent there is a spec, it=E2=80=99s mixed = in with multiple Pull Requests. These pull requests mix in Client Hints, = a feature unlikely to gain multi implementor support, so they probably = won=E2=80=99t land any time soon. And the language about Content-DPR is = broken and does not not seem to match Chrome=E2=80=99s actual = implementation.

So in summary, there is no proper spec, and Chrome has shown = willingness to have their implementation change faster than even draft = spec language in PRs can keep up = with. 

Although = the use case seems valid, I don=E2=80=99t think it=E2=80=99s a good idea = for WebKit to implement the feature in this state. It seems likely to = cause interop headaches and a need to reverse engineer Blink rather than = developing against specs and test cases.

There has been a lot of work done on = the spec front since the comments above, and I think we're ready for a = re-review the state.
* The next pull request: https://github.com/whatwg/html/pull/5112 has been = thumbed up to make it into the HTML spec, and is separate from = client-hints, but requires second implementor interest, which is what = this email thread is for :)

I = don=E2=80=99t think it has? I=E2=80=99m seeing:
"[ ]  At = least two implementers are interested (and none = opposed):=E2=80=9D

And satisfying = this is a requirement for adding features to WHATWG Living = Standards.

And the review block at = the bottom says =E2=80=9CReview required=E2=80=9D and =E2=80=9CMerging = is Blocked with big red X=E2=80=99s.

* = Web platform tests are comprehensive, separated from client-hints, and = match both the spec and the Chrome implementation: https://github.com/web-platform-tests/wpt/tree/master/content-d= pr. We can always add more if we find more use-cases.

Would love to hear more thoughts!

Sounds like a good time to re-review.

Regards,
Maciej

= --Apple-Mail=_38AEE2A9-07B4-44FE-A9F4-B62C54EFEDDE-- --===============0927812751== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev --===============0927812751==--