[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-2d-dev
Subject: Re: [OpenJDK 2D-Dev] [9] Review request for 8029339 Custom MultiResolution image support on HiDPI di
From: Alexander Scherbatiy <alexandr.scherbatiy () oracle ! com>
Date: 2014-02-27 12:54:42
Message-ID: 530F3592.40600 () oracle ! com
[Download RAW message or body]
On 2/22/2014 3:54 AM, Jim Graham wrote:
> Hi Alexandr,
>
> On 2/18/14 7:33 AM, Alexander Scherbatiy wrote:
> > Hi Jim,
> >
> > Let's divide the discussion into two part.
> >
> > 1. Where it is better to hold resolution variants?
> > Putting resolution variants in Image class brings some questions
> > like:
> > - Some type of images do not need to have resolution variants
> > - Should resolution variants have the same type as the base image?
> > - getResolutionVariants() method can return copy of the original list
> > so add/removeRV methods should be also added.
> >
> > There are pros and cons for placing resolution variants to Image
> > class or to a separate intreface.
>
> I agree that this could be a separate interface. In my examples below
> I was just sticking them inside an "Image{}" to show where they lived
> in the set of involved objects, not a specific recommendation that
> they actually be new methods on the base class itself. I probably
> should have put a comment there about that.
>
> With respect to add/remove - that is assuming a need for manual
> construction of an image set, right? Forgive me if I'm forgetting
> something, but I seem to recall that manual Multi-Res images was
> proposed as a way for developers to introduce @2x support themselves,
> but if we are internally managing @2x and -DPI variants for them, then
> I'm not sure if there is actual developer need to manually construct
> their own. Am I forgetting something?
The NSImage has addRepresentation/removeRepresentation methods to
work with image representations on Mac OS X.
The java.awt.Image class should provide similar functionality to
have the possibilities as Cocoa on HiDPI displays.
>
> > 2. Using scale factor/image sizes/scaled image sizes to retreive a
> > resolution variant.
> >
> > May be it is better to have a structure that provide all necessary
> > information to query the resolution variant: scale factor, draw area
> > width/height, transformed area width/height?
> >
> > For example:
> > ---------------------
> > public interface MultiResolutionImage {
> >
> > interface DrawAreaInfo {
> >
> > float getScaleFactor();
> > float getAreaWidth();
> > float getAreaHeight();
> > float getTransformedAreaWidth();
> > float getTransformedAreaHeight();
> > }
> >
> > public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
> > public List<Image> getResolutionVariants();
> > }
> > ---------------------
>
> The problem with a constructor is that this is something that is
> (potentially) done on every drawImage() call, which means we are
> inviting GC into the equation. If we can come up with a simple "just
> a couple/3/4 numbers" way to embed that data into a method call
> argument list then we can make this lighter weight.
>
> What about simply having floating point (double) dimensions on the
> rendered size
There should be a way to choose a resolution variant based on
requested drawing size or transformed drawing size.
At least a current transformation should be included too.
> plus a single floating point "logical DPI" for the screen?
There is the ID2D1Factory::GetDesktopDpi method which returns dpiX
and dpiY.
http://msdn.microsoft.com/en-us/library/windows/apps/dd371316
That means that logicalDPIX/Y can have different values.
At least it is described in the
http://msdn.microsoft.com/en-us/library/windows/apps/ff684173
"To get the DPI setting, call the ID2D1Factory::GetDesktopDpi
method. The DPI is returned as two floating-point values, one for the
x-axis and one for the y-axis. In theory, these values can differ.
Calculate a separate scaling factor for each axis."
The getResolutionVariant method could look like:
--------------------------------------
public Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
float widthX, float widthY, AffineTransform transform);
--------------------------------------
> If the image is known (either passed as an argument or the method is
> called on the image), then it can provide the original WH.
>
> > The MultiResolutionImage default implementation could allow to use
> > different strategies like scale factor/transfom/OS based
> > to query a resolution variant. The OS based strategy can be used by
> > default.
>
> For Mac policy, all we need is the transformed dimensions, which can
> be passed in as FP for generality. For Windows policy, all we need is
> logical DPI for the screen. What other information would we need, or
> would an algorithm like to use, that can't be computed from those 2
> pieces?
The aim is to provide a base class that can be used to create a
MultiResolutionImage like:
http://hg.openjdk.java.net/jdk9/client/jdk/diff/ae53ebce5fa3/src/share/classes/sun/awt/image/MultiResolutionBufferedImage.java
A developer should be able to implement a custom algorithm to
query a resolution variant.
It can be done by overriding the getResolutionVariant image:
-----------------------
Image mrImage = new MultiResolutionBufferedImage(){
@Override
public Image getResolutionVariant(...) {
// Custom logic here
}
};
-----------------------
Or it can be done by using resolution variant choosers so a
developer can implement custom resolution variant query:
-----------------------
public class MultiResolutionBufferedImage implements MultiResolutionImage{
interface ResolutionVariantChooser{
Image getResolutionVariant(dpi, size,..., List<Image>
resolutionVariants);
}
ResolutionVariantChooser TRANSFORM_BASED = null;
ResolutionVariantChooser DPI_BASED = null;
ResolutionVariantChooser rvChooser;
@Override
public Image getResolutionVariant(dpi, size,...,) {
return rvChooser.getResolutionVariant(dpi, size,...,
getResolutionVariants());
}
}
-----------------------
Thanks,
Alexandr.
>
> ...jim
>
> > Thanks,
> > Alexandr.
> >
> >
> > On 2/13/2014 4:42 AM, Jim Graham wrote:
> > > On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
> > > > On 2/8/2014 4:19 AM, Jim Graham wrote:
> > > > > The primary thing that I was concerned about was the presence of
> > > > > integers in the API when Windows uses non-integer multiples
> > > > It would make sense to pass real numbers to the
> > > > getResolutionVariant() method if the difference between resolution
> > > > variants sizes is 1.
> > > > It seems that it is not a common case.
> > >
> > > I was thinking of other API that is related to this, such as the API
> > > that queries the scaling factor from a SurfaceManager. I seem to
> > > remember some integer return values in that, but Windows might have
> > > the answer 1.4 or 1.8, depending on the screen scaling factor that was
> > > determined from the UI.
> > >
> > > In terms of the getResolutionVariant() method here, those non-integer
> > > screen scaling factors don't directly impact this API. But, we have
> > > some issues with the use of integers there from other sources:
> > >
> > > - That API assumes that the caller will determine the pixel size
> > > needed, but the actual media choice is determined with different
> > > techniques on Mac and Windows so this means that the caller will have
> > > to worry about platform conventions. Is that the right tradeoff?
> > >
> > > - The technique recommended for Mac involves computing the precise
> > > size desired using the current transform, which may be a floating
> > > point value, so the integer values used in this API are already
> > > approximations and there is no documentation on how to generate the
> > > proper integer. In particular, the current code in SG2D naively uses
> > > a cast to integer to determine the values to supply which means a
> > > transformed size of W+0.5 will be truncated to W and the lower
> > > resolution image will be used. Does that conform to Mac guidelines? Do
> > > they require the truncated size to reach W+1 before the next size is
> > > used? Passing in float or double values would sidestep all of that
> > > since then the comparisons would be done with full precision, but as
> > > long as we can determine a "best practices compatible with all
> > > platforms" rule on how to round to integers, then integers are OK
> > > there.
> > >
> > > - The Windows document you cite below suggests that the determination
> > > should be made by looking at the Screen DPI and choosing the next
> > > higher media variant based on that screen DPI. They do not specify
> > > choosing media based on the current transform as is done for Mac. If
> > > we stick with supplying values that are used to determine which media
> > > to use, then on Windows we should not take the transform into account,
> > > but instead query the SurfaceManager for the scale factor and only
> > > transform by those values (even if the current transform was manually
> > > overridden to identity).
> > >
> > > There are pros and cons to both approaches.
> > >
> > > Mac ensures that you are always using the best media for any given
> > > render operation.
> > >
> > > But, Windows ensure more consistency in the face of other scaling.
> > >
> > > The thing to consider is that if you have a 500x500 image with a
> > > 1000x1000 variant and you rendering it at 500x500 and then 501x501,
> > > that first jump will be fairly jarring as the scaled version of the
> > > 1000x1000 will not look precisely like the original 500x500 did. With
> > > @2x images only, this effect is minimized so the advantage of always
> > > using "the best media for a given render operation" may outweigh the
> > > inconsistency issue. But, on Windows where the media are 1.4x or 1.8x
> > > in size, a downscaled image will start to show more interpolation
> > > noise and so the balance of the two choices may shift more towards not
> > > wanting a jarring shift.
> > >
> > > We might want one or more of the following:
> > >
> > > - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE,
> > > PLATFORM) where the last policy would use TX_AWARE on Mac and
> > > DPI_AWARE on Windows
> > >
> > > - We create our own policy and always use it (TX_AWARE? or DPI_AWARE?)
> > >
> > > - We create our own policy that dynamically chooses one of the above
> > > strategies depending on platform or available media or ???
> > >
> > > - We could create an optional interface for them to install their own
> > > algorithm as well. I think it would work best as a delegate interface
> > > that one installs into Image so that it can be used with any image
> > > without having to subclass (it wouldn't really have much to do for
> > > BufferedImages or VolatileImages, though):
> > >
> > > class Image {
> > > void setResolutionHelper(ImageResolutionHelper foo);
> > > List<Image> getResolutionVariants();
> > > }
> > >
> > > or:
> > >
> > > class Graphics {
> > > void setResolutionHelper(ImageResolutionHelper foo);
> > > }
> > >
> > > or - anywhere else it could be installed more centrally (per App
> > > context)?
> > >
> > > and the interface would be something like one of these variants:
> > >
> > > interface ImageResolutionHelper {
> > > // This version would prevent substituting a random image:
> > > // They have to return an index into the List<Image> for that
> > > image...
> > > public int chooseVariant(Image img, double dpi, number w, number
> > > h);
> > >
> > > or:
> > >
> > > // This version would allow substituting an arbitrary image:
> > > public Image getVariant(Image img, double dpi, number w, number h);
> > > }
> > >
> > > Since they would be in full control of the policy, though, we would
> > > unfortunately always have to call this, there would be no more testing
> > > in SG2D to see "if" we need to deal with DPI, though perhaps we could
> > > document some internal conditions in which we do not call it for
> > > common cases (but that would have to be well agreed not to get in the
> > > way of reasonable uses of the API and well documented)?
> > >
> > > Note that we would have to do a security audit to make sure that
> > > random image substitution couldn't allow any sort of "screen phishing"
> > > exploit.
> > >
> > > ...jim
> > >
> > > > > and also what policy they use for choosing scaled images.
> > > > >
> > > > > I don't see a mention of taking the current transform into account,
> > > > > just physical issues like screen DPI and form factor. They talk about
> > > > > resolution plateaus and in their recommendations section they tell
> > > > > the
> > > > > developer to use a particular property that tells them the screen
> > > > > resolution to figure out which image to load if they are loading
> > > > > manually. There is no discussion about dynamically loading multiple
> > > > > versions of the image based on a dynamic program-applied transform
> > > > > factor as is done on MacOS.
> > > > >
> > > > > Also, they tell developers to draw images to a specific size rather
> > > > > than using auto-sizing. That begs the question as to how they
> > > > > interpret a call to draw an image just using a location in the
> > > > > presence of various DPI factors.
> > > > There is an interesting doc that describes how to write
> > > > DPI-aware
> > > > Win32 applications:
> > > > http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
> > > > It is suggested to handle WM_DPICHANGED message, load the
> > > > graphic
> > > > that has slightly greater resolution to the current DPI and use
> > > > StretchBlt
> > > > to scale down the image.
> > > >
> > > > Thanks,
> > > > Alexandr.
> > > >
> > > > >
> > > > > ...jim
> > > > >
> > > > > On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
> > > > > > On 1/22/2014 6:40 AM, Jim Graham wrote:
> > > > > > > Hi Alexander,
> > > > > > >
> > > > > > > Before we get too far down the road on this API, I think we
> > > > > > > understand
> > > > > > > the way in which MacOS processes multi-resolution images for HiDPI
> > > > > > > screens, but have we investigated the processes that Windows uses
> > > > > > > under Windows 8? My impression is that Windows 8 has included a
> > > > > > > number of new techniques for dealing with the high resolution
> > > > > > > displays
> > > > > > > that it will run on in the Windows tablet and mobile industries and
> > > > > > > that these will also come into play as 4K displays (already
> > > > > > > available)
> > > > > > > become more common on the desktop. We should make sure that
> > > > > > > what we
> > > > > > > come up with here can provide native compatibility with either
> > > > > > > platform's policies and standard practices.
> > > > > > >
> > > > > > > If you've investigated the MS policies I'd like to see a summary so
> > > > > > > that we can consider them as we review this API...
> > > > > > There is the Windows Guidelines for scaling to pixel density:
> > > > > > http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
> > > > > > which says that Windows has automatic resource loading that
> > > > > > supports
> > > > > > three version of images scaling (100%, 140%, and 180%)
> > > > > > --------------------------------
> > > > > > Without scaling, as the pixel density of a display device
> > > > > > increases, the
> > > > > > physical sizes of objects on screen get smaller.
> > > > > > When UI would otherwise be too small to touch and when text gets too
> > > > > > small to read,
> > > > > > Windows scales the system and app UI to one of the following scaling
> > > > > > plateaus:
> > > > > >
> > > > > > 1.0 (100%, no scaling is applied)
> > > > > > 1.4 (140% scaling)
> > > > > > 1.8 (180% scaling)
> > > > > >
> > > > > > Windows determines which scaling plateau to use based on the
> > > > > > physical
> > > > > > screen size, the screen resolution, the DPI of the screen, and form
> > > > > > factor.
> > > > > >
> > > > > > Use resource loading for bitmap images in the app package For bitmap
> > > > > > images stored
> > > > > > in the app package, provide a separate image for each scaling
> > > > > > factor(100%, 140%, and 180%),
> > > > > > and name your image files using the "scale" naming convention
> > > > > > described
> > > > > > below.
> > > > > > Windows loads the right image for the current scale automatically.
> > > > > > --------------------------------
> > > > > >
> > > > > > The image name convention for the various scales is:
> > > > > > images/logo.scale-100.png
> > > > > > images/logo.scale-140.png
> > > > > > images/logo.scale-180.png
> > > > > >
> > > > > > The 'ms-appx:///images/logo.png' uri is used to load the image
> > > > > > in an
> > > > > > application.
> > > > > >
> > > > > > If we want to support this in the same way as it is done for Mac
> > > > > > OS X
> > > > > > the WToolkit should return MultiResolution image in case if the
> > > > > > loaded image has .scale-* qualifiers.
> > > > > > The Graphics class can request an image with necessary resolution
> > > > > > from the MultiResolution image.
> > > > > >
> > > > > > It seems that nothing should be changed in the MultiResolution
> > > > > > interface in this case.
> > > > > >
> > > > > > Thanks,
> > > > > > Alexandr.
> > > > > > >
> > > > > > > ...jim
> > > > > > >
> > > > > > > On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > Could you review the fix:
> > > > > > > > bug: https://bugs.openjdk.java.net/browse/JDK-8029339
> > > > > > > > webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
> > > > > > > >
> > > > > > > > This is a proposal to introduce an API that allows to create a
> > > > > > > > custom
> > > > > > > > multi resolution image.
> > > > > > > >
> > > > > > > > I. It seems reasonable that the API should provide two basic
> > > > > > > > operations:
> > > > > > > >
> > > > > > > > 1. Get the resolution variant based on the requested image
> > > > > > > > width and
> > > > > > > > height:
> > > > > > > > - Image getResolutionVariant(int width, int height)
> > > > > > > >
> > > > > > > > Usually the system provides the scale factor which represents
> > > > > > > > the
> > > > > > > > number of pixels corresponding to each linear unit on the display.
> > > > > > > > However, it has sense to combine the scale factor and the
> > > > > > > > current
> > > > > > > > transformations to get the actual image size to be displayed.
> > > > > > > >
> > > > > > > > 2. Get all provided resolution variants:
> > > > > > > > - List<Image> getResolutionVariants()
> > > > > > > >
> > > > > > > > There are several uses cases:
> > > > > > > > - Create a new multi-resolution image based on the given
> > > > > > > > multi-resolution image.
> > > > > > > > - Pass to the native system the multi-resolution image. For
> > > > > > > > example,
> > > > > > > > a use can set to the system the custom multi-resolution cursor.
> > > > > > > >
> > > > > > > > II. There are some possible ways where the new API can be added
> > > > > > > >
> > > > > > > > 1. java.awt.Image.
> > > > > > > >
> > > > > > > > The 2 new methods can be added to the Image class. A user can
> > > > > > > > override
> > > > > > > > the getResolutionVariant() and getResolutionVariants()
> > > > > > > > methods to
> > > > > > > > provide the resolution variants
> > > > > > > > or there can be default implementations of these methods if a
> > > > > > > > user
> > > > > > > > puts resolution variants
> > > > > > > > to the list in the sorted order.
> > > > > > > >
> > > > > > > > To check that the image has resolution variants the following
> > > > > > > > statement can be used: image.getResolutionVariants().size() != 1
> > > > > > > >
> > > > > > > > The disadvantage is that there is an overhead that the Image
> > > > > > > > class
> > > > > > > > should contain the List object and not all
> > > > > > > > images can have resolution variants.
> > > > > > > >
> > > > > > > > 2. Introduce new MultiResolutionImage interface.
> > > > > > > >
> > > > > > > > A user should extend Image class and implement the
> > > > > > > > MultiResolutionImage interface.
> > > > > > > >
> > > > > > > > For example:
> > > > > > > > ---------------------
> > > > > > > > public class CustomMultiResolutionImage extends BufferedImage
> > > > > > > > implements MultiResolutionImage {
> > > > > > > >
> > > > > > > > Image highResolutionImage;
> > > > > > > >
> > > > > > > > public CustomMultiResolutionImage(BufferedImage
> > > > > > > > baseImage,
> > > > > > > > BufferedImage highResolutionImage) {
> > > > > > > > super(baseImage.getWidth(), baseImage.getHeight(),
> > > > > > > > baseImage.getType());
> > > > > > > > this.highResolutionImage = highResolutionImage;
> > > > > > > > Graphics g = getGraphics();
> > > > > > > > g.drawImage(baseImage, 0, 0, null);
> > > > > > > > g.dispose();
> > > > > > > > }
> > > > > > > >
> > > > > > > > @Override
> > > > > > > > public Image getResolutionVariant(int width, int
> > > > > > > > height) {
> > > > > > > > return ((width <= getWidth() && height <=
> > > > > > > > getHeight()))
> > > > > > > > ? this : highResolutionImage;
> > > > > > > > }
> > > > > > > >
> > > > > > > > @Override
> > > > > > > > public List<Image> getResolutionVariants() {
> > > > > > > > return Arrays.asList(this, highResolutionImage);
> > > > > > > > }
> > > > > > > > }
> > > > > > > > ---------------------
> > > > > > > >
> > > > > > > > The current fix adds the MultiResolutionImage interface and
> > > > > > > > public
> > > > > > > > resolution variant rendering hints.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alexandr.
> > > > > > > >
> > > > > >
> > > >
> >
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic