[prev in list] [next in list] [prev in thread] [next in thread]
List: whatwg
Subject: Re: [whatwg] Adding features needed for WebGL to ImageBitmap
From: Rik Cabanier <cabanier () gmail ! com>
Date: 2013-06-20 3:19:05
Message-ID: CAGN7qDDX2yKP+eK400SAToxdRBn4tX0muEH6jyy8CWCjt=Npew () mail ! gmail ! com
[Download RAW message or body]
On Wed, Jun 19, 2013 at 4:47 PM, Gregg Tavares <gman@google.com> wrote:
>
>
>
> On Wed, Jun 19, 2013 at 4:03 PM, Rik Cabanier <cabanier@gmail.com> wrote:
>
> >
> >
> > On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares <gman@google.com> wrote:
> >
> > >
> > >
> > >
> > > On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier <cabanier@gmail.com>wrote:
> > >
> > > >
> > > > On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares <gman@google.com> wrote:
> > > >
> > > > > In order for ImageBitmap to be useful for WebGL we need more options
> > > > >
> > > > > Specifically
> > > > >
> > > > > premultipliedAlpha: true/false (default true)
> > > > > Nearly all GL games use non-premultipiled alpha textures. So all those
> > > > > games people want to port to WebGL will require non-premultipied
> > > > > textures.
> > > > > Often in games the alpha might not even be used for alpha but rather
> > > > > for
> > > > > glow maps or specular maps or the other kinds of data.
> > > > >
> > > >
> > > > When would premultipliedAlpha ever be true?
> > > > 2D Canvas always works with non-premultiplied alpha in its APIs.
> > > >
> > >
> > > AFAIK the canvas API expects all images to be premultiplied. Certainly
> > > in WebKit and Blink images used in the canvas and displayed in the img tag
> > > are loaded premultiplied which is why we had to add the option on WebGL
> > > since we needed those images before they had lost data.
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > flipY: true/false (default false)
> > > > > Nearly all 3D modeling apps expect the bottom left pixel to be the
> > > > > first
> > > > > pixel in a texture so many 3D engines flip the textures on load. WebGL
> > > > > provides this option but it takes time and memory to flip a large image
> > > > > therefore it would be nice if that flip happened before the callback
> > > > > from
> > > > > ImageBitmap
> > > > >
> > > >
> > > > Couldn't you just draw upside down?
> > > >
> > >
> > > No, games often animate texture coordinates and other things making it
> > > far more complicated. There are ways to work around this issue in code yes,
> > > they often require a ton of work.
> > >
> > > Most professional game engines pre-process the textures and flip them
> > > offline but that doesn't help when you're downloading models off say
> > > http://sketchup.google.com/3dwarehouse/
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > colorspaceConversion: true/false (default true)
> > > > > Some browsers apply color space conversion to match monitor settings.
> > > > > That's fine for images with color but WebGL apps often load heightmaps,
> > > > > normalmaps, lightmaps, global illumination maps and many other kinds of
> > > > > data through images. If the browser applies a colorspace conversion the
> > > > > data is not longer suitable for it's intended purpose therefore many
> > > > > WebGL
> > > > > apps turn off color conversions. As it is now, when an image is
> > > > > uploaded to
> > > > > WebGL, if colorspace conversion is
> > > > > off<
> > > > > http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
> > > > >
> > > > > > ,
> > > >
> > > >
> > OK, I see what you're trying to accomplish. You want to pass
> > non-premultiplied data and color converted (from sRGB to monitor) pixels to
> > WebGL
> > I think your API looks fine, except that the defaults should all be
> > false...
> >
>
> Yes, that's what I meant. I think I choose bad labels. the intent of the
> colorspaceConversion flag is
>
> colorspaceConversion: true = browser does whatever it thinks is best for
> color images.
> colorspaceConversion: false = give me the bits in the image file. Don't
> manipulate them with either embedded color data or local machine gamma
> corrections or anything else.
>
> So maybe a better name?
>
yeah. I'm still fuzzy about what exactly you're trying to do. Is this
specified anywhere?
>
> for premultipiedAlpha again, maybe there are 3 options needed
>
> 1) do whatever is best for drawing with drawImage for perf
> 2) give me the data with premutlipied alpha
> 3) give me the data with non-premultied alpha.
>
I think that sounds good. That way implementations aren't forced to
premultiply the alpha.
Maybe not passing premultipiedAlpha in the dictionary (= leave it
undefined) will signal this.
>
> It's possible that #1 is not needed as maybe GPU code can use different
> blend modes for drawImage with non-premultpiled alpha. It's just my
> understanding that at least in Chrome all images are loaded premultiplied.
> In fact I don't think you can get non-premultipied data from canvas. At
> least this does not make it appear that way
>
> c = document.createElement("canvas");
> ctx = c.getContext("2d");
> i = ctx.getImageData(0, 0, 1, 1);
> i.data[0] = 255;
> ctx.putImageData(i, 0, 0);
> i2 = ctx.getImageData(0, 0, 1, 1);
> console.log(i2.data[0]) // prints 0 on both FF and Chrome
>
> I mean I know you get unpremultiplied data from getIamgeData but the data
> in the canvas is premultipied which means if alpha is zero you lose the RGB
> data. In other words a round trip of putImageData, getImageData is lossy
> for any non 255 alpha. Change the above to
>
> c = document.createElement("canvas");
> ctx = c.getContext("2d");
> i = ctx.getImageData(0, 0, 1, 1);
> i.data[0] = 128;
> i.data[3] = 1;
> ctx.putImageData(i, 0, 0);
> i2 = ctx.getImageData(0, 0, 1, 1);
> console.log(i2.data[0]) // prints 255.
>
> This is because the data in the canvas is premutliplied and it's being
> un-premultiplied when calling getImageData
>
Yes, the canvas spec calls this out:
Due to the lossy nature of converting to and from premultiplied alpha color
values, pixels that have just been set using
putImageDataHD()<http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-putimagedatahd>
might
be returned to an equivalent
getImageDataHD()<http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-getimagedatahd>
as
different values.
I think this is really unfortunate...
>
>
> >
> > >
> > > > > WebGL has to synchronously re-decode the image. It would be nice if
> > > > > ImageBitmap could handle this case so it can decode the image without
> > > > > applying any colorspace manipulations.
> > > > >
> > > >
> > > > Shouldn't the color space conversion happen when the final canvas bit
> > > > are blitted to the screen?
> > > > It seems like you should never do it during compositing since you could
> > > > get double conversions.
> > > >
> > >
> > > Maybe but that's not relevant to ImageBitmap is it? The point here is we
> > > want the ImageBitmap to give us the data in the format we need. It's
> > > designed to be async so it can do this but it we can't prevent it from
> > > applying colorspace conversions.
> > > Some browsers did that for regular img tags which pointed out the
> > > original problem. The browser can't guess how the image is going to be used
> > > and since it's a lot of work to decode an image you'd like to be able to
> > > tell it what you really need before it guesses wrong.
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > If it was up to me I'd make createImageBitmap take on object with
> > > > > properties so that new options can be added later as in
> > > > >
> > > > > createImageBitmap(src, callback, {
> > > > > premultipliedAlpha: false,
> > > > > colorspaceConversion: false,
> > > > > x: 123,
> > > > > });
> > > > >
> > > > > But I'm not familiar if there is a common way to make APIs take a
> > > > > options
> > > > > like this except for the XHR way which is to create a request, set
> > > > > properties on the request, and finally execute the request.
> > > >
> > > >
> > > > Like Tab said, it's fine to implement it that way.
> > > > Be aware that you might have to do some work in your idl compiler since
> > > > I *think* there are no other APIs (in Blink) that take a dictionary.
> > > >
> > > >
> > >
> >
>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic