[prev in list] [next in list] [prev in thread] [next in thread]
List: vtk-developers
Subject: Re: [vtk-developers] VTK image error 0 despite differences - false pass
From: "Marcus D. Hanwell" <marcus.hanwell () kitware ! com>
Date: 2018-06-08 14:39:37
Message-ID: CAMkPkZUKGCwskozFwra+J-sjc7Gyj_D5MwFXFF5duJbUfn1xXQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
On Fri, May 4, 2018 at 2:08 PM, Marcus D. Hanwell <
marcus.hanwell@kitware.com> wrote:
> On Fri, May 4, 2018 at 1:59 PM, Marcus D. Hanwell <
> marcus.hanwell@kitware.com> wrote:
>
>> On Fri, May 4, 2018 at 1:24 PM, Marcus D. Hanwell <
>> marcus.hanwell@kitware.com> wrote:
>>
>>> On Fri, May 4, 2018 at 9:34 AM, Marcus D. Hanwell <
>>> marcus.hanwell@kitware.com> wrote:
>>>
>>>> A while ago I opened issue https://gitlab.kitware.com/vtk
>>>> /vtk/issues/17201 that talked about a bug on Windows where setting the
>>>> precision to 0 resulted in 6 precision on Windows. There was a test, but it
>>>> was passing even on Windows, this week we had a new employee start and I
>>>> thought it would be a good first bug to create a failing test, and then
>>>> make it pass by addressing the issue.
>>>>
>>>> I thought that TestAxes was a little crowded, and that had perhaps been
>>>> enough to throw the image diff off. So he wrote a test that just had this
>>>> axis in, made a baseline for what was expected, and then changed it so that
>>>> we could see the test fail. Changing to 6 or 8 is still showing passing
>>>> test:
>>>>
>>>>
>>>>
>>>> This looks like a good difference to me (not sure if we are allowed
>>>> attachments, but I tried). The ImageError is being shown as 0. We could
>>>> make the text bigger, but there are a few chart tests where we use text of
>>>> this size and I thought they would fail with extra numbers like this.
>>>>
>>>
>>> Digging into this for anyone following, I can change vtkTesting.cxx to
>>> use 'rtId->SetAllowShift(0);' which removes the single pixel shift logic,
>>> and I get a diff showing my extra digits, as I would expect. We likely have
>>> this on for a reason, and the text is pretty tiny, so I could see a nearest
>>> neighbor search turning up no difference for every character added.
>>>
>>
>> Lots of tests fail when this is turned off, we can actually make the text
>> larger and see the expected failure in the axis test. We could redo the
>> TestAxes to use a larger font, or perhaps add an option to disable the
>> pixel shift on a per test basis (I didn't see anything exposing that in the
>> test driver.
>>
>
> For anyone interested, a merge request with dashboard results looking at
> the impact of not allowing shifts in image diffs globally,
>
> https://gitlab.kitware.com/vtk/vtk/merge_requests/4261
>
For anyone following along, I wrote up a blog post summarizing the issue,
linking to this thread, and the resolution,
https://blog.kitware.com/image-comparison-tests-vtk/
It turned up a number of issues that had been lurking with passing tests
that should have been failing. You may want to be aware if you reuse this
mechanism too.
[Attachment #7 (text/html)]
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, May 4, 2018 \
at 2:08 PM, Marcus D. Hanwell <span dir="ltr"><<a \
href="mailto:marcus.hanwell@kitware.com" \
target="_blank">marcus.hanwell@kitware.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><span class="gmail-">On Fri, May 4, 2018 at 1:59 PM, Marcus D. \
Hanwell <span dir="ltr"><<a href="mailto:marcus.hanwell@kitware.com" \
target="_blank">marcus.hanwell@kitware.com</a>></span> \
wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div><div class="gmail-m_-5952259593481058556gmail-h5"><div \
class="gmail_extra"><div class="gmail_quote"><span class="gmail-">On Fri, May 4, 2018 \
at 1:24 PM, Marcus D. Hanwell <span dir="ltr"><<a \
href="mailto:marcus.hanwell@kitware.com" \
target="_blank">marcus.hanwell@kitware.com</a>></span> \
wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
class="gmail_extra"><div class="gmail_quote"><span><span class="gmail-">On Fri, May \
4, 2018 at 9:34 AM, Marcus D. Hanwell <span dir="ltr"><<a \
href="mailto:marcus.hanwell@kitware.com" \
target="_blank">marcus.hanwell@kitware.com</a>></span> wrote:<br></span><span \
class="gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>A \
while ago I opened issue <a href="https://gitlab.kitware.com/vtk/vtk/issues/17201" \
target="_blank">https://gitlab.kitware.com/vtk<wbr>/vtk/issues/17201</a> that talked \
about a bug on Windows where setting the precision to 0 resulted in 6 precision on \
Windows. There was a test, but it was passing even on Windows, this week we had a new \
employee start and I thought it would be a good first bug to create a failing test, \
and then make it pass by addressing the issue.<br></div><div><br></div><div>I thought \
that TestAxes was a little crowded, and that had perhaps been enough to throw the \
image diff off. So he wrote a test that just had this axis in, made a baseline for \
what was expected, and then changed it so that we could see the test fail. Changing \
to 6 or 8 is still showing passing test:</div><div><br><img \
src="cid:ii_jgrzw7zj0_1632b57082296690" width="100" height="100"><img \
src="cid:ii_jgrzzhzy1_1632b595d87f0c7e" width="100" \
height="100"><br><br></div><div>This looks like a good difference to me (not sure if \
we are allowed attachments, but I tried). The ImageError is being shown as 0. We \
could make the text bigger, but there are a few chart tests where we use text of this \
size and I thought they would fail with extra numbers like \
this.</div></div></blockquote><div><br></div></span></span><span \
class="gmail-"><div>Digging into this for anyone following, I can change \
vtkTesting.cxx to use 'rtId->SetAllowShift(0);' which removes the single \
pixel shift logic, and I get a diff showing my extra digits, as I would expect. We \
likely have this on for a reason, and the text is pretty tiny, so I could see a \
nearest neighbor search turning up no difference for every character \
added.</div></span></div></div></div> </blockquote></div><br></div></div></div><span \
class="gmail-"><div class="gmail_extra">Lots of tests fail when this is turned off, \
we can actually make the text larger and see the expected failure in the axis test. \
We could redo the TestAxes to use a larger font, or perhaps add an option to disable \
the pixel shift on a per test basis (I didn't see anything exposing that in the \
test driver.</div></span></div></blockquote><div><br></div><div>For anyone \
interested, a merge request with dashboard results looking at the impact of not \
allowing shifts in image diffs globally,</div><div><br></div><div><a \
href="https://gitlab.kitware.com/vtk/vtk/merge_requests/4261" \
target="_blank">https://gitlab.kitware.com/<wbr>vtk/vtk/merge_requests/4261</a> \
</div></div></div></div></blockquote><div><br></div><div>For anyone following along, \
I wrote up a blog post summarizing the issue, linking to this thread, and the \
resolution,</div><div><br></div><div><a \
href="https://blog.kitware.com/image-comparison-tests-vtk/">https://blog.kitware.com/image-comparison-tests-vtk/</a><br></div></div><br></div><div \
class="gmail_extra">It turned up a number of issues that had been lurking with \
passing tests that should have been failing. You may want to be aware if you reuse \
this mechanism too.</div></div>
--0000000000008f6d90056e225e2e--
["TestAxes2.png" (image/png)]
["TestAxes22.png" (image/png)]
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
Search the list archives at: http://markmail.org/search/?q=vtk-developers
Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic