[prev in list] [next in list] [prev in thread] [next in thread]
List: llvm-dev
Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers"
From: David Blaikie via llvm-dev <llvm-dev () lists ! llvm ! org>
Date: 2017-03-31 15:06:11
Message-ID: CAENS6EvwLOh0bhGrO7JQZrYig5g=07GSCB6hk+hwwm8=njp38Q () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
FWIW - a -NOT only test looks especially problematic to me. The test would
pass if the program produced no output, basically - which sounds overly
loose (a shade less problematic than the "run this and don't check anything
- not crashing is all we're testing for here" - which should have some
positive test of the behavior that is expected other than crashing).
So I'd see such a constraint as potentially interesting on that basis - if
it fixes this other issue as well, eh, helpful.
But I'm also OK with it being a thing we document, and every new
contributor eventually learns once... it doesn't seem to have a lot of
overhead.
On Thu, Mar 30, 2017 at 2:50 PM Robinson, Paul via llvm-dev <
llvm-dev@lists.llvm.org> wrote:
> re-add llvm-dev…
>
>
>
> *From:* Robinson, Paul
> *Sent:* Thursday, March 30, 2017 2:50 PM
> *To:* 'Keane, Erich'; Reid Kleckner
> *Subject:* RE: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> I think the most practical thing to do would be to make FileCheck reject
> test cases that consist only CHECK-NOT directives, unless a flag is passed
> to disable this behavior.
>
>
>
> That might help a certain class of tests, but I'm not so sure that's such
> a common case. It's about as easy to write a positive CHECK[-LABEL] that
> will unexpectedly match something that you didn't expect. At least the
> test with only CHECK-NOT in it *will* fail if you get it wrong, which is
> certainly better than passing when it should fail.
>
> So, I am reluctant to encourage yet another wart/hack on how FileCheck
> works.
>
> --paulr
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org
> <llvm-dev-bounces@lists.llvm.org>] *On Behalf Of *Keane, Erich via
> llvm-dev
> *Sent:* Thursday, March 30, 2017 11:18 AM
> *To:* Reid Kleckner
>
>
> *Cc:* llvm-dev@lists.llvm.org
> *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> Alright, I guess it isn't just my pain then, it makes it feel better J I
> think that proposed feature would be really nice, since it would encourage
> people to write tests that have a //CHECK: some-thing-after-header first!
>
>
>
> *From:* Reid Kleckner [mailto:rnk@google.com <rnk@google.com>]
> *Sent:* Thursday, March 30, 2017 11:15 AM
> *To:* Keane, Erich <erich.keane@intel.com>
> *Cc:* llvm-dev@lists.llvm.org
> *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore
> IR-"headers"
>
>
>
> Yeah, this is a documented shortcoming of FileCheck:
>
> http://llvm.org/docs/TestingGuide.html#fragile-tests
>
>
>
> I think the most practical thing to do would be to make FileCheck reject
> test cases that consist only CHECK-NOT directives, unless a flag is passed
> to disable this behavior.
>
>
>
> On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <
> llvm-dev@lists.llvm.org> wrote:
>
> Hi all-
> I hope this is the right place for this.
>
> Anyway, the primary usage for the utility "FileCheck" is to pattern match
> for specific values in a stream. This is perfectly consistent and
> deterministic for the most part! However, when validating clang's LLVM-IR
> generation, it is possible to make an invalid match against the top few
> generated lines (the LLVM-IR Headers), which are inconsistent.
>
> For example (from my personal pain), we have a test:
> clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word
> "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave
> me a false match. I'm a bit ashamed to say how long this took me to figure
> out :)
>
> Anywya, to my feature request: to prevent someone else from going through
> this pain again, would it be possible to have FileCheck ignore these first
> 4 lines as default behavior? I suspect there might be a few tests that
> would want to validate those headers, though I expect those are the vast
> minority, so I would think this should be opt in.
>
> -Erich
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
[Attachment #5 (text/html)]
<div dir="ltr">FWIW - a -NOT only test looks especially problematic to me. The test \
would pass if the program produced no output, basically - which sounds overly loose \
(a shade less problematic than the "run this and don't check anything - not \
crashing is all we're testing for here" - which should have some positive \
test of the behavior that is expected other than crashing).<br><br>So I'd see \
such a constraint as potentially interesting on that basis - if it fixes this other \
issue as well, eh, helpful.<br><br>But I'm also OK with it being a thing we \
document, and every new contributor eventually learns once... it doesn't seem to \
have a lot of overhead.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar \
30, 2017 at 2:50 PM Robinson, Paul via llvm-dev <<a \
href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple" class="gmail_msg">
<div class="m_-7118729956468737877WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg">re-add llvm-dev…<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p> <p class="MsoNormal gmail_msg"><a \
name="m_-7118729956468737877__MailEndCompose" class="gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></a></p> \
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" \
class="gmail_msg"> <div class="gmail_msg">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" \
class="gmail_msg"> <p class="MsoNormal gmail_msg"><b class="gmail_msg"><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" \
class="gmail_msg">From:</span></b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" \
class="gmail_msg"> Robinson, Paul <br class="gmail_msg">
<b class="gmail_msg">Sent:</b> Thursday, March 30, 2017 2:50 PM<br class="gmail_msg">
<b class="gmail_msg">To:</b> 'Keane, Erich'; Reid Kleckner<br \
class="gmail_msg"> <b class="gmail_msg">Subject:</b> RE: [llvm-dev] FileCheck feature \
request- by default ignore IR-"headers"<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p> </div>
</div></div></div></div><div lang="EN-US" link="blue" vlink="purple" \
class="gmail_msg"><div class="m_-7118729956468737877WordSection1 gmail_msg"><div \
style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" \
class="gmail_msg"> <p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u \
class="gmail_msg"></u></p> <p class="MsoNormal gmail_msg" style="margin-left:.5in">I \
think the most practical thing to do would be to make FileCheck reject test cases \
that consist only CHECK-NOT directives, unless a flag is passed to disable this \
behavior.<u class="gmail_msg"></u><u class="gmail_msg"></u></p> <p class="MsoNormal \
gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p> \
</div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div \
class="m_-7118729956468737877WordSection1 gmail_msg"><div \
style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" \
class="gmail_msg"><p class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg">That might help a certain class of tests, but I'm not so sure \
that's such a common case. It's about as easy to write a positive \
CHECK[-LABEL] that will unexpectedly match something that you didn't expect. \
At least the test with only CHECK-NOT in it *will* fail if you get it wrong, which is \
certainly better than passing when it should fail.<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p> <p class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg">So, I am reluctant to encourage yet another wart/hack on how \
FileCheck works.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p> <p \
class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg">--paulr<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p> <p class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p> \
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" \
class="gmail_msg"> <div class="gmail_msg">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" \
class="gmail_msg"> <p class="MsoNormal gmail_msg"><b class="gmail_msg"><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" \
class="gmail_msg">From:</span></b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" \
class="gmail_msg"> llvm-dev [<a href="mailto:llvm-dev-bounces@lists.llvm.org" \
class="gmail_msg" target="_blank">mailto:llvm-dev-bounces@lists.llvm.org</a>] <b \
class="gmail_msg">On Behalf Of </b>Keane, Erich via llvm-dev<br class="gmail_msg"> <b \
class="gmail_msg">Sent:</b> Thursday, March 30, 2017 11:18 AM<br class="gmail_msg"> \
<b class="gmail_msg">To:</b> Reid \
Kleckner</span></p></div></div></div></div></div></div><div lang="EN-US" link="blue" \
vlink="purple" class="gmail_msg"><div class="m_-7118729956468737877WordSection1 \
gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in \
4.0pt" class="gmail_msg"><div style="border:none;border-left:solid blue \
1.5pt;padding:0in 0in 0in 4.0pt" class="gmail_msg"><div class="gmail_msg"><div \
style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" \
class="gmail_msg"><p class="MsoNormal gmail_msg"><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" \
class="gmail_msg"><br class="gmail_msg"> <b class="gmail_msg">Cc:</b> <a \
href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" \
target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg"> <b \
class="gmail_msg">Subject:</b> Re: [llvm-dev] FileCheck feature request- by default \
ignore IR-"headers"<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p></div></div></div></div></div></div><div \
lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div \
class="m_-7118729956468737877WordSection1 gmail_msg"><div \
style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt" \
class="gmail_msg"><div style="border:none;border-left:solid blue 1.5pt;padding:0in \
0in 0in 4.0pt" class="gmail_msg"> <p class="MsoNormal gmail_msg"><u \
class="gmail_msg"></u> <u class="gmail_msg"></u></p> <p class="MsoNormal \
gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg">Alright, I guess it isn't just my pain then, it makes it feel \
better </span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d" \
class="gmail_msg">J</span><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg"> I think that proposed feature would be really nice, since it \
would encourage people to write tests that have a //CHECK: some-thing-after-header \
first!<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p> <p \
class="MsoNormal gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" \
class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p> <p \
class="MsoNormal gmail_msg"><b class="gmail_msg"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif"" \
class="gmail_msg">From:</span></b><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif"" \
class="gmail_msg"> Reid Kleckner [<a href="mailto:rnk@google.com" class="gmail_msg" \
target="_blank">mailto:rnk@google.com</a>] <br class="gmail_msg">
<b class="gmail_msg">Sent:</b> Thursday, March 30, 2017 11:15 AM<br \
class="gmail_msg"> <b class="gmail_msg">To:</b> Keane, Erich <<a \
href="mailto:erich.keane@intel.com" class="gmail_msg" \
target="_blank">erich.keane@intel.com</a>><br class="gmail_msg"> <b \
class="gmail_msg">Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" \
target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg"> <b \
class="gmail_msg">Subject:</b> Re: [llvm-dev] FileCheck feature request- by default \
ignore IR-"headers"<u class="gmail_msg"></u><u \
class="gmail_msg"></u></span></p> <p class="MsoNormal gmail_msg"><u \
class="gmail_msg"></u> <u class="gmail_msg"></u></p> <div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Yeah, this is a documented shortcoming of FileCheck:<u \
class="gmail_msg"></u><u class="gmail_msg"></u></p> </div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><a \
href="http://llvm.org/docs/TestingGuide.html#fragile-tests" class="gmail_msg" \
target="_blank">http://llvm.org/docs/TestingGuide.html#fragile-tests</a><u \
class="gmail_msg"></u><u class="gmail_msg"></u></p> </div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u \
class="gmail_msg"></u></p> </div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">I think the most practical thing to do would be to \
make FileCheck reject test cases that consist only CHECK-NOT directives, unless a \
flag is passed to disable this behavior.<u class="gmail_msg"></u><u \
class="gmail_msg"></u></p> </div>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u \
class="gmail_msg"></u></p> <div class="gmail_msg">
<p class="MsoNormal gmail_msg">On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via \
llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" \
target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u class="gmail_msg"></u><u \
class="gmail_msg"></u></p> <blockquote style="border:none;border-left:solid #cccccc \
1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" \
class="gmail_msg"> <p class="MsoNormal gmail_msg">Hi all-<br class="gmail_msg">
I hope this is the right place for this.<br class="gmail_msg">
<br class="gmail_msg">
Anyway, the primary usage for the utility "FileCheck" is to pattern match \
for specific values in a stream. This is perfectly consistent and deterministic for \
the most part! However, when validating clang's LLVM-IR generation, it is \
possible to make an invalid match against the top few generated lines (the LLVM-IR \
Headers), which are inconsistent.<br class="gmail_msg"> <br class="gmail_msg">
For example (from my personal pain), we have a test: \
clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word \
"macros". Unfortunately, my SVN workspace name was "ms_macros", \
which gave me a false match. I'm a bit ashamed to say how long this took me to \
figure out :)<br class="gmail_msg"> <br class="gmail_msg">
Anywya, to my feature request: to prevent someone else from going through this pain \
again, would it be possible to have FileCheck ignore these first 4 lines as default \
behavior? I suspect there might be a few tests that would want to validate those \
headers, though I expect those are the vast minority, so I would think this should \
be opt in.<br class="gmail_msg"> <br class="gmail_msg">
-Erich<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
LLVM Developers mailing list<br class="gmail_msg">
<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" \
target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg"> <a \
href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="gmail_msg" \
target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u \
class="gmail_msg"></u><u class="gmail_msg"></u></p> </blockquote>
</div>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u \
class="gmail_msg"></u></p> </div>
</div></div></div></div>
_______________________________________________<br class="gmail_msg">
LLVM Developers mailing list<br class="gmail_msg">
<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" \
target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg"> <a \
href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" \
class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br \
class="gmail_msg"> </blockquote></div>
[Attachment #6 (text/plain)]
_______________________________________________
LLVM Developers mailing list
llvm-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic