[prev in list] [next in list] [prev in thread] [next in thread]
List: cfe-dev
Subject: Re: [cfe-dev] libclang: failing assert
From: Erik Verbruggen <erik.verbruggen () me ! com>
Date: 2013-03-19 13:56:05
Message-ID: 2C87CDED-D5CA-4C31-B956-AD7AF932C9DC () me ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Mar 19, 2013, at 0:07, Argyrios Kyrtzidis <kyrtzidis@apple.com> wrote:
> On Mar 17, 2013, at 8:35 AM, Erik Verbruggen <erik.verbruggen@me.com> wrote:
>
> >
> > On Mar 16, 2013, at 18:56, Argyrios Kyrtzidis <kyrtzidis@apple.com> wrote:
> >
> > >
> > > On Mar 16, 2013, at 10:48 AM, Argyrios Kyrtzidis <kyrtzidis@apple.com> wrote:
> > >
> > > > On Mar 16, 2013, at 3:09 AM, Erik Verbruggen <erik.verbruggen@me.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > When using libclang's clang_annotateTokens, I have a consistently failing \
> > > > > assert in CIndex.cpp, line 5429. This is for both 3.2 and trunk. When I \
> > > > > comment the assert out, everything looks to be fine. Is this assert valid, \
> > > > > and if so, what is it checking?
> > > >
> > > > It makes sure that when lexing for preprocessor directives tokens, that any \
> > > > cursors produced will refer to the correct clang_tokenize'd token. Are you \
> > > > able to provide a preprocessed file for testing ?
> > >
> > > You can use "-E -frewrite-includes" to preserve preprocessor directives.
> >
> > See attachment.
>
> I can't reproduce using:
>
> $ c-index-test -test-annotate-tokens=main.pp:1:1:100000:1 -x c++ main.pp > \
> /dev/null
> Does it occur with a particular source range ? What is the file whose tokens you \
> are annotating ? Does using c-index-test on it reproduce the assertion hit ? If \
> yes, could you separate the includes of that file, for example:
> #include "a.h"
> #include "b.h"
> #include "c.h"
>
> <code>
>
> ====>
>
> #include "preprocessed-headers.h"
>
> <code>
I cannot reproduce it either with c-index-test, so I modified it to do it in the same \
way as I use the annotation. Patch is attached. With that patch, I get:
erik@Road-Warrior:untitled6$ ~/dev/builds/llvm-release-build/bin/c-index-test \
-test-annotate-tokens=main.pp:62024:1:1000000:1 -std=gnu++98 -x c++ main.pp Assertion \
failed: (Tok.getLocation() == \
SourceLocation::getFromRawEncoding(Tokens[TokIdx].int_data[1])), function \
annotatePreprocessorTokens, file \
/Users/erik/dev/clang-llvm/llvm-git/tools/clang/tools/libclang/CIndex.cpp, \
line 5429.
libclang: crash detected while annotating tokens
Identifier: "std" [62025:17 - 62025:20]
Identifier: "main" [62027:5 - 62027:9]
Identifier: "cout" [62029:5 - 62029:9]
Identifier: "endl" [62029:31 - 62029:35]
When I move everything up to (and including) line 62022 into the a header file (like \
you mentioned above), and include that file, I get exactly the same crash.
[Attachment #5 (multipart/mixed)]
[Attachment #7 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space; "><br><div><div>On Mar 19, 2013, at \
0:07, Argyrios Kyrtzidis <<a \
href="mailto:kyrtzidis@apple.com">kyrtzidis@apple.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><blockquote type="cite"><meta \
http-equiv="Content-Type" content="text/html charset=us-ascii"><div style="word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">On Mar \
17, 2013, at 8:35 AM, Erik Verbruggen <<a \
href="mailto:erik.verbruggen@me.com">erik.verbruggen@me.com</a>> \
wrote:<br><div><br class="Apple-interchange-newline"><blockquote type="cite"><div \
style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><br>On Mar 16, 2013, \
at 18:56, Argyrios Kyrtzidis <<a \
href="mailto:kyrtzidis@apple.com">kyrtzidis@apple.com</a>> \
wrote:<br><br><blockquote type="cite"><br>On Mar 16, 2013, at 10:48 AM, Argyrios \
Kyrtzidis <<a href="mailto:kyrtzidis@apple.com">kyrtzidis@apple.com</a>> \
wrote:<br><br><blockquote type="cite">On Mar 16, 2013, at 3:09 AM, Erik Verbruggen \
<<a href="mailto:erik.verbruggen@me.com">erik.verbruggen@me.com</a>> \
wrote:<br><br><blockquote type="cite">Hi,<br><br>When using libclang's \
clang_annotateTokens, I have a consistently failing assert in CIndex.cpp, line 5429. \
This is for both 3.2 and trunk. When I comment the assert out, everything looks to be \
fine. Is this assert valid, and if so, what is it checking?<span \
class="Apple-converted-space"> </span><br></blockquote><br>It makes sure that \
when lexing for preprocessor directives tokens, that any cursors produced will refer \
to the correct clang_tokenize'd token.<br>Are you able to provide a preprocessed file \
for testing ?<br><br></blockquote><br>You can use "-E -frewrite-includes" to preserve \
preprocessor directives.<br></blockquote><br>See \
attachment.<br></div></blockquote><div><br></div><div>I can't reproduce \
using:</div><div><br></div><div>$ c-index-test \
-test-annotate-tokens=main.pp:1:1:100000:1 -x c++ main.pp > \
/dev/null</div><div><br></div><div>Does it occur with a particular source range ? \
What is the file whose tokens you are annotating ? Does using c-index-test on it \
reproduce the assertion hit ?</div><div>If yes, could you separate the includes of \
that file, for example:</div><div><br></div><div>#include "a.h"</div>#include \
"b.h"</div><div>#include \
"c.h"</div><div><br></div><div><code></div><div><br></div><div>====></div><div><br></div><div>#include \
"preprocessed-headers.h"</div><div><br></div><div><code></div></div></blockquote><br></div><div>I \
cannot reproduce it either with c-index-test, so I modified it to do it in the same \
way as I use the annotation. Patch is attached. With that patch, I \
get:</div><div><br></div><div><div><div>erik@Road-Warrior:untitled6$ \
~/dev/builds/llvm-release-build/bin/c-index-test \
-test-annotate-tokens=main.pp:62024:1:1000000:1 -std=gnu++98 -x c++ \
main.pp</div><div>Assertion failed: (Tok.getLocation() == \
SourceLocation::getFromRawEncoding(Tokens[TokIdx].int_data[1])), function \
annotatePreprocessorTokens, file \
/Users/erik/dev/clang-llvm/llvm-git/tools/clang/tools/libclang/CIndex.cpp, line \
5429.</div><div>libclang: crash detected while annotating \
tokens</div><div>Identifier: "std" [62025:17 - 62025:20]</div><div>Identifier: "main" \
[62027:5 - 62027:9]</div><div>Identifier: "cout" [62029:5 - \
62029:9]</div><div>Identifier: "endl" [62029:31 - \
62029:35]</div><div><br></div></div><div>When I move everything up to (and including) \
line 62022 into the a header file (like you mentioned above), and include that file, \
I get exactly the same crash.</div><div><br></div><div></div></div></body></html>
["my-c-indexing.patch" (my-c-indexing.patch)]
diff --git a/tools/c-index-test/c-index-test.c b/tools/c-index-test/c-index-test.c
index 88b49ed..e644554 100644
--- a/tools/c-index-test/c-index-test.c
+++ b/tools/c-index-test/c-index-test.c
@@ -2987,7 +2987,8 @@ int perform_token_annotation(int argc, const char **argv) {
struct CXUnsavedFile *unsaved_files = 0;
int num_unsaved_files = 0;
CXToken *tokens;
- unsigned num_tokens;
+ CXToken *idTokens;
+ unsigned num_tokens, num_id_tokens;
CXSourceRange range;
CXSourceLocation startLoc, endLoc;
CXFile file = 0;
@@ -3072,21 +3073,30 @@ int perform_token_annotation(int argc, const char **argv) {
goto teardown;
}
+ num_id_tokens = 0;
+ idTokens = (CXToken *)malloc(num_tokens * sizeof(CXToken));
+ for (i = 0; i != num_tokens; ++i) {
+ if (CXToken_Identifier == clang_getTokenKind(tokens[i])) {
+ idTokens[num_id_tokens] = tokens[i];
+ ++num_id_tokens;
+ }
+ }
+
cursors = (CXCursor *)malloc(num_tokens * sizeof(CXCursor));
- clang_annotateTokens(TU, tokens, num_tokens, cursors);
+ clang_annotateTokens(TU, idTokens, num_id_tokens, cursors);
if (checkForErrors(TU) != 0) {
errorCode = -1;
goto teardown;
}
- for (i = 0; i != num_tokens; ++i) {
+ for (i = 0; i != num_id_tokens; ++i) {
const char *kind = "<unknown>";
- CXString spelling = clang_getTokenSpelling(TU, tokens[i]);
- CXSourceRange extent = clang_getTokenExtent(TU, tokens[i]);
+ CXString spelling = clang_getTokenSpelling(TU, idTokens[i]);
+ CXSourceRange extent = clang_getTokenExtent(TU, idTokens[i]);
unsigned start_line, start_column, end_line, end_column;
- switch (clang_getTokenKind(tokens[i])) {
+ switch (clang_getTokenKind(idTokens[i])) {
case CXToken_Punctuation: kind = "Punctuation"; break;
case CXToken_Keyword: kind = "Keyword"; break;
case CXToken_Identifier: kind = "Identifier"; break;
[Attachment #9 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space; "><div><div></div></div></body></html>
_______________________________________________
cfe-dev mailing list
cfe-dev@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic