[prev in list] [next in list] [prev in thread] [next in thread]
List: sonar-dev
Subject: Re: [sonar-dev] Where can i find an Intruction to SSLR?
From: patschius <c.patscheider () gmail ! com>
Date: 2014-09-16 8:56:41
Message-ID: CAP_0_rwR=nfgEMmuw7XJFa+s1hGOUkyuwc+DXEkchzbpBgOcdA () mail ! gmail ! com
[Download RAW message or body]
Hi Dinesh,
Thanks again for the hint. I managed to get the data by adding this line to
my FileLineVistor.leaveFile:
getContext().peekSourceCode().add(PLSQLMetric.COMMENT_LINES,
linesOfComments.size());
And add the saveMeasure in the Squid:
context.saveMeasure(sonarFile, CoreMetrics.COMMENT_LINES,
squidFile.getDouble(PLSQLMetric.COMMENT_LINES));
Now it works:
[image: Inline image 1]
Yes i am trying to write a PL-SQL plugin and i have already discovered the
commercial plugin, but my company says no to the price :D
But since i think its highly important to make the code quality visible and
we have nothing the like, i figured i just give it a try in my own time.
So i reached my first goal now :)
I will check where i can take it from here.
Huge thanks for your help so far. I hope you dont mind if i post questions
that I encounter along the way in this thread.
Best regards,
Christoph
On Tue, Sep 16, 2014 at 10:36 AM, Dinesh Bolkensteyn-2 [via SonarQube] <
ml-node+s15n5028278h20@n6.nabble.com> wrote:
> The metrics CoreMetrics.NCLOC_DATA_KEY and CoreMetrics.COMMENT_LINES_DATA_KEY
> are used by the developer cockpit plugin.
>
> You will additionally need to feed: CoreMetrics.COMMENT_LINES.
>
> In the Python plugin, this is done here:
> https://github.com/SonarCommunity/sonar-python/blob/master/sonar-python-plugin/src/main/java/org/sonar/plugins/python/PythonSquidSensor.java#L115
>
> By the way, are you rewriting a PL/SQL plugin?
> There is already one available commercially,
> http://www.sonarsource.com/products/plugins/languages/plsql/ , whose
> development has been ongoing for several years already.
>
>
> --
> Dinesh Bolkensteyn
> www.SonarSource.com <http://www.sonarsource.com/>
> twitter.com/DBolkensteyn <http://www.SonarSource.com>
>
> On Tue, Sep 16, 2014 at 10:17 AM, patschius <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=5028278&i=0>> wrote:
>
> > Hi Dinesh,
> >
> > Thanks a lot for your help. I hope i may bother you (or anybody else who
> > wants to help) a little further along the way.
> >
> > I have managed to compute the lines of code and NLOC. Also in my Unit
> > test the comments get recognized.
> > However when i run some source files through the plugin in Sonar the LOC
> > and NLOC is correctly displayed.
> > But i see no information in the comment widget:
> > [image: Inline image 1]
> > (It says Python because i took the Python plugin as skeleton and tried to
> > strip it down and convert it to what i want :) )
> >
> > So this is what i have (most of it just like you suggested):
> >
> > A Lexer with:
> >
> > .withChannel(new BlackHoleChannel("\\s"))
> > .withChannel(regexp(GenericTokenType.COMMENT, "--[^\\n\\r]*+"))
> > .withChannel(regexp(GenericTokenType.COMMENT, "/\\*[\\s\\S]*?\\*/"))
> > .withChannel(new UnknownCharacterChannel())
> >
> > A Grammar with:
> >
> > b.rule(FILE_INPUT).is
> >
> > b.zeroOrMore(b.anyTokenButNot(GenericTokenType.EOF)), // Consume all
> > tokens types except EOF
> >
> > GenericTokenType.EOF); // Expect an end of file
> >
> > A FileLineVisitor:
> >
> > public void visitToken(Token token) {
> >
> > if (token.getType().equals(GenericTokenType.EOF)) {
> >
> > return;
> >
> > }
> >
> > if(token.getType() == GenericTokenType.UNKNOWN_CHAR){
> >
> > linesOfCode.add(token.getLine());
> >
> > }
> >
> > if(token.getType() == GenericTokenType.COMMENT){
> >
> > //System.out.println("Comment good: " + token.getLine());
> >
> > linesOfComments.add(token.getLine());
> >
> > }
> > }
> > @Override
> > public void leaveFile(AstNode astNode) {
> > File sonarFile = File.fromIOFile(getContext().getFile(), project);
> > FileLinesContext fileLinesContext =
> > fileLinesContextFactory.createFor(sonarFile);
> >
> > int fileLength =
> > getContext().peekSourceCode().getInt(PLSQLMetric.LINES);
> > for (int line = 1; line <= fileLength; line++) {
> > fileLinesContext.setIntValue(CoreMetrics.NCLOC_DATA_KEY, line,
> > linesOfCode.contains(line) ? 1 : 0);
> > fileLinesContext.setIntValue(CoreMetrics.COMMENT_LINES_DATA_KEY,
> > line, linesOfComments.contains(line) ? 1 : 0);
> > }
> > fileLinesContext.save();
> >
> > linesOfCode.clear();
> > linesOfComments.clear();
> > }
> >
> >
> >
> > By the way is there anywhere some documentation for this regex syntax?
> > Its not really what i am used to.
> >
> > I guess it hase something to do with the fact that this example with
> > COMMENT_LINES_DATA_KEY i took from a language which uses a commentRegexp in
> > the Channel?
> > May that be the case?
> >
> > Again thanks a lot for your help (and for your patience).
> >
> > Best regards,
> > Christoph
> >
> > On Tue, Sep 16, 2014 at 7:43 AM, Dinesh Bolkensteyn-2 [via SonarQube] <[hidden
> > email] <http://user/SendEmail.jtp?type=node&node=5028276&i=0>> wrote:
> >
> > > Hi Christoph!
> > >
> > > First of you, thank you for your interest in SonarQube and SSLR!
> > >
> > > You're definitely on the right track:
> > >
> > > 1. All you need is a lexer to recognize comments, whitespaces (which
> > > will not count towards LOC), and everything else
> > > 2. You indeed do not need a grammar
> > >
> > > So let's have a look at MiniC's lexer (and test):
> > >
> > >
> > > https://github.com/SonarSource/sslr/blob/master/sslr-testing-harness/src/main/java/com/sonar/sslr/test/minic/MiniCLexer.java
> > >
> > > https://github.com/SonarSource/sslr/blob/master/sslr-tests/src/test/java/com/sonar/sslr/test/minic/MiniCLexerTest.java
> > >
> > > A lexer splits the original file (a string), into tokens. A token has a
> > > type, and a value. If you merge the values of all tokens, you get the
> > > original file contents back.
> > >
> > > If your case, you need 3 token types: One for comments, one for
> > > whitespaces (including new lines), and one for everything else.
> > > But, in most programming languages, whitespaces are not significant, and
> > > you can ask the lexer to filter them out of its output.
> > >
> > > So, this means you really need just 2 types.
> > >
> > > You can either define your own (as it is done in MiniCLexer) by
> > > implementing TokenType (and the required boilerplate code), or use
> > > predefined ones from GenericTokenType:
> > >
> > > https://github.com/SonarSource/sslr/blob/master/sslr-core/src/main/java/com/sonar/sslr/api/GenericTokenType.java
> > >
> > > It seems to me that, in your case, GenericTokenType.COMMENT and
> > > GenericTokenType.UNKNOWN_CHAR will do the job.
> > >
> > > Then, you need to set up 3 channels. Channels are evaluated in their
> > > listed order, and the first one that matches will "win" and create a token.
> > > So, if your case, you want:
> > >
> > > 1. A RegexpChannel channel to lex your comments (don't use
> > > CommentRegexpChannel for the time being, as this will place comment tokens
> > > as "trivias" on other tokens, instead of producing individual comment
> > > tokens)
> > >
> > > 2. A BlackHoleChannel channel to lex whitespaces and new lines. This
> > > type of channel is similar to RegexpChannel, but matched tokens will be
> > > filtered out of the lexer output. This is why you do not need to specify a
> > > type, but only a regular expression.
> > >
> > > 3. Finally, a UnknownCharacterChannel channel to match all other
> > > characters as individual tokens of type GenericTokenType.UNKNOWN_CHAR.
> > >
> > > Note that SSLR will create an artificial token at the very end to
> > > indicate the end of the file, of type GenericTokenType.EOF.
> > >
> > > Once you have this is place, you can already develop a SonarQube Sensor,
> > > iterate over all source files of the ModuleFileSystem, run your lexer, and
> > > save the metrics.
> > >
> > > Then, you can add a basic grammar, that will simply match every file:
> > >
> > > b.rule(COMPILATION_UNIT).is(
> > > b.zeroOrMore(b.anyTokenBut(GenericTokenType.EOF)), // Consume all
> > > tokens types except EOF
> > > GenericTokenType.EOF); // Expect an end of file
> > >
> > > The MiniCGrammar shows how a more realistic grammar can be implemented,
> > > and there also are a few more minimalistic ones available as examples in
> > > the "sslr-examples" module of SSLR:
> > >
> > > https://github.com/SonarSource/sslr/tree/master/sslr-examples/src/main/java/org/sonar/sslr/examples/grammars
> > > (note that most of those examples, unlike MiniC, use the "lexerless"
> > > grammar builder, which is going to be slightly different from the
> > > "lexerful" grammar builder)
> > >
> > > I hope it makes more sense now.
> > >
> > > The best way to proceed is to play with writing a basic grammar, for
> > > instance for arithmetic expressions.
> > >
> > > You can also quickly create an SSLR Toolkit to visualize the parse trees
> > > for various inputs, or use AstXmlPrinter.print() to print a parse tree to a
> > > string for debugging/learning purposes.
> > >
> > > Kind regards,
> > >
> > > --
> > > Dinesh Bolkensteyn
> > > www.SonarSource.com <http://www.sonarsource.com/>
> > > twitter.com/DBolkensteyn <http://www.SonarSource.com>
> > >
> > > On Mon, Sep 15, 2014 at 11:13 PM, patschius <[hidden email]
> > > <http://user/SendEmail.jtp?type=node&node=5028268&i=0>> wrote:
> > >
> > > > Hi All,
> > > >
> > > > First of all i am new to Sonar as a plugin developer, but i was a Sonar
> > > > User
> > > > before that and really like it!
> > > >
> > > > So when starting the development i have mastered the easy part and made
> > > > Sonar recognize and install my Plugin as well as to detect the newly
> > > > defined
> > > > language and so on.
> > > >
> > > > As a starting goal i have set for myself that i just want some simple
> > > > metrics about Files, LOC and Comments. This stuff is almost language
> > > > independent. If my thoughts are correct i don't need any sophisticated
> > > > grammar for this purpose. Just a channel to recognize the comments and
> > > > ...
> > > > something else...
> > > >
> > > > And this is where got stuck. I understand (or at least i think i do) the
> > > > principles of PEG and so on, but somehow i fail to put it into context
> > > > with
> > > > SSLR. So i am a bit confused how the Channels defined in the Lexer and
> > > > the
> > > > Grammar actually work together. Also the Methods in the GrammarBuilder
> > > > do
> > > > not make that much sense to me, although i have read their javadoc back
> > > > and
> > > > forth, but its not really verbose. I also looked at other languages
> > > > that use
> > > > SSLR, but they are already so far progressed that it is kind of hard to
> > > > figure out the simple connections.
> > > >
> > > > So what i would need is like a step by step tutorial that explains how
> > > > to
> > > > build a simple grammar from scratch (I also looked at the
> > > > MiniCGrammar...).
> > > > Is there any chance such a thing exists somewhere or can someone point
> > > > me to
> > > > where i could gather such information?
> > > >
> > > > Thanks a lot for your help in advance.
> > > >
> > > > Best Regards,
> > > > Christoph Patscheider
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > > http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267.html
> > > > Sent from the SonarQube Developers mailing list archive at Nabble.com.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe from this list, please visit:
> > > >
> > > > http://xircles.codehaus.org/manage_email
> > > >
> > > >
> > > >
> > >
> > >
> > > ------------------------------
> > > If you reply to this email, your message will be added to the
> > > discussion below:
> > >
> > > http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028268.html
> > > To unsubscribe from Where can i find an Intruction to SSLR?, click here
> > > .
> > > NAML
> > > <http://sonarqube.15.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&i \
> > > d=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace- \
> > > nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace& \
> > > breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >
> >
> >
> > ------------------------------
> > View this message in context: Re: [sonar-dev] Where can i find an
> > Intruction to SSLR?
> > <http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028276.html>
> >
> > Sent from the SonarQube Developers mailing list archive
> > <http://sonarqube.15.x6.nabble.com/SonarQube-Developers-f4523654.html>
> > at Nabble.com.
> >
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028278.html
> To unsubscribe from Where can i find an Intruction to SSLR?, click here
> <http://sonarqube.15.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5028267&code=Yy5wYXRzY2hlaWRlckBnbWFpbC5jb218NTAyODI2N3w5NjE5NTY1NzQ=>
>
> .
> NAML
> <http://sonarqube.15.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=in \
> stant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.v \
> iew.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs= \
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
image.png (36K) <http://sonarqube.15.x6.nabble.com/attachment/5028280/0/image.png>
--
View this message in context: \
http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028280.html
Sent from the SonarQube Developers mailing list archive at Nabble.com.
[Attachment #3 (text/html)]
<div dir="ltr">Hi Dinesh,<div><br></div><div>Thanks again for the hint. I managed to \
get the data by adding this line to my \
FileLineVistor.leaveFile:</div><div>getContext().peekSourceCode().add(PLSQLMetric.COMMENT_LINES, \
linesOfComments.size());<br></div><div>And add the saveMeasure in the \
Squid:</div><div>context.saveMeasure(sonarFile, CoreMetrics.COMMENT_LINES, \
squidFile.getDouble(PLSQLMetric.COMMENT_LINES));<br></div><div><br></div><div>Now it \
works:</div><div><img \
src="http://sonarqube.15.x6.nabble.com/attachment/5028280/0/image.png" alt="Inline \
image 1" width="562" height="52"><br></div><div><br></div><div>Yes i am trying to \
write a PL-SQL plugin and i have already discovered the commercial plugin, but my \
company says no to the price :D</div><div>But since i think its highly important to \
make the code quality visible and we have nothing the like, i figured i just give it \
a try in my own time.</div><div><br></div><div>So i reached my first goal now \
:)</div><div><br></div><div>I will check where i can take it from \
here.</div><div>Huge thanks for your help so far. I hope you dont mind if i post \
questions that I encounter along the way in this \
thread.</div><div><br></div><div>Best regards,</div><div>Christoph</div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 16, 2014 at 10:36 AM, \
Dinesh Bolkensteyn-2 [via SonarQube] <span dir="ltr"><<a \
href="/user/SendEmail.jtp?type=node&node=5028280&i=0" target="_top" rel="nofollow" \
link="external">[hidden email]</a>></span> wrote:<br><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">The metrics CoreMetrics.NCLOC_DATA_KEY and <span \
style="font-family:arial,sans-serif;font-size:13px">CoreMetrics.COMMENT_LINES_</span><span \
style="font-family:arial,sans-serif;font-size:13px">DATA_KEY are used by the \
developer cockpit plugin.</span><div><span \
style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span \
style="font-family:arial,sans-serif;font-size:13px">You will additionally need to \
feed: </span>CoreMetrics.COMMENT_LINES.</div><div><br></div><div>In the Python \
plugin, this is done here: <a \
href="https://github.com/SonarCommunity/sonar-python/blob/master/sonar-python-plugin/src/main/java/org/sonar/plugins/python/PythonSquidSensor.java#L115" \
rel="nofollow" link="external" \
target="_blank">https://github.com/SonarCommunity/sonar-python/blob/master/sonar-pytho \
n-plugin/src/main/java/org/sonar/plugins/python/PythonSquidSensor.java#L115</a></div><div><br></div><div>By \
the way, are you rewriting a PL/SQL plugin?</div><div>There is already one available \
commercially, <a href="http://www.sonarsource.com/products/plugins/languages/plsql/" \
rel="nofollow" link="external" \
target="_blank">http://www.sonarsource.com/products/plugins/languages/plsql/</a> , \
whose development has been ongoing for several years already.</div></div><div \
class="gmail_extra"><span class=""><br clear="all"><div><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><br>--<br>Dinesh \
Bolkensteyn<br></span><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><a \
href="http://www.sonarsource.com/" rel="nofollow" link="external" \
target="_blank">www.SonarSource.com</a></span><br><a \
href="http://twitter.com/DBolkensteyn" rel="nofollow" link="external" \
target="_blank">twitter.com/DBolkensteyn</a><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"></span><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><a \
href="http://www.SonarSource.com" rel="nofollow" link="external" \
target="_blank"></a></span><br></div> <br></span><div class="gmail_quote"><div><div \
class="h5">On Tue, Sep 16, 2014 at 10:17 AM, patschius <span dir="ltr"><<a \
href="http://user/SendEmail.jtp?type=node&node=5028278&i=0" rel="nofollow" \
link="external" target="_blank">[hidden email]</a>></span> \
wrote:<br></div></div><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em" class="gmail_quote"><div><div \
class="h5"><div dir="ltr">Hi Dinesh,<div><br></div><div>Thanks a lot for your help. I \
hope i may bother you (or anybody else who wants to help) a little further along the \
way.</div><div><br></div><div>I have managed to compute the lines of code and NLOC. \
Also in my Unit test the comments get recognized.</div><div>However when i run some \
source files through the plugin in Sonar the LOC and NLOC is correctly \
displayed.</div><div>But i see no information in the comment widget:</div><div><img \
src="http://sonarqube.15.x6.nabble.com/attachment/5028276/0/image.png" alt="Inline \
image 1" width="562" height="84"><br></div><div>(It says Python because i took the \
Python plugin as skeleton and tried to strip it down and convert it to what i want :) \
)</div><div><br></div><div>So this is what i have (most of it just like you \
suggested):</div><div><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"><div>A Lexer \
with:</div></blockquote></div><blockquote style='border-left:2px solid \
#CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 \
1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"><div>.withChannel(new \
BlackHoleChannel("\\s")) \
</div><div>.withChannel(regexp(GenericTokenType.COMMENT, "--[^\\n\\r]*+")) \
</div><div> .withChannel(regexp(GenericTokenType.COMMENT, \
"/\\*[\\s\\S]*?\\*/"))</div><div>.withChannel(new \
UnknownCharacterChannel())</div></blockquote></blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"><div>A Grammar with:</div></blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 \
1em' style="border-left:2px solid #cccccc;padding:0 1em"> \
b.rule(FILE_INPUT).is<br></blockquote></blockquote><blockquote style='border-left:2px \
solid #CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 \
1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em">b.zeroOrMore(b.anyTokenButNot(GenericTokenType.EOF)), // \
Consume all tokens types except \
EOF</blockquote></blockquote></blockquote><span><blockquote style='border-left:2px \
solid #CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 \
1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em">GenericTokenType.EOF); // Expect an end of \
file<br></blockquote></blockquote></blockquote></span><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em">A FileLineVisitor:<br></blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 \
1em' style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"> public void visitToken(Token token) \
{</blockquote></blockquote></blockquote><blockquote style='border-left:2px solid \
#CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 \
1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"> if (token.getType().equals(GenericTokenType.EOF)) \
{</blockquote><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"> \
return;</blockquote><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"> } \
</blockquote><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em"> if(token.getType() == \
GenericTokenType.UNKNOWN_CHAR){</blockquote><blockquote style='border-left:2px solid \
#CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 1em"> \
<span style="white-space:pre-wrap"> </span>linesOfCode.add(token.getLine());</blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"> }</blockquote><blockquote style='border-left:2px solid \
#CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 1em"> \
if(token.getType() == GenericTokenType.COMMENT){</blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"> <span \
style="white-space:pre-wrap"> </span>//System.out.println("Comment good: " \
+ token.getLine());</blockquote><blockquote style='border-left:2px solid \
#CCCCCC;padding:0 1em' style="border-left:2px solid #cccccc;padding:0 1em"> \
<span style="white-space:pre-wrap"> </span>linesOfComments.add(token.getLine());</blockquote></blockquote></blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 \
1em' style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"> }<br>}<br> @Override<br> public void \
leaveFile(AstNode astNode) {<br> File sonarFile = \
File.fromIOFile(getContext().getFile(), project);<br> FileLinesContext \
fileLinesContext = fileLinesContextFactory.createFor(sonarFile);<br><br> int \
fileLength = getContext().peekSourceCode().getInt(PLSQLMetric.LINES);<br> for \
(int line = 1; line <= fileLength; line++) {<br> \
fileLinesContext.setIntValue(CoreMetrics.NCLOC_DATA_KEY, line, \
linesOfCode.contains(line) ? 1 : 0);<br> \
fileLinesContext.setIntValue(CoreMetrics.COMMENT_LINES_DATA_KEY, line, \
linesOfComments.contains(line) ? 1 : 0);<br> }<br> \
fileLinesContext.save();<br><br> linesOfCode.clear();<br> \
linesOfComments.clear();<br> }</blockquote></blockquote></blockquote><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"><blockquote style='border-left:2px solid #CCCCCC;padding:0 \
1em' style="border-left:2px solid #cccccc;padding:0 1em"><blockquote \
style='border-left:2px solid #CCCCCC;padding:0 1em' style="border-left:2px solid \
#cccccc;padding:0 1em"></blockquote></blockquote></blockquote><div><br></div><div><br></div><div>By \
the way is there anywhere some documentation for this regex syntax? </div><div>Its \
not really what i am used to.</div><div><br></div><div>I guess it hase something to \
do with the fact that this example with COMMENT_LINES_DATA_KEY i took from a language \
which uses a commentRegexp in the Channel?</div><div>May that be the \
case?</div><div><br></div><div>Again thanks a lot for your help (and for your \
patience).</div><div><br></div><div>Best \
regards,</div><div>Christoph</div></div></div></div><div><div class="h5"><div \
class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, Sep 16, 2014 at \
7:43 AM, Dinesh Bolkensteyn-2 [via SonarQube] <span dir="ltr"><<a \
href="http://user/SendEmail.jtp?type=node&node=5028276&i=0" rel="nofollow" \
link="external" target="_blank">[hidden email]</a>></span> \
wrote:<br></div></div><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em" class="gmail_quote">
<div dir="ltr"><div><div>Hi Christoph!<div><br></div><div>First of you, thank you \
for your interest in SonarQube and SSLR!</div><div><br></div><div>You're \
definitely on the right track:</div><div><br></div><div> 1. All you need is a lexer \
to recognize comments, whitespaces (which will not count towards LOC), and everything \
else</div><div> 2. You indeed do not need a grammar</div><div><br></div><div>So \
let's have a look at MiniC's lexer (and test):</div><div><br></div><div> \
<a href="https://github.com/SonarSource/sslr/blob/master/sslr-testing-harness/src/main/java/com/sonar/sslr/test/minic/MiniCLexer.java" \
rel="nofollow" link="external" \
target="_blank">https://github.com/SonarSource/sslr/blob/master/sslr-testing-harness/src/main/java/com/sonar/sslr/test/minic/MiniCLexer.java</a></div><div> \
<a href="https://github.com/SonarSource/sslr/blob/master/sslr-tests/src/test/java/com/sonar/sslr/test/minic/MiniCLexerTest.java" \
rel="nofollow" link="external" \
target="_blank">https://github.com/SonarSource/sslr/blob/master/sslr-tests/src/test/java/com/sonar/sslr/test/minic/MiniCLexerTest.java</a></div><div><br></div><div>A \
lexer splits the original file (a string), into tokens. A token has a type, and a \
value. If you merge the values of all tokens, you get the original file contents \
back.</div><div><br></div><div>If your case, you need 3 token types: One for \
comments, one for whitespaces (including new lines), and one for everything \
else.</div><div>But, in most programming languages, whitespaces are not significant, \
and you can ask the lexer to filter them out of its \
output.</div><div><br></div><div>So, this means you really need just 2 \
types.</div><div><br></div><div>You can either define your own (as it is done in \
MiniCLexer) by implementing TokenType (and the required boilerplate code), or use \
predefined ones from GenericTokenType:</div><div> <a \
href="https://github.com/SonarSource/sslr/blob/master/sslr-core/src/main/java/com/sonar/sslr/api/GenericTokenType.java" \
rel="nofollow" link="external" \
target="_blank">https://github.com/SonarSource/sslr/blob/master/sslr-core/src/main/java/com/sonar/sslr/api/GenericTokenType.java</a></div><div><br></div><div>It \
seems to me that, in your case, GenericTokenType.COMMENT and \
GenericTokenType.UNKNOWN_CHAR will do the job.</div><div><br></div><div>Then, you \
need to set up 3 channels. Channels are evaluated in their listed order, and the \
first one that matches will "win" and create a token. So, if your case, you \
want:</div><div><br></div><div>1. A RegexpChannel channel to lex your comments \
(don't use CommentRegexpChannel for the time being, as this will place comment \
tokens as "trivias" on other tokens, instead of producing individual \
comment tokens)</div><div><br></div><div>2. A BlackHoleChannel channel to lex \
whitespaces and new lines. This type of channel is similar to RegexpChannel, but \
matched tokens will be filtered out of the lexer output. This is why you do not need \
to specify a type, but only a regular expression.</div><div><br></div><div>3. \
Finally, a UnknownCharacterChannel channel to match all other characters as \
individual tokens of type \
GenericTokenType.UNKNOWN_CHAR.</div><div><br></div><div>Note that SSLR will create an \
artificial token at the very end to indicate the end of the file, of type \
GenericTokenType.EOF.</div><div><br></div><div>Once you have this is place, you can \
already develop a SonarQube Sensor, iterate over all source files of the \
ModuleFileSystem, run your lexer, and save the \
metrics.</div><div><br></div><div>Then, you can add a basic grammar, that will simply \
match every file:</div><div><br></div><div>b.rule(COMPILATION_UNIT).is(</div><div> \
b.zeroOrMore(b.anyTokenBut(GenericTokenType.EOF)), // Consume all tokens types except \
EOF</div><div> GenericTokenType.EOF); // Expect an end of \
file</div><div><br></div><div>The MiniCGrammar shows how a more realistic grammar can \
be implemented, and there also are a few more minimalistic ones available as examples \
in the "sslr-examples" module of SSLR:</div><div> <a \
href="https://github.com/SonarSource/sslr/tree/master/sslr-examples/src/main/java/org/sonar/sslr/examples/grammars" \
rel="nofollow" link="external" \
target="_blank">https://github.com/SonarSource/sslr/tree/master/sslr-examples/src/main/java/org/sonar/sslr/examples/grammars</a></div><div>(note \
that most of those examples, unlike MiniC, use the "lexerless" grammar \
builder, which is going to be slightly different from the "lexerful" \
grammar builder)</div><div><br></div><div>I hope it makes more sense \
now.</div><div><br></div><div>The best way to proceed is to play with writing a basic \
grammar, for instance for arithmetic expressions.</div><div><br></div><div>You can \
also quickly create an SSLR Toolkit to visualize the parse trees for various inputs, \
or use AstXmlPrinter.print() to print a parse tree to a string for debugging/learning \
purposes.</div><div><br></div><div>Kind regards,</div></div></div><div \
class="gmail_extra"><div><div><div><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><br>--<br>Dinesh \
Bolkensteyn<br></span><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><a \
href="http://www.sonarsource.com/" rel="nofollow" link="external" \
target="_blank">www.SonarSource.com</a></span><br><a \
href="http://twitter.com/DBolkensteyn" rel="nofollow" link="external" \
target="_blank">twitter.com/DBolkensteyn</a><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"></span><span \
style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><a \
href="http://www.SonarSource.com" rel="nofollow" link="external" \
target="_blank"></a></span><br></div> <br></div></div><div><div><div \
class="gmail_quote">On Mon, Sep 15, 2014 at 11:13 PM, patschius <span \
dir="ltr"><<a href="http://user/SendEmail.jtp?type=node&node=5028268&i=0" \
rel="nofollow" link="external" target="_blank">[hidden email]</a>></span> \
wrote:<br><blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' \
style="border-left:2px solid #cccccc;padding:0 1em" class="gmail_quote">Hi All,<br> \
<br> First of all i am new to Sonar as a plugin developer, but i was a Sonar User<br>
before that and really like it!<br>
<br>
So when starting the development i have mastered the easy part and made<br>
Sonar recognize and install my Plugin as well as to detect the newly defined<br>
language and so on.<br>
<br>
As a starting goal i have set for myself that i just want some simple<br>
metrics about Files, LOC and Comments. This stuff is almost language<br>
independent. If my thoughts are correct i don't need any sophisticated<br>
grammar for this purpose. Just a channel to recognize the comments and ...<br>
something else...<br>
<br>
And this is where got stuck. I understand (or at least i think i do) the<br>
principles of PEG and so on, but somehow i fail to put it into context with<br>
SSLR. So i am a bit confused how the Channels defined in the Lexer and the<br>
Grammar actually work together. Also the Methods in the GrammarBuilder do<br>
not make that much sense to me, although i have read their javadoc back and<br>
forth, but its not really verbose. I also looked at other languages that use<br>
SSLR, but they are already so far progressed that it is kind of hard to<br>
figure out the simple connections.<br>
<br>
So what i would need is like a step by step tutorial that explains how to<br>
build a simple grammar from scratch (I also looked at the MiniCGrammar...).<br>
Is there any chance such a thing exists somewhere or can someone point me to<br>
where i could gather such information?<br>
<br>
Thanks a lot for your help in advance.<br>
<br>
Best Regards,<br>
Christoph Patscheider<br>
<br>
<br>
<br>
<br>
--<br>
View this message in context: <a \
href="http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267.html" \
rel="nofollow" link="external" \
target="_blank">http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267.html</a><br>
Sent from the SonarQube Developers mailing list archive at Nabble.com.<br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe from this list, please visit:<br>
<br>
<a href="http://xircles.codehaus.org/manage_email" rel="nofollow" \
link="external" target="_blank">http://xircles.codehaus.org/manage_email</a><br> <br>
<br>
</blockquote></div><br></div></div></div></div>
<br>
<br>
<hr noshade size="1" color="#cccccc">
<div style="color:#444;font:12px tahoma,geneva,helvetica,arial,sans-serif">
<div style="font-weight:bold">If you reply to this email, your message will be \
added to the discussion below:</div> <a \
href="http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028268.html" \
rel="nofollow" link="external" \
target="_blank">http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028268.html</a>
</div>
<div style="color:#666;font:11px \
tahoma,geneva,helvetica,arial,sans-serif;margin-top:.4em;line-height:1.5em">
To unsubscribe from Where can i find an Intruction to SSLR?, <a rel="nofollow" \
link="external" target="_top">click here</a>.<br> <a \
href="http://sonarqube.15.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&am \
p;id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace \
-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&b \
readcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" \
rel="nofollow" style="font:9px serif" link="external" target="_blank">NAML</a> \
</div></blockquote></div><br></div>
<br></div></div><hr align="left" width="300">
View this message in context: <a \
href="http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028276.html" \
rel="nofollow" link="external" target="_blank">Re: [sonar-dev] Where can i find an \
Intruction to SSLR?</a><span class=""><div><div><br> Sent from the <a \
href="http://sonarqube.15.x6.nabble.com/SonarQube-Developers-f4523654.html" \
rel="nofollow" link="external" target="_blank">SonarQube Developers mailing list \
archive</a> at Nabble.com.<br></div></div></span></blockquote></div><br></div>
<br>
<br>
<hr noshade size="1" color="#cccccc">
<div style="color:#444;font:12px tahoma,geneva,helvetica,arial,sans-serif"><span \
class=""> <div style="font-weight:bold">If you reply to this email, your message \
will be added to the discussion below:</div> </span><a \
href="http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028278.html" \
target="_blank" rel="nofollow" \
link="external">http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028278.html</a>
</div><div class="HOEnZb"><div class="h5">
<div style="color:#666;font:11px \
tahoma,geneva,helvetica,arial,sans-serif;margin-top:.4em;line-height:1.5em">
To unsubscribe from Where can i find an Intruction to SSLR?, <a href="" \
target="_blank" rel="nofollow" link="external">click here</a>.<br> <a \
href="http://sonarqube.15.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&am \
p;id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace \
-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&b \
readcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" \
rel="nofollow" style="font:9px serif" target="_blank" link="external">NAML</a> \
</div></div></div></blockquote></div><br></div>
<br/><hr align="left" width="300" />
View this message in context: <a \
href="http://sonarqube.15.x6.nabble.com/Where-can-i-find-an-Intruction-to-SSLR-tp5028267p5028280.html">Re: \
[sonar-dev] Where can i find an Intruction to SSLR?</a><br/> Sent from the <a \
href="http://sonarqube.15.x6.nabble.com/SonarQube-Developers-f4523654.html">SonarQube \
Developers mailing list archive</a> at Nabble.com.<br/>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic