[prev in list] [next in list] [prev in thread] [next in thread]
List: cfe-dev
Subject: Re: [cfe-dev] Is there a way to let the AST matcher run on multiple files in parallel?
From: Han Wang <wanghan02 () gmail ! com>
Date: 2015-07-31 8:21:10
Message-ID: CAPDNnCZQUKk=dY1kiLooA3FxGkLfipG5e7UVX9aKQh3ywj5MWA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thanks! :) Let me see if I could generate some patch files then it would be
easier for the synchronization.
On Fri, Jul 31, 2015 at 10:17 AM, Manuel Klimek <klimek@google.com> wrote:
> That's why I wrote in my previous mail:
> "make sure your tool outputs the replacements that you want to apply in
> some format you can later slurp in and apply easily."
> The "apply" part is going to be single threaded.
>
> On Fri, Jul 31, 2015 at 9:30 AM Han Wang <wanghan02@gmail.com> wrote:
>
>> Hi Manuel,
>>
>> I currently got some synchronization problem with this method. If a
>> header file is included by multiple source files, the write operations to
>> the header file are not synchronized. But if we use only 1 thread, it is
>> synchronized.
>>
>> Best regards,
>> Han
>>
>> On Wed, Jul 22, 2015 at 2:49 PM, Manuel Klimek <klimek@google.com> wrote:
>>
>>> Ok, so make sure your tool outputs the replacements that you want to
>>> apply in some format you can later slurp in and apply easily. Make the name
>>> of the output file depend on on the name of the translation unit you got at
>>> the command line.
>>>
>>> Then you can do:
>>> $ ls test1.cc test2.cc |xargs -n1 -P2 ./matchertool.out
>>>
>>> Unfortunately currently I don't think there's a simpler way than that.
>>>
>>> On Wed, Jul 22, 2015 at 2:42 PM Han Wang <wanghan02@gmail.com> wrote:
>>>
>>>> Hi Manuel,
>>>>
>>>> That I have the compilation database already. :)
>>>>
>>>> Let's say we have a simple case with only 2 source files (test1.cc and
>>>> test2.cc). We also have the compilation database for these 2 files. And we
>>>> have the AST matcher tool up and running. Normally I'd run
>>>> ./matchertool.out test1.cc test2.cc to do the refactoring. To make the
>>>> analyzing of the 2 files running in parallel, what shall we do next?
>>>>
>>>> Thanks!
>>>> Best regards,
>>>> Han
>>>>
>>>> On Wed, Jul 22, 2015 at 2:29 PM, Manuel Klimek <klimek@google.com>
>>>> wrote:
>>>>
>>>>> Those are not 2 options. You'll want to have both :)
>>>>> You need the compilation database so the tool only needs the source
>>>>> file to run.
>>>>>
>>>>> On Wed, Jul 22, 2015 at 2:27 PM Han Wang <wanghan02@gmail.com> wrote:
>>>>>
>>>>>> Hi Manuel,
>>>>>>
>>>>>> Thank you for your answer. The goal is to do refactoring to the
>>>>>> project and apply the replacements to the original files. For the option
>>>>>> (a), could you describe more specifically how to do it? Thanks. I checked
>>>>>> the run function of clang::tooling::ClangTool, but it's just a for loop
>>>>>> over all the source files.
>>>>>>
>>>>>> Best regards,
>>>>>> Han
>>>>>>
>>>>>> On Wed, Jul 22, 2015 at 12:33 PM, Manuel Klimek <klimek@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> You can use
>>>>>>> a) a compilation database
>>>>>>> b) find . -name '*.cc' |xargs -n1 -P32
>>>>>>> depending on the output data you want to produce, you'll need to
>>>>>>> pass an additional flag to your tool to get the shard number...
>>>>>>>
>>>>>>> On Tue, Jul 21, 2015 at 6:20 PM Han Wang <wanghan02@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Is there a way to let the AST matcher run on multiple files in
>>>>>>>> parallel? I'm working on Mac. I have some old windows code which may slow
>>>>>>>> down the compiling time (maybe for some other reasons I don't know
>>>>>>>> exactly). So this makes it necessary for me to analyze each translation
>>>>>>>> unit in parallel.
>>>>>>>>
>>>>>>>> At the moment I have only one mac which has 32 cores. I can live
>>>>>>>> with that if it could be fully parallel.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Best regards,
>>>>>>>> Han
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> cfe-dev mailing list
>>>>>>>> cfe-dev@cs.uiuc.edu
>>>>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
[Attachment #5 (text/html)]
<div dir="ltr">Thanks! :) Let me see if I could generate some patch files then it \
would be easier for the synchronization. </div><div class="gmail_extra"><br><div \
class="gmail_quote">On Fri, Jul 31, 2015 at 10:17 AM, Manuel Klimek <span \
dir="ltr"><<a href="mailto:klimek@google.com" \
target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">That's why I wrote in my previous \
mail:<span class=""><div>"make sure your tool outputs the replacements that you \
want to apply in some format you can later slurp in and apply \
easily."</div></span><div>The "apply" part is going to be single \
threaded.</div></div><div class="HOEnZb"><div class="h5"><br><div \
class="gmail_quote"><div dir="ltr">On Fri, Jul 31, 2015 at 9:30 AM Han Wang <<a \
href="mailto:wanghan02@gmail.com" target="_blank">wanghan02@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Manuel, \
<div><br></div><div>I currently got some synchronization problem with this method. If \
a header file is included by multiple source files, the write operations to the \
header file are not synchronized. But if we use only 1 thread, it is synchronized. \
</div><div><br></div><div>Best regards,</div><div>Han</div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 22, 2015 at 2:49 PM, \
Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" \
target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>Ok, so make sure your tool outputs the \
replacements that you want to apply in some format you can later slurp in and apply \
easily. Make the name of the output file depend on on the name of the translation \
unit you got at the command line.</div><div><br></div><div>Then you can do:</div>$ ls \
test1.cc test2.cc |xargs -n1 -P2 ./matchertool.out<div><br></div><div>Unfortunately \
currently I don't think there's a simpler way than \
that.</div></div><div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jul \
22, 2015 at 2:42 PM Han Wang <<a href="mailto:wanghan02@gmail.com" \
target="_blank">wanghan02@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Manuel, <div><br></div><div>That I have \
the compilation database already. :)</div><div><br></div><div>Let's say we have a \
simple case with only 2 source files (test1.cc and test2.cc). We also have the \
compilation database for these 2 files. And we have the AST matcher tool up and \
running. Normally I'd run ./matchertool.out test1.cc test2.cc to do the \
refactoring. To make the analyzing of the 2 files running in parallel, what shall we \
do next? </div><div><br></div><div>Thanks!</div><div>Best \
regards,</div><div>Han</div></div><div class="gmail_extra"><br><div \
class="gmail_quote">On Wed, Jul 22, 2015 at 2:29 PM, Manuel Klimek <span \
dir="ltr"><<a href="mailto:klimek@google.com" \
target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Those are not 2 options. You'll want to \
have both :)<div>You need the compilation database so the tool only needs the source \
file to run.</div></div><div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, \
Jul 22, 2015 at 2:27 PM Han Wang <<a href="mailto:wanghan02@gmail.com" \
target="_blank">wanghan02@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Manuel, <div><br></div><div>Thank you for \
your answer. The goal is to do refactoring to the project and apply the replacements \
to the original files. For the option (a), could you describe more specifically how \
to do it? Thanks. I checked the run function of clang::tooling::ClangTool, but \
it's just a for loop over all the source files. </div><div><br></div><div>Best \
regards,</div><div>Han</div></div><div class="gmail_extra"><br><div \
class="gmail_quote">On Wed, Jul 22, 2015 at 12:33 PM, Manuel Klimek <span \
dir="ltr"><<a href="mailto:klimek@google.com" \
target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>You can use</div><div>a) a compilation \
database</div><div>b) find . -name '*.cc' |xargs -n1 -P32</div><div>depending \
on the output data you want to produce, you'll need to pass an additional flag to \
your tool to get the shard number...</div></div><br><div \
class="gmail_quote"><div><div><div dir="ltr">On Tue, Jul 21, 2015 at 6:20 PM Han Wang \
<<a href="mailto:wanghan02@gmail.com" target="_blank">wanghan02@gmail.com</a>> \
wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi all, \
<div><br></div><div>Is there a way to let the AST matcher run on multiple files in \
parallel? I'm working on Mac. I have some old windows code which may slow down \
the compiling time (maybe for some other reasons I don't know exactly). So this \
makes it necessary for me to analyze each translation unit in parallel. \
<br></div><div><br></div><div>At the moment I have only one mac which has 32 cores. I \
can live with that if it could be fully parallel. \
</div><div><br></div><div>Thanks!</div><div>Best \
regards,</div><div>Han</div><div><br></div></div></div></div> \
_______________________________________________<br> cfe-dev mailing list<br>
<a href="mailto:cfe-dev@cs.uiuc.edu" target="_blank">cfe-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" rel="noreferrer" \
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br> \
</blockquote></div> </blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
_______________________________________________
cfe-dev mailing list
cfe-dev@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic