'Re: [Paraview] Help with optimizing Paraview Python script'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       paraview
Subject:    Re: [Paraview] Help with optimizing Paraview Python script
From:       Cory Quammen <cory.quammen () kitware ! com>
Date:       2015-10-22 13:29:09
Message-ID: CAB5Fpx4M7Yt9haL5A=9qyYadp=KxZYruAYdU=sux+D98zgafjw () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

>
> In terms of optimization I'm inclined to believe that my script isn't
> written very well and that it is the source of the slowdown that I'm seeing
> when running it. I CC'd you on my reply to Andy's email where we identified
> some areas of the script that were particularly slow during execution.
>
Agreed.

- Cory

> Best regards,
>
>
> Christopher R. Neal
> Graduate Student
> Center for Compressible Multiphase Turbulence
> Mechanical and Aerospace Engineering Department
> University of Florida
> Cell: (863)-697-1958
> E-mail:  chrisneal@ufl.edu
>
>
> ------------------------------
> *From:* Cory Quammen <cory.quammen@kitware.com>
> *Sent:* Wednesday, October 21, 2015 11:18 PM
> *To:* Andy Bauer
> *Cc:* Neal,Christopher R; paraview@paraview.org
> *Subject:* Re: [Paraview] Help with optimizing Paraview Python script
>
> A few questions:
>
> * How did you obtain the ParaView executable? (download, build)? If you
> built ParaView in Debug mode, that could slow things down.
>
> * How big are your data files, both in terms of disk size and grid
> elements?
>
> * How much RAM is on your system?
>
> - Cory
>
> On Wed, Oct 21, 2015 at 8:50 PM, Andy Bauer <andy.bauer@kitware.com>
> wrote:
>
>> Did you profile this at all to see where most of the time being spent?
>>
>> Looking at this, I don't see anything really bad. I think you can take
>> out the time.sleep(1.0) part though. Also, replacing the Calculator filter
>> with the PythonCalculator filter should help some. If you're running this
>> in parallel, the fetch is probably hurting performance as well.
>>
>> On Wed, Oct 21, 2015 at 4:02 PM, Neal,Christopher R <chrisneal@ufl.edu>
>> wrote:
>>
>>> Hello,
>>>
>>>
>>> I have written a python script that loads in an Ensight data file which
>>> has multiple solution times associated with it and performs a set of fairly
>>> simple operations on it. It is running very slow, and I'm wondering if
>>> there are any experts who use Python to control Paraview that could help me
>>> to optimize the script.  In a nutshell the script does the following.
>>>
>>>
>>> 1.) Go to a timestep
>>>
>>> 2.) Generate a series of 2D slices in the 3D domain
>>>
>>> 3.) Compute some values of the solution variables on those slices
>>>
>>> 4.) Write data to a file
>>>
>>> 5.) Delete the entire pipeline except the original data
>>>
>>> 6.) Go back to 1.)
>>>
>>>
>>> This process is painfully slow(2+days just to run the script on a modest
>>> size data set).  I believe that the script having to delete the pipeline
>>> and then re-create it for every timestep is contributing to the slowdown.
>>> My  trouble is that I do not know how to selectively activate certain
>>> elements in the pipeline if I want to extract information from them, so I
>>> have to add them sequentially so that the newest one is the active one.
>>> The script may also be rendering each step, which is unnecessary for me,
>>> but I do not know how to disable that as well.
>>>
>>>
>>> I have attached a copy of the script. I have a minimal working example
>>> that contains the data set with only 3 timesteps worth of data.
>>>
>>>
>>> I am even willing to pay  $100 for an expert to help me out with this.
>>>
>>>
>>> Thank you,
>>>
>>>
>>> Christopher R. Neal
>>> Graduate Student
>>> Center for Compressible Multiphase Turbulence
>>> Mechanical and Aerospace Engineering Department
>>> University of Florida
>>> Cell: (863)-697-1958
>>> E-mail:  chrisneal@ufl.edu
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>
>>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/paraview
>>
>>
>
>
> --
> Cory Quammen
> R&D Engineer
> Kitware, Inc.
>

-- 
Cory Quammen
R&D Engineer
Kitware, Inc.

[Attachment #5 (text/html)]

<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div \
style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><p><span \
style="font-size:12pt">In terms of optimization I&#39;m inclined to believe that my \
script isn&#39;t written very well and that it is the source of the slowdown that \
I&#39;m seeing when running it. I CC&#39;d you on my reply to Andy&#39;s email where \
we identified some areas of the script that were  particularly slow during \
execution.</span></p></div></div></blockquote><div>Agreed.</div><div><br></div><div>- \
Cory</div><div>  </div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div \
style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
 <p>Best regards,<br>
</p><span class="">
<p><br>
</p>
<div>
<div name="divtagdefaultwrapper">
<span lang="en-US">
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Christopher R. Neal<span \
name="searchHitInReadingPane"></span></span></font></div> <div style="margin:0"><font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Graduate Student <br>
Center for Compressible Multiphase Turbulence<br>
</span></font></div>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Mechanical and Aerospace Engineering Department<br> \
</span></font></div> <div style="margin:0"><font face="Calibri,sans-serif" \
size="2"><span style="font-size:11pt">University of Florida</span></font></div> <div \
style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Cell: <a href="tel:%28863%29-697-1958" value="+18636971958" \
target="_blank">(863)-697-1958</a></span></font></div> <div style="margin:0"><font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">E-mail:   <a \
href="mailto:chrisneal@ufl.edu" \
target="_blank">chrisneal@ufl.edu</a></span></font></div> </span></div>
</div>
<br>
<br>
</span><div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" \
color="#000000"><b>From:</b> Cory Quammen &lt;<a \
href="mailto:cory.quammen@kitware.com" \
target="_blank">cory.quammen@kitware.com</a>&gt;<br> <b>Sent:</b> Wednesday, October \
21, 2015 11:18 PM<br> <b>To:</b> Andy Bauer<br>
<b>Cc:</b> Neal,Christopher R; <a href="mailto:paraview@paraview.org" \
target="_blank">paraview@paraview.org</a><span class=""><br> <b>Subject:</b> Re: \
[Paraview] Help with optimizing Paraview Python script</span></font> <div>  </div>
</div><div><div class="h5">
<div>
<div dir="ltr">A few questions:
<div><br>
</div>
<div>* How did you obtain the ParaView executable? (download, build)? If you built \
ParaView in Debug mode, that could slow things down.</div> <div><br>
</div>
<div>* How big are your data files, both in terms of disk size and grid \
elements?</div> <div><br>
</div>
<div>* How much RAM is on your system?</div>
<div><br>
</div>
<div>- Cory</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Oct 21, 2015 at 8:50 PM, Andy Bauer <span dir="ltr">
&lt;<a href="mailto:andy.bauer@kitware.com" \
target="_blank">andy.bauer@kitware.com</a>&gt;</span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <div dir="ltr">Did you profile this at all to see where most \
of the time being spent?<br> <div><br>
Looking at this, I don&#39;t see anything really bad. I think you can take out the \
time.sleep(1.0) part though. Also, replacing the Calculator filter with the \
PythonCalculator filter should help some. If you&#39;re running this in parallel, the \
fetch is probably hurting  performance as well. <br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Wed, Oct 21, 2015 at 4:02 PM, Neal,Christopher R <span dir="ltr">
&lt;<a href="mailto:chrisneal@ufl.edu" \
target="_blank">chrisneal@ufl.edu</a>&gt;</span> wrote:<br> </div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"> <div>
<div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
 <p>Hello,</p>
<p><br>
</p>
<p>I have written a python script that loads in an Ensight data file which has \
multiple solution times associated with it and performs a set of fairly simple \
operations on it. It is running very slow, and I&#39;m wondering if there are any \
experts who use Python  to control Paraview that could help me to optimize the \
script.   In a nutshell the script does the following.</p> <p><br>
</p>
<p>1.) Go to a timestep</p>
<p>2.) Generate a series of 2D slices in the 3D domain</p>
<p>3.) Compute some values of the solution variables on those slices</p>
<p>4.) Write data to a file</p>
<p>5.) Delete the entire pipeline except the original data</p>
<p>6.) Go back to 1.)</p>
<p><br>
</p>
<p>This process is painfully slow(2+days just to run the script on a modest size data \
set).   I believe that the script having to delete the pipeline and then re-create it \
for every timestep is contributing to the slowdown. My   trouble is that I do not \
know  how to selectively activate certain elements in the pipeline if I want to \
extract information from them, so I have to add them sequentially so that the newest \
one is the active one.   The script may also be rendering each step, which is \
unnecessary for me,  but I do not know how to disable that as well.</p>
<p><br>
</p>
<p>I have attached a copy of the script. I have a minimal working example that \
contains the data set with only 3 timesteps worth of data.</p> <p><br>
</p>
<p>I am even willing to pay   $100 for an expert to help me out with this.<br>
</p>
<p><br>
</p>
<p>Thank you,<br>
</p>
<p><br>
</p>
<div>
<div name="divtagdefaultwrapper"><span lang="en-US">
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Christopher R. Neal<span \
name="searchHitInReadingPane"></span></span></font></div> <div style="margin:0"><font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Graduate Student <br>
Center for Compressible Multiphase Turbulence<br>
</span></font></div>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Mechanical and Aerospace Engineering Department<br> \
</span></font></div> <div style="margin:0"><font face="Calibri,sans-serif" \
size="2"><span style="font-size:11pt">University of Florida</span></font></div> <div \
style="margin:0"><font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">Cell: <a href="tel:%28863%29-697-1958" value="+18636971958" \
target="_blank">(863)-697-1958</a></span></font></div> <div style="margin:0"><font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">E-mail:   <a \
href="mailto:chrisneal@ufl.edu" \
target="_blank">chrisneal@ufl.edu</a></span></font></div> </span></div>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" \
target="_blank">www.kitware.com</a><br> <br>
Visit other Kitware open-source projects at <a \
href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" \
target="_blank"> http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a \
href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank"> \
http://paraview.org/Wiki/ParaView</a><br> <br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" \
rel="noreferrer" target="_blank"> http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" \
target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br> <br>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" \
target="_blank">www.kitware.com</a><br> <br>
Visit other Kitware open-source projects at <a \
href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" \
target="_blank"> http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a \
href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank"> \
http://paraview.org/Wiki/ParaView</a><br> <br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" \
rel="noreferrer" target="_blank"> http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" \
target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br> <br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>Cory Quammen<br>
R&amp;D Engineer<br>
Kitware, Inc.</div>
</div>
</div>
</div></div></div>
</div>
</div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
class="gmail_signature">Cory Quammen<br>R&amp;D Engineer<br>Kitware, Inc.</div> \
</div></div>

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView

Search the list archives at: http://markmail.org/search/?q=ParaView

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/paraview

[prev in list] [next in list] [prev in thread] [next in thread] 