[prev in list] [next in list] [prev in thread] [next in thread]
List: asterisk-dev
Subject: Re: [asterisk-dev] [Code Review] 3668: refcounter.py uses excessive RAM when processing large refs f
From: "Corey Farrell" <reviewboard () asterisk ! org>
Date: 2014-06-27 19:17:21
Message-ID: 20140627191721.10720.20817 () sonic ! digium ! api
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> On June 26, 2014, 11:57 a.m., wdoekes wrote:
> > Some micro-optimization below:
Thanks for teaching me this about Python!
- Corey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3668/#review12341
-----------------------------------------------------------
On June 27, 2014, 3:14 p.m., Corey Farrell wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3668/
> -----------------------------------------------------------
>
> (Updated June 27, 2014, 3:14 p.m.)
>
>
> Review request for Asterisk Developers and Matt Jordan.
>
>
> Bugs: ASTERISK-23921
> https://issues.asterisk.org/jira/browse/ASTERISK-23921
>
>
> Repository: Asterisk
>
>
> Description
> -------
>
> When processing a 212MB refs file, refcounter.py used over 3GB of RAM. This caused \
> swap thrashing and temporarily froze my system. The included patch makes the \
> following memory optimizations:
> * skewed and finished object lists are only populated if not disabled
> * lines are saved to each object as the final output line
>
> Saving the whole lines in output format seems to reduce memory usage by 80-90%. \
> Ignoring finished/skewed objects caused an additional reduction of about 75% on my \
> system.
>
> Diffs
> -----
>
> /branches/12/contrib/scripts/refcounter.py 417247
>
> Diff: https://reviewboard.asterisk.org/r/3668/diff/
>
>
> Testing
> -------
>
> Watched 'top -c' with refcounter.py running on the 212MB refs log. The highest \
> memory usage I saw was 127MB with '-sn' options and 472MB with full output.
>
> Thanks,
>
> Corey Farrell
>
>
[Attachment #5 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/3668/">https://reviewboard.asterisk.org/r/3668/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On June 26th, 2014, 11:57 a.m. EDT, <b>wdoekes</b> \
wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; \
padding-left: 10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Some \
micro-optimization below:</pre> </blockquote>
</blockquote>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Thanks for teaching me \
this about Python!</pre> <br />
<p>- Corey</p>
<br />
<p>On June 27th, 2014, 3:14 p.m. EDT, Corey Farrell wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('https://reviewboard.asterisk.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Asterisk Developers and Matt Jordan.</div>
<div>By Corey Farrell.</div>
<p style="color: grey;"><i>Updated June 27, 2014, 3:14 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>
<a href="https://issues.asterisk.org/jira/browse/ASTERISK-23921">ASTERISK-23921</a>
</div>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
Asterisk
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">When processing a 212MB refs file, refcounter.py used over 3GB of RAM. \
This caused swap thrashing and temporarily froze my system. The included patch makes \
the following memory optimizations:
* skewed and finished object lists are only populated if not disabled
* lines are saved to each object as the final output line
Saving the whole lines in output format seems to reduce memory usage by 80-90%. \
Ignoring finished/skewed objects caused an additional reduction of about 75% on my \
system.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Watched 'top -c' with refcounter.py running on the 212MB refs \
log. The highest memory usage I saw was 127MB with '-sn' options and 472MB \
with full output.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/branches/12/contrib/scripts/refcounter.py <span style="color: \
grey">(417247)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/3668/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic