[prev in list] [next in list] [prev in thread] [next in thread]
List: asterisk-dev
Subject: Re: [asterisk-dev] [Code Review] 4188: segfault when playing back voicemail under high concurrency w
From: "Ben Smithurst" <reviewboard () asterisk ! org>
Date: 2014-11-21 10:03:11
Message-ID: 20141121100311.16469.89183 () sonic ! digium ! api
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> On Nov. 18, 2014, 1:25 p.m., Matt Jordan wrote:
> > From http://www.washington.edu/imap/documentation/internal.txt.html:
> >
> > {quote}
> > * * * IMPORTANT * * *
> >
> > Any multi-threaded application should test stream->lock prior to
> > calling any c-client stream functions. Any attempt to call a
> > mail_xxx() function while one is already in progress on the same
> > stream will cause the application to fail in unpredictable ways.
> >
> > Note that this check is insufficient in a preemptive-scheduling
> > multi-tasking application due to the possibility of a timing race.
> > Such applications must be written so that only one process accesses
> > the stream, or to have a higher level lock.
> >
> > Since MAIL operations will not finish until they are completed, a
> > single-tasking application does not have to worry about this problem,
> > except in the callback invoked from MAIL (e.g. mm_exists(), etc.) in which
> > case the stream is *always* locked.
> > {quote}
> >
> > So, a few things from this:
> > (1) Based on the description from the UW IMAP documentation, locking vms->lock is \
> > sufficient *if* other mail accesses also lock the same vms object for that \
> > stream. If that isn't happening, then your patch probably won't fix anything, \
> > since both mail_open calls were already protected by the vms->lock and your \
> > global lock doesn't protect any other mail_XXXX accesses. The only way your patch \
> > would affect anything is if there were two different vms objects being used to \
> > open the same stream at the same time. (2) Your backtrace on the issue doesn't \
> > show concurrent accesses to vm_execmain - although a lot of symbols are missing - \
> > so right now it's impossible to tell who the offending accesses are. It's also \
> > impossible to know if it is due to two concurrent accesses of mail_open with \
> > different vms objects, or something else.
> > Generally, I'm not this patch fixes your issue.
The concern we found was that some code called by mail_open uses static variables, \
specifically the ip_nametoaddr function - at least the ip6_unix.c implementation of \
it, which is where we're seeing crashes. We believe the problem may be 2 DIFFERENT \
streams are being opened at once, correctly locked individually, but ultimately \
causing a conflict with a static variable deep inside the uw-imap code.
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/4188/#review13790
-----------------------------------------------------------
On Nov. 17, 2014, 3:03 p.m., David Duncan Ross Palmer wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/4188/
> -----------------------------------------------------------
>
> (Updated Nov. 17, 2014, 3:03 p.m.)
>
>
> Review request for Asterisk Developers.
>
>
> Bugs: ASTERISK-24516
> https://issues.asterisk.org/jira/browse/ASTERISK-24516
>
>
> Repository: Asterisk
>
>
> Description
> -------
>
> Asterisk segfaults when playing back voicemail under high concurrency with an IMAP \
> backend
>
> Diffs
> -----
>
> /trunk/apps/app_voicemail.c 427675
>
> Diff: https://reviewboard.asterisk.org/r/4188/diff/
>
>
> Testing
> -------
>
> Rapid load testing performed with {{SIPp}}:
> {code}
> sipp -sn uac -d 7000 -s ‘*1' 127.0.0.1 -l 400 -mp 5606
> {code}
>
>
> Thanks,
>
> David Duncan Ross Palmer
>
>
[Attachment #5 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/4188/">https://reviewboard.asterisk.org/r/4188/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On November 18th, 2014, 1:25 p.m. UTC, <b>Matt \
Jordan</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px solid \
#d0d0d0; padding-left: 10px;"> <pre style="white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">From http://www.washington.edu/imap/documentation/internal.txt.html:
{quote}
* * * IMPORTANT * * *
Any multi-threaded application should test stream->lock prior to
calling any c-client stream functions. Any attempt to call a
mail_xxx() function while one is already in progress on the same
stream will cause the application to fail in unpredictable ways.
Note that this check is insufficient in a preemptive-scheduling
multi-tasking application due to the possibility of a timing race.
Such applications must be written so that only one process accesses
the stream, or to have a higher level lock.
Since MAIL operations will not finish until they are completed, a
single-tasking application does not have to worry about this problem,
except in the callback invoked from MAIL (e.g. mm_exists(), etc.) in which
case the stream is *always* locked.
{quote}
So, a few things from this:
(1) Based on the description from the UW IMAP documentation, locking vms->lock is \
sufficient *if* other mail accesses also lock the same vms object for that stream. If \
that isn't happening, then your patch probably won't fix anything, since both \
mail_open calls were already protected by the vms->lock and your global lock \
doesn't protect any other mail_XXXX accesses. The only way your patch would \
affect anything is if there were two different vms objects being used to open the \
same stream at the same time. (2) Your backtrace on the issue doesn't show \
concurrent accesses to vm_execmain - although a lot of symbols are missing - so right \
now it's impossible to tell who the offending accesses are. It's also \
impossible to know if it is due to two concurrent accesses of mail_open with \
different vms objects, or something else.
Generally, I'm not this patch fixes your issue.</pre>
</blockquote>
</blockquote>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">The concern we found was \
that some code called by mail_open uses static variables, specifically the \
ip_nametoaddr function - at least the ip6_unix.c implementation of it, which is where \
we're seeing crashes. We believe the problem may be 2 DIFFERENT streams are \
being opened at once, correctly locked individually, but ultimately causing a \
conflict with a static variable deep inside the uw-imap code.</pre> <br />
<p>- Ben</p>
<br />
<p>On November 17th, 2014, 3:03 p.m. UTC, David Duncan Ross Palmer wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('https://reviewboard.asterisk.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Asterisk Developers.</div>
<div>By David Duncan Ross Palmer.</div>
<p style="color: grey;"><i>Updated Nov. 17, 2014, 3:03 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>
<a href="https://issues.asterisk.org/jira/browse/ASTERISK-24516">ASTERISK-24516</a>
</div>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
Asterisk
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Asterisk segfaults when playing back voicemail under high concurrency \
with an IMAP backend</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Rapid load testing performed with {{SIPp}}: {code}
sipp -sn uac -d 7000 -s ‘*1' 127.0.0.1 -l 400 -mp 5606
{code}</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/trunk/apps/app_voicemail.c <span style="color: grey">(427675)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/4188/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic