[prev in list] [next in list] [prev in thread] [next in thread]
List: asterisk-dev
Subject: Re: [asterisk-dev] Asterisk Load Performance
From: bala murugan <fightwithme () gmail ! com>
Date: 2017-01-11 1:58:59
Message-ID: CAJQn6_ze-7aDYfPaSrMm8eqdVh5J6aiyB5=uu6aKBTfp2RmG=A () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
How to disabled the channel_varset from stasis in asterisk 12 , can you
please provide the steps or configuration .
thanks,
bala
On Fri, Jun 17, 2016 at 5:31 PM, Matthew Jordan <mjordan@digium.com> wrote:
> On Fri, Jun 17, 2016 at 1:37 PM, Richard Mudgett <rmudgett@digium.com>
> wrote:
> >
> >
> > On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello
> > <michael.petruzzello@civi.com> wrote:
> >>
> >> Hello,
> >>
> >> I am currently working on determining bottlenecks in Asterisk and a
> Stasis
> >> App. I'm currently trying to handle 83.3 calls/second. For the most
> part,
> >> Asterisk and the Stasis APP handle that well, but there is a 60+ second
> >> delay in response time.
> >>
> >> On the Asterisk side, I am seeing the following warnings. [Jun 17
> >> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The
> >> 'subm:cdr_engine-00000003' task processor queue reached 500 scheduled
> tasks.
> >> [Jun 17 12:00:18] WARNING[25477][C-00000068]: taskprocessor.c:803
> >> taskprocessor_push: The 'subm:devService-test-00000038' task processor
> queue
> >> reached 500 scheduled tasks.
> >> [Jun 17 12:00:21] WARNING[26298][C-000000a3]: taskprocessor.c:803
> >> taskprocessor_push: The 'subp:PJSIP/sippeer-00000022' task processor
> queue
> >> reached 500 scheduled tasks.
> >> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: taskprocessor.c:803
> >> taskprocessor_push: The 'subm:ast_channel_topic_all-cached-00000032'
> task
> >> processor queue reached 500 scheduled tasks.
> >> [Jun 17 12:01:32] WARNING[31697][C-000003b2]: taskprocessor.c:803
> >> taskprocessor_push: The 'subm:ast_channel_topic_all-00000036' task
> processor
> >> queue reached 500 scheduled tasks.
> >> [Jun 17 12:05:55] WARNING[23280]: taskprocessor.c:803
> taskprocessor_push:
> >> The 'SIP' task processor queue reached 500 scheduled tasks.
> >>
> >> I have not seen a configuration setting on Asterisk to prevent these
> >> warnings from occurring (I'm trying to avoid modifying Asterisk source
> code
> >> if possible). Looking at the task processors, I see the queue to the
> stasis
> >> app bottlenecks:
> >> subm:devService-test-00000038 4560990 0
> >> 1041689. It does clear up relatively quickly. The CDR engine also bottle
> >> necks (extremely badly), but I don't use that. Nothing else comes close
> to
> >> having a large queue.
> >>
> >> The stasis app itself is extremely streamlined and is very capable of
> >> handling a large number of messages at a time. The app runs with the
> JVM so
> >> I am also researching into that as well as the netty library I am using
> for
> >> the websocket connections.
> >>
> >> Any insight into Asterisk's side of the equation and how it scales on 40
> >> vCPUs would be greatly appreciated.
> >
> >
> > There are no options to disable those taskprocessor queue size warnings.
> > They are a
> > symptom of the system being severely stressed. If the stress continues
> it
> > is possible
> > that the system could consume all memory in those taskprocessor queues.
> >
> > Recent changes to the Asterisk v13 branch were made to help throttle back
> > incoming
> > SIP requests on PJSIP when the taskprocessors become backlogged like you
> are
> > seeing.
> > These changes will be in the forthcoming v13.10.0 release. If you want,
> you
> > can test with
> > the current v13 branch to see how these changes affect your stress
> testing.
> >
> > If you don't need CDR's then you really need to disable them as they
> consume
> > a lot of
> > processing time and the CDR taskprocessor queue backlog can take minutes
> to
> > clear.
> >
>
> To echo what Richard said, because Asterisk is now sharing state
> across the Stasis message bus, turning off subscribers to that bus
> will help performance. Some easy ones to disable, if you aren't using
> them, are CDRs, CEL, and AMI. Those all do a reasonable amount of
> processing, and you can get some noticeable improvement by disabling
> them.
>
> Once you get past that, you can start fiddling with some of the lower
> level options. To start, you can throttle things back further by
> disabling certain internal messages in stasis.conf. As stasis.conf
> notes, functionality within Asterisk can break (or just not happen) if
> some messages are removed. For example, disabling
> 'ast_channel_snapshot_type' would break ... most things. You may
> however be able to streamline your application by looking at what ARI
> messages it cares about, what messages it doesn't, inspecting the
> code, and disabling those that you don't care about. Lots of testing
> should occur before doing this, of course.
>
> You may also be able to get some different performance characteristics
> by changing the threadpool options for the message bus in stasis.conf.
> This may make a difference, depending on the underlying machine.
>
> --
> Matthew Jordan
> Digium, Inc. | CTO
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-dev
>
[Attachment #5 (text/html)]
<div dir="ltr">How to <span class="gmail-il" \
style="font-size:12.8px;background-color:rgb(255,255,255)">disabled</span><span \
style="font-size:12.8px"> the </span><span class="gmail-il" \
style="font-size:12.8px;background-color:rgb(255,255,255)">channel_varset</span><span \
style="font-size:12.8px"> from stasis in asterisk 12 , can you please provide the \
steps or configuration .</span><div><span \
style="font-size:12.8px"><br></span></div><div><span \
style="font-size:12.8px">thanks,</span></div><div><span \
style="font-size:12.8px">bala</span></div></div><div class="gmail_extra"><br><div \
class="gmail_quote">On Fri, Jun 17, 2016 at 5:31 PM, Matthew Jordan <span \
dir="ltr"><<a href="mailto:mjordan@digium.com" \
target="_blank">mjordan@digium.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Fri, Jun 17, 2016 at \
1:37 PM, Richard Mudgett <<a \
href="mailto:rmudgett@digium.com">rmudgett@digium.com</a>> wrote:<br> ><br>
><br>
> On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello<br>
> <<a href="mailto:michael.petruzzello@civi.com">michael.petruzzello@civi.com</a>> \
wrote:<br> >><br>
>> Hello,<br>
>><br>
>> I am currently working on determining bottlenecks in Asterisk and a \
Stasis<br> >> App. I'm currently trying to handle 83.3 calls/second. For \
the most part,<br> >> Asterisk and the Stasis APP handle that well, but there \
is a 60+ second<br> >> delay in response time.<br>
>><br>
>> On the Asterisk side, I am seeing the following warnings. [Jun 17<br>
>> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The<br>
>> 'subm:cdr_engine-00000003' task processor queue reached 500 \
scheduled tasks.<br> >> [Jun 17 12:00:18] WARNING[25477][C-00000068]: \
taskprocessor.c:803<br> >> taskprocessor_push: The \
'subm:devService-test-<wbr>00000038' task processor queue<br> >> \
reached 500 scheduled tasks.<br> >> [Jun 17 12:00:21] \
WARNING[26298][C-000000a3]: taskprocessor.c:803<br> >> taskprocessor_push: The \
'subp:PJSIP/sippeer-00000022' task processor queue<br> >> reached 500 \
scheduled tasks.<br> >> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: \
taskprocessor.c:803<br> >> taskprocessor_push: The \
'subm:ast_channel_topic_all-<wbr>cached-00000032' task<br> >> processor \
queue reached 500 scheduled tasks.<br> >> [Jun 17 12:01:32] \
WARNING[31697][C-000003b2]: taskprocessor.c:803<br> >> taskprocessor_push: The \
'subm:ast_channel_topic_all-<wbr>00000036' task processor<br> >> queue \
reached 500 scheduled tasks.<br> >> [Jun 17 12:05:55] WARNING[23280]: \
taskprocessor.c:803 taskprocessor_push:<br> >> The 'SIP' task processor \
queue reached 500 scheduled tasks.<br> >><br>
>> I have not seen a configuration setting on Asterisk to prevent these<br>
>> warnings from occurring (I'm trying to avoid modifying Asterisk source \
code<br> >> if possible). Looking at the task processors, I see the queue to \
the stasis<br> >> app bottlenecks:<br>
>> subm:devService-test-00000038 4560990 \
0<br> >> 1041689. It does clear up relatively quickly. The CDR engine also \
bottle<br> >> necks (extremely badly), but I don't use that. Nothing else \
comes close to<br> >> having a large queue.<br>
>><br>
>> The stasis app itself is extremely streamlined and is very capable of<br>
>> handling a large number of messages at a time. The app runs with the JVM \
so<br> >> I am also researching into that as well as the netty library I am \
using for<br> >> the websocket connections.<br>
>><br>
>> Any insight into Asterisk's side of the equation and how it scales on \
40<br> >> vCPUs would be greatly appreciated.<br>
><br>
><br>
> There are no options to disable those taskprocessor queue size warnings.<br>
> They are a<br>
> symptom of the system being severely stressed. If the stress continues it<br>
> is possible<br>
> that the system could consume all memory in those taskprocessor queues.<br>
><br>
> Recent changes to the Asterisk v13 branch were made to help throttle back<br>
> incoming<br>
> SIP requests on PJSIP when the taskprocessors become backlogged like you are<br>
> seeing.<br>
> These changes will be in the forthcoming v13.10.0 release. If you want, \
you<br> > can test with<br>
> the current v13 branch to see how these changes affect your stress testing.<br>
><br>
> If you don't need CDR's then you really need to disable them as they \
consume<br> > a lot of<br>
> processing time and the CDR taskprocessor queue backlog can take minutes to<br>
> clear.<br>
><br>
<br>
</div></div>To echo what Richard said, because Asterisk is now sharing state<br>
across the Stasis message bus, turning off subscribers to that bus<br>
will help performance. Some easy ones to disable, if you aren't using<br>
them, are CDRs, CEL, and AMI. Those all do a reasonable amount of<br>
processing, and you can get some noticeable improvement by disabling<br>
them.<br>
<br>
Once you get past that, you can start fiddling with some of the lower<br>
level options. To start, you can throttle things back further by<br>
disabling certain internal messages in stasis.conf. As stasis.conf<br>
notes, functionality within Asterisk can break (or just not happen) if<br>
some messages are removed. For example, disabling<br>
'ast_channel_snapshot_type' would break ... most things. You may<br>
however be able to streamline your application by looking at what ARI<br>
messages it cares about, what messages it doesn't, inspecting the<br>
code, and disabling those that you don't care about. Lots of testing<br>
should occur before doing this, of course.<br>
<br>
You may also be able to get some different performance characteristics<br>
by changing the threadpool options for the message bus in stasis.conf.<br>
This may make a difference, depending on the underlying machine.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Matthew Jordan<br>
Digium, Inc. | CTO<br>
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA<br>
Check us out at: <a href="http://digium.com" rel="noreferrer" \
target="_blank">http://digium.com</a> & <a href="http://asterisk.org" \
rel="noreferrer" target="_blank">http://asterisk.org</a><br> <br>
--<br>
______________________________<wbr>______________________________<wbr>_________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" \
rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br> <br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" rel="noreferrer" \
target="_blank">http://lists.digium.com/<wbr>mailman/listinfo/asterisk-dev</a><br> \
</font></span></blockquote></div><br></div>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic