[prev in list] [next in list] [prev in thread] [next in thread]
List: wsas-java-dev
Subject: Re: [Dev] New data publisher is hanging
From: Sinthuja Ragendran <sinthuja () wso2 ! com>
Date: 2015-08-31 17:01:44
Message-ID: CAA=iADXTRTcq=HVthZEcsihhDrkbTv0SLtK16GH2BTWHxZ-E5A () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi maninda,
Ok, thanks for information. I'll do the test locally and get back to you.
Thanks,
Sinthuja.
On Mon, Aug 31, 2015 at 9:53 AM, Maninda Edirisooriya <maninda@wso2.com>
wrote:
> Hi Sinthuja,
>
> I tested with smart-home sample in latest DAS with [1] config and DAS with
> the attached config directory. (There data-bridge-config.xml is as [2])
> I did the test on EC2 instances with MySQL RDS instance as DBs.
> This issue was always reproducible when 10M events are published with the
> sample. For some time events get published and then it will suddenly stop
> receiving events. But you can see the client is busy with the CPU usage
> while DAS is almost idling.
> No debug or logging was enabled.
>
> [1]
>
> <Agent>
> <Name>Thrift</Name>
>
> <DataEndpointClass>org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint</DataEndpointClass>
> <TrustSore>src/main/resources/client-truststore.jks</TrustSore>
> <TrustSorePassword>wso2carbon</TrustSorePassword>
> <QueueSize>32768</QueueSize>
> <BatchSize>200</BatchSize>
> <CorePoolSize>5</CorePoolSize>
> <MaxPoolSize>10</MaxPoolSize>
> <KeepAliveTimeInPool>20</KeepAliveTimeInPool>
> <ReconnectionInterval>30</ReconnectionInterval>
> <MaxTransportPoolSize>250</MaxTransportPoolSize>
> <MaxIdleConnections>250</MaxIdleConnections>
> <EvictionTimePeriod>5500</EvictionTimePeriod>
> <MinIdleTimeInPool>5000</MinIdleTimeInPool>
> <SecureMaxTransportPoolSize>250</SecureMaxTransportPoolSize>
> <SecureMaxIdleConnections>250</SecureMaxIdleConnections>
> <SecureEvictionTimePeriod>5500</SecureEvictionTimePeriod>
> <SecureMinIdleTimeInPool>5000</SecureMinIdleTimeInPool>
> </Agent>
>
> [2]
>
> <dataBridgeConfiguration>
>
> <workerThreads>10</workerThreads>
> <eventBufferCapacity>1000</eventBufferCapacity>
> <clientTimeoutMin>30</clientTimeoutMin>
>
> <dataReceiver name="Thrift">
> <config name="tcpPort">7611</config>
> <config name="sslPort">7711</config>
> </dataReceiver>
>
> <dataReceiver name="Binary">
> <config name="tcpPort">9611</config>
> <config name="sslPort">9711</config>
> <config name="sslReceiverThreadPoolSize">100</config>
> <config name="tcpReceiverThreadPoolSize">100</config>
> </dataReceiver>
>
> </dataBridgeConfiguration>
>
> Thanks.
>
>
> *Maninda Edirisooriya*
> Senior Software Engineer
>
> *WSO2, Inc.*lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *E-mail* : maninda@wso2.com
> *Skype* : @manindae
> *Twitter* : @maninda
>
> On Mon, Aug 31, 2015 at 8:08 PM, Sinthuja Ragendran <sinthuja@wso2.com>
> wrote:
>
> > Are you running with debug mode in logging? And can you constantly
> > reproduce this? Or it's intermittent?
> >
> > Please provide the publisher and receiver side configs to test this and
> > see. As I have already tested more than 10M records, I'm not sure what is
> > the case here.
> >
> > Thanks,
> > Sinthuja.
> >
> >
> > On Monday, August 31, 2015, Maninda Edirisooriya <maninda@wso2.com>
> > wrote:
> >
> > > Hi,
> > >
> > > When I started a 10M load test from Smart Home sample in DAS it runs for
> > > some time and stops receiving events suddenly.
> > > But publisher in client was running in higher CPU usage when DAS was
> > > running with very low CPU.
> > > When another data agent was spawned it started to publish correctly
> > > which was confirming that the issue is with the client side.
> > > We analyzed the thread dump and found the highest using thread is with
> > > the following stack traces when we analyzed it twice.
> > >
> > > 1.
> > > "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable
> > > [0x00007f85f250f000]
> > > java.lang.Thread.State: RUNNABLE
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:148)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94)
> > > at
> > > org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183)
> > > at
> > > org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown
> > > Source)
> > > at org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown
> > > Source)
> > >
> > > 2.
> > > "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable
> > > [0x00007f85f250f000]
> > > java.lang.Thread.State: RUNNABLE
> > > at org.apache.log4j.Category.callAppenders(Category.java:202)
> > > at org.apache.log4j.Category.forcedLog(Category.java:391)
> > > at org.apache.log4j.Category.log(Category.java:856)
> > > at
> > > org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.isActiveDataEndpointExists(DataEndpointGroup.java:264)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.access$400(DataEndpointGroup.java:46)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:155)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97)
> > > at
> > > org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94)
> > > at
> > > org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183)
> > > at
> > > org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown
> > > Source)
> > > at
> > > org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown Source)
> > >
> > >
> > > We suspect that *isActiveDataEndpointExists()* method is called in
> > > *org.wso2.carbon.analytics.eventsink.internal.queue.DataEndpointGroup*
> > > class repeatedly because the disruptor ring buffer is filled in client
> > > side. Not sure why this happens.
> > >
> > >
> > > *Maninda Edirisooriya*
> > > Senior Software Engineer
> > >
> > > *WSO2, Inc.*lean.enterprise.middleware.
> > >
> > > *Blog* : http://maninda.blogspot.com/
> > > *E-mail* : maninda@wso2.com
> > > *Skype* : @manindae
> > > *Twitter* : @maninda
> > >
> >
> >
> > --
> > Sent from iPhone
> >
>
>
--
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com
Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
[Attachment #5 (text/html)]
<div dir="ltr"><div><div><div>Hi maninda,<br><br></div>Ok, thanks for information. \
I'll do the test locally and get back to \
you.<br><br></div>Thanks,<br></div>Sinthuja.<br></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 31, 2015 at 9:53 AM, \
Maninda Edirisooriya <span dir="ltr"><<a href="mailto:maninda@wso2.com" \
target="_blank">maninda@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi \
Sinthuja,<br><br></div>I tested with smart-home sample in latest DAS with [1] config \
and DAS with the attached config directory. (There data-bridge-config.xml is as \
[2])<br></div>I did the test on EC2 instances with MySQL RDS instance as \
DBs.<br></div>This issue was always reproducible when 10M events are published with \
the sample. For some time events get published and then it will suddenly stop \
receiving events. But you can see the client is busy with the CPU usage while DAS is \
almost idling.<br></div>No debug or logging was \
enabled.<br><div><div><div><div><br>[1]<br><br> <Agent><br> \
<Name>Thrift</Name><br> \
<DataEndpointClass>org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint</DataEndpointClass><br> \
<TrustSore>src/main/resources/client-truststore.jks</TrustSore><br> \
<TrustSorePassword>wso2carbon</TrustSorePassword><br> \
<QueueSize>32768</QueueSize><br> \
<BatchSize>200</BatchSize><br> \
<CorePoolSize>5</CorePoolSize><br> \
<MaxPoolSize>10</MaxPoolSize><br> \
<KeepAliveTimeInPool>20</KeepAliveTimeInPool><br> \
<ReconnectionInterval>30</ReconnectionInterval><br> \
<MaxTransportPoolSize>250</MaxTransportPoolSize><br> \
<MaxIdleConnections>250</MaxIdleConnections><br> \
<EvictionTimePeriod>5500</EvictionTimePeriod><br> \
<MinIdleTimeInPool>5000</MinIdleTimeInPool><br> \
<SecureMaxTransportPoolSize>250</SecureMaxTransportPoolSize><br> \
<SecureMaxIdleConnections>250</SecureMaxIdleConnections><br> \
<SecureEvictionTimePeriod>5500</SecureEvictionTimePeriod><br> \
<SecureMinIdleTimeInPool>5000</SecureMinIdleTimeInPool><br> \
</Agent><br><br>[2]<br><br><dataBridgeConfiguration><br><br> \
<workerThreads>10</workerThreads><br> \
<eventBufferCapacity>1000</eventBufferCapacity><br> \
<clientTimeoutMin>30</clientTimeoutMin><br><br> <dataReceiver \
name="Thrift"><br> <config \
name="tcpPort">7611</config><br> <config \
name="sslPort">7711</config><br> \
</dataReceiver><br><br> <dataReceiver name="Binary"><br> \
<config name="tcpPort">9611</config><br> \
<config name="sslPort">9711</config><br> \
<config name="sslReceiverThreadPoolSize">100</config><br> \
<config name="tcpReceiverThreadPoolSize">100</config><br> \
</dataReceiver><br><br></dataBridgeConfiguration><br><br></div><div>Thanks.<br></div></div></div></div></div><div \
class="gmail_extra"><span class=""><br clear="all"><div><div><div \
dir="ltr"><div><div><font size="1"><b><span \
style="font-family:georgia,serif;color:rgb(102,102,102)"><br>Maninda \
Edirisooriya</span><span \
style="font-family:georgia,serif;color:rgb(102,102,102)"><span \
style="border-collapse:collapse"><font><span></span></font></span></span></b></font><br><font \
size="1"><span style="font-family:georgia,serif">Senior Software \
Engineer</span><br><span><font color="#888888"><span \
style="border-collapse:collapse"><font color="#000000"><b><span \
style="font-weight:normal"><font face="georgia, serif">WSO2, Inc.</font></span><font \
face="georgia, serif"><br></font></b><font face="georgia, \
serif">lean.enterprise.middleware.</font></font></span></font></span><br><br><b>Blog</b> \
: <a href="http://maninda.blogspot.com/" \
target="_blank">http://maninda.blogspot.com/</a><br><b>E-mail</b> : <a \
href="mailto:maninda@wso2.com" \
target="_blank">maninda@wso2.com</a><br></font></div><font size="1"><b>Skype</b> : \
@manindae<br></font></div><font size="1"><b>Twitter</b> : \
@maninda<br></font></div></div></div> <br></span><div><div class="h5"><div \
class="gmail_quote">On Mon, Aug 31, 2015 at 8:08 PM, Sinthuja Ragendran <span \
dir="ltr"><<a href="mailto:sinthuja@wso2.com" \
target="_blank">sinthuja@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Are you running with debug mode in logging? And can you \
constantly reproduce this? Or it's intermittent? <div><br></div><div>Please \
provide the publisher and receiver side configs to test this and see. As I have \
already tested more than 10M records, I'm not sure what is the case \
here.<span></span><br><div><br></div><div>Thanks,</div><div>Sinthuja.<div><div><br><br>On \
Monday, August 31, 2015, Maninda Edirisooriya <<a href="mailto:maninda@wso2.com" \
target="_blank">maninda@wso2.com</a>> wrote:<br><blockquote class="gmail_quote" \
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
dir="ltr"><div><div><div><div><div>Hi,<br><br></div>When I started a 10M load test \
from Smart Home sample in DAS it runs for some time and stops receiving events \
suddenly.<br></div>But publisher in client was running in higher CPU usage when DAS \
was running with very low CPU.<br></div>When another data agent was spawned it \
started to publish correctly which was confirming that the issue is with the client \
side.<br></div>We analyzed the thread dump and found the highest using thread is with \
the following stack traces when we analyzed it twice.<br><br>1.<br><font \
size="1">"main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable \
[0x00007f85f250f000]<br> java.lang.Thread.State: RUNNABLE<br> at \
org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:148)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94)<br> \
at org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183)<br> \
at org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown \
Source)<br> at org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown \
Source)</font><br><br>2.<br><font size="1">"main" prio=10 \
tid=0x00007f85ec00a800 nid=0x7843 runnable [0x00007f85f250f000]<br> \
java.lang.Thread.State: RUNNABLE<br> at \
org.apache.log4j.Category.callAppenders(Category.java:202)<br> at \
org.apache.log4j.Category.forcedLog(Category.java:391)<br> at \
org.apache.log4j.Category.log(Category.java:856)<br> at \
org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.isActiveDataEndpointExists(DataEndpointGroup.java:264)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.access$400(DataEndpointGroup.java:46)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:155)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97)<br> \
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94)<br> \
at org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183)<br> \
at org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown \
Source)<br> at \
org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown \
Source)</font><br><br><br></div>We suspect that <b>isActiveDataEndpointExists()</b> \
method is called in <b>org.wso2.carbon.analytics.eventsink.internal.queue.DataEndpointGroup</b> \
class repeatedly because the disruptor ring buffer is filled in client side. Not sure \
why this happens.<br><div><div><br \
clear="all"><div><div><div><div><div><div><div><div><div><div><div><div \
dir="ltr"><div><div><font size="1"><b><span \
style="font-family:georgia,serif;color:rgb(102,102,102)"><br>Maninda \
Edirisooriya</span><span \
style="font-family:georgia,serif;color:rgb(102,102,102)"><span \
style="border-collapse:collapse"><font><span></span></font></span></span></b></font><br><font \
size="1"><span style="font-family:georgia,serif">Senior Software \
Engineer</span><br><span><font color="#888888"><span \
style="border-collapse:collapse"><font color="#000000"><b><span \
style="font-weight:normal"><font face="georgia, serif">WSO2, Inc.</font></span><font \
face="georgia, serif"><br></font></b><font face="georgia, \
serif">lean.enterprise.middleware.</font></font></span></font></span><br><br><b>Blog</b> \
: <a href="http://maninda.blogspot.com/" \
target="_blank">http://maninda.blogspot.com/</a><br><b>E-mail</b> : \
<a>maninda@wso2.com</a><br></font></div><font size="1"><b>Skype</b> : \
@manindae<br></font></div><font size="1"><b>Twitter</b> : \
@maninda<br></font></div></div></div> \
</div></div></div></div></div></div></div></div></div></div></div></div> \
</blockquote></div></div></div></div><span><font color="#888888"><br><br>-- <br>Sent \
from iPhone<br> </font></span></blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div \
dir="ltr"><div><div dir="ltr"><font style="color:rgb(102,102,102);font-family:times \
new roman,serif" size="2"><b>Sinthuja Rajendran</b><br><span>Associate Technical \
Lead<br></span>WSO2, Inc.:<a href="http://wso2.com" \
target="_blank">http://wso2.com</a><br><br>Blog: <a \
href="http://sinthu-rajan.blogspot.com/" \
target="_blank">http://sinthu-rajan.blogspot.com/</a><br>Mobile: \
+94774273955<br></font><font size="1"><br></font><br></div></div></div></div> </div>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic