[prev in list] [next in list] [prev in thread] [next in thread]
List: wsf-javascript-dev
Subject: Re: [Dev] [APIM-2.0.0-Cluster] continuous PoolExhaustedException on APIM and keymanager node
From: Isabelle Mauny <isabelle () wso2 ! com>
Date: 2016-06-30 14:32:01
Message-ID: CAHR-JBB+NsmDtLHe_y2uOg4pTqBYRMuSMMz_Z3pkZ1HBBm6ucw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
Increasing the pool only makes it fail later.. The connections are not
freed apparently. Who holds them ? Can't you see that in JMX console ?
-------------------------------------------------------------------------------------
*Isabelle Mauny*
VP, Product Strategy - WSO2, Inc. - http://wso2.com/
On Thu, Jun 30, 2016 at 11:12 AM, Chandana Napagoda <chandana@wso2.com>
wrote:
> Hi Chalitha,
>
> You can disable the indexing in the APIM GW and KM nodes by setting
> "startIndexing" parameter as false. This "startIndexing" parameter should
> be configured in the registry.xml file under "indexingConfiguration"
> section.
>
> Anyway, it's better to check the DB query execution time by using log4jdbc
> driver.
>
> Regards,
> Chandana
>
> On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage <chalitha@wso2.com>
> wrote:
>
> > Hi Abimaran,
> >
> > As per Danesh comment, I have increase the max active connection till
> > 120. Still i am seeing this error printed continuously. Do we need to
> > increase the value further
> >
> >
> > [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start new
> > registry transaction.
> > org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10]
> > Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none
> > available[size:120; busy:120; idle:0; lastwait:60000].
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
> > at
> > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
> > at
> > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
> > at
> > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
> > at
> > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
> > at
> > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
> > at
> > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)
> > at
> > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing.
> > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> > start new registry transaction.
> > at
> > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)
> > at
> > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
> > at
> > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
> > at
> > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
> > at
> > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)
> > at
> > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
> > [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in
> > 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000].
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
> > at
> > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
> > at
> > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
> >
> > ... 16 more
> >
> >
> >
> >
> > On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <abimaran@wso2.com>
> > wrote:
> >
> > > Hi Chalitha,
> > >
> > > Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
> > > [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60
> > > seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
> > >
> > > From the above logs, the problem lies with the database connection which
> > > has pool size of 50. So it should be carbon db, because for other dbsm you
> > > are having 100 as the pool size. You need to increase it as well.
> > >
> > > On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage <chalitha@wso2.com
> > > > wrote:
> > >
> > > > Hi Danesh,
> > > >
> > > > Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100
> > > > which are pointed to oracle database. But for the carbon db we didn't
> > > > change it since it's local h2.
> > > > Do we need to change it also?
> > > >
> > > > On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <danesh@wso2.com>
> > > > wrote:
> > > >
> > > > > Hi Chalitha,
> > > > >
> > > > > Can you increase the maxActive in Registry DB configuration. Seems
> > > > > like there are no available connections in tomcat jdbc pool. Please set
> > > > > this to 80 at least.
> > > > >
> > > > > Thanks
> > > > > Danesh
> > > > >
> > > > > On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage <
> > > > > chalitha@wso2.com> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > We have setup an API Manager cluster (beta 15th pack) fronted by
> > > > > > nginx with following details.
> > > > > >
> > > > > > Publisher
> > > > > > Two store nodes
> > > > > > Gateway Cluster(one manager with two workers)
> > > > > > IS as key manager(two node cluster)
> > > > > > DB-Oracle 12c
> > > > > > Userstore-Read/write ldap
> > > > > >
> > > > > > We have ran a long running test for some days and observed out of
> > > > > > memory on one of the IS nodes. Once we have cleaned all created heap
> > > > > > dumps,logs and try to restart the cluster again and found the following
> > > > > > error continuously on publisher, gateway and IS nodes.
> > > > > >
> > > > > > I have attached the current database running process details with
> > > > > > [1]. Further we have used the [2] for fine tunning the servers.
> > > > > >
> > > > > > Appreciate your feedback on resolving this.
> > > > > >
> > > > > >
> > > > > >
> > > > > > ERROR - TransactionManager Failed to start new registry transaction.
> > > > > > org.apache.tomcat.jdbc.pool.PoolExhaustedException:
> > > > > > [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in \
> > > > > > 60 seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
> > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
> > > > > > at
> > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing.
> > > > > > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> > > > > > start new registry transaction.
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
> > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
> > > > > > at
> > > > > > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
> > > > > > at
> > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
> > > > > > [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in \
> > > > > > 60 seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
> > > > > > at
> > > > > > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
> > > > > > at
> > > > > > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
> > > > > >
> > > > > > ... 18 more
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [2] https://docs.wso2.com/display/AM1100/Tuning+Performance
> > > > > >
> > > > > >
> > > > > > Thank you,
> > > > > > --
> > > > > > *Chalitha Maheshwari*
> > > > > > Software Engineer-QA,
> > > > > > WSO2 Inc.
> > > > > >
> > > > > > *E-mail:* chalitha@wso2.com
> > > > > > *Mobile: *+94710 411 112
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Dev mailing list
> > > > > > Dev@wso2.org
> > > > > > http://wso2.org/cgi-bin/mailman/listinfo/dev
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Danesh Kuruppu
> > > > > Software Engineer
> > > > > WSO2 Inc,
> > > > > Mobile: +94 (77) 1690552
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Chalitha Maheshwari*
> > > > Software Engineer-QA,
> > > > WSO2 Inc.
> > > >
> > > > *E-mail:* chalitha@wso2.com
> > > > *Mobile: *+94710 411 112
> > > >
> > > > _______________________________________________
> > > > Dev mailing list
> > > > Dev@wso2.org
> > > > http://wso2.org/cgi-bin/mailman/listinfo/dev
> > > >
> > > >
> > >
> > >
> > > --
> > > Thanks
> > > Abimaran Kugathasan
> > > Senior Software Engineer
> > >
> > > Email : abimaran@wso2.com
> > > Mobile : +94 773922820
> > >
> > > <http://stackoverflow.com/users/515034>
> > > <http://lk.linkedin.com/in/abimaran>
> > > <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank>
> > > <https://twitter.com/abimaran>
> > >
> > >
> >
> >
> > --
> > *Chalitha Maheshwari*
> > Software Engineer-QA,
> > WSO2 Inc.
> >
> > *E-mail:* chalitha@wso2.com
> > *Mobile: *+94710 411 112
> >
> > _______________________________________________
> > Dev mailing list
> > Dev@wso2.org
> > http://wso2.org/cgi-bin/mailman/listinfo/dev
> >
> >
>
>
> --
> *Chandana Napagoda*
> Senior Software Engineer
> WSO2 Inc. - http://wso2.org
>
> *Email : chandana@wso2.com <chandana@wso2.com>**Mobile : +94718169299
> <%2B94718169299>*
>
> *Blog : http://cnapagoda.blogspot.com <http://cnapagoda.blogspot.com>*
>
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
[Attachment #7 (text/html)]
<div dir="ltr"><div class="gmail_default" style="font-family:comic sans \
ms,sans-serif;font-size:small;color:#444444">Increasing the pool only makes it fail \
later.. The connections are not freed apparently. Who holds them ? Can't you see \
that in JMX console ? </div><div class="gmail_extra"><br clear="all"><div><div \
class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font size="1"><span \
style="font-family:comic sans \
ms,sans-serif">-------------------------------------------------------------------------------------<br><b>Isabelle \
Mauny</b><br>VP, Product Strategy - WSO2, Inc. - <a href="http://wso2.com/" \
target="_blank">http://wso2.com/</a><br><br></span></font></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Jun 30, 2016 at 11:12 AM, Chandana Napagoda \
<span dir="ltr"><<a href="mailto:chandana@wso2.com" \
target="_blank">chandana@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>Hi Chalitha,<br><br>You can disable the \
indexing in the APIM GW and KM nodes by setting "startIndexing" parameter \
as false. This "startIndexing" parameter should be configured in the \
registry.xml file under "indexingConfiguration" section.<br><br>Anyway, \
it's better to check the DB query execution time by using log4jdbc \
driver.<br></div><div><br>Regards,<br>Chandana<span \
style="color:rgb(51,51,51);font-family:Arial,serif;font-size:14.3px;font-style:normal; \
font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.88px;text- \
align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;dis \
play:inline!important;float:none;background-color:rgb(255,255,255)"></span></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 30, 2016 at 2:29 PM, \
Chalitha Waldeniyage <span dir="ltr"><<a href="mailto:chalitha@wso2.com" \
target="_blank">chalitha@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>Hi Abimaran,<br><br></div>As per Danesh \
comment, I have increase the max active connection till 120. Still i am seeing this \
error printed continuously. Do we need to increase the value \
further<br><br><br>[2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to \
start new registry transaction.<br>org.apache.tomcat.jdbc.pool.PoolExhaustedException: \
[pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, \
none available[size:120; busy:120; idle:0; lastwait:60000].<span><br> at \
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)<br> \
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)<br> \
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)<br> \
at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)<br></span> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)<br> \
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)<br> \
at java.security.AccessController.doPrivileged(Native Method)<br> at \
org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)<span><br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)<br> \
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> \
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> \
at java.lang.Thread.run(Thread.java:745)<br></span>[2016-06-30 08:58:28,803] ERROR - \
AsyncIndexer Error while \
indexing.<span><br>org.wso2.carbon.registry.core.exceptions.RegistryException: Failed \
to start new registry transaction.<br> at \
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)<br></span> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)<br> \
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)<br> \
at java.security.AccessController.doPrivileged(Native Method)<br> at \
org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)<span><br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)<br> \
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> \
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> \
at java.lang.Thread.run(Thread.java:745)<br></span>Caused by: \
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10] Timeout: \
Pool empty. Unable to fetch a connection in 60 seconds, none available[size:120; \
busy:120; idle:0; lastwait:60000].<span><br> at \
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)<br> \
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)<br> \
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)<br> \
at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)<br></span> \
... 16 more<br><br><br><br></div><div><div><div class="gmail_extra"><br><div \
class="gmail_quote">On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <span \
dir="ltr"><<a href="mailto:abimaran@wso2.com" \
target="_blank">abimaran@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Chalitha,<span><div><br></div><div><span \
style="font-size:12.8px">Caused by: org.apache.tomcat.jdbc.pool.</span><span \
style="font-size:12.8px">PoolExhaustedException: [pool-216-thread-7] Timeout: Pool \
empty. Unable to fetch a connection in 60 seconds, none available[size:50; busy:50; \
idle:0; lastwait:60000].</span><br></div><div><br></div></span><div>From the above \
logs, the problem lies with the database connection which has pool size of 50. So it \
should be carbon db, because for other dbsm you are having 100 as the pool size. You \
need to increase it as well.</div></div><div class="gmail_extra"><div><div><br><div \
class="gmail_quote">On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage <span \
dir="ltr"><<a href="mailto:chalitha@wso2.com" \
target="_blank">chalitha@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>Hi Danesh,<br><br></div>Yes, We have done \
that on Registry,AM,UM,Conf,Gov databases up to 100 which are pointed to oracle \
database. But for the carbon db we didn't change it since it's local h2. \
<br>Do we need to change it also?<br></div><div><div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 30, 2016 at 1:34 PM, \
Danesh Kuruppu <span dir="ltr"><<a href="mailto:danesh@wso2.com" \
target="_blank">danesh@wso2.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Chalitha,<br><br></div>Can \
you increase the maxActive in Registry DB configuration. Seems like there are no \
available connections in tomcat jdbc pool. Please set this to 80 at \
least.<br><br></div>Thanks<br></div>Danesh <br></div><div \
class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Jun 30, 2016 at \
8:29 AM, Chalitha Waldeniyage <span dir="ltr"><<a href="mailto:chalitha@wso2.com" \
target="_blank">chalitha@wso2.com</a>></span> wrote:<br></div></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div><div><div dir="ltr"><div>Hi All,<br><br></div>We have \
setup an API Manager cluster (beta 15th pack) fronted by nginx with following \
details.<br><br><div><div><div>Publisher<br></div><div>Two store \
nodes<br></div><div>Gateway Cluster(one manager with two workers)<br></div><div>IS as \
key manager(two node cluster)<br></div><div>DB-Oracle \
12c<br></div><div>Userstore-Read/write ldap<br><br></div><div>We have ran a long \
running test for some days and observed out of memory on one of the IS nodes. Once we \
have cleaned all created heap dumps,logs and try to restart the cluster again and \
found the following error continuously on publisher, gateway and IS nodes. <br> \
<br>I have attached the current database running process details with [1]. Further we \
have used the [2] for fine tunning the servers.<br><br></div><div>Appreciate your \
feedback on resolving this.<br><br><br><br></div><div>ERROR - TransactionManager \
Failed to start new registry \
transaction.<br>org.apache.tomcat.jdbc.pool.PoolExhaustedException: \
[pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, \
none available[size:50; busy:50; idle:0; lastwait:60000].<br> at \
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)<br> \
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)<br> \
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)<br> \
at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)<br> \
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)<br> \
at java.security.AccessController.doPrivileged(Native Method)<br> at \
org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)<br> \
at org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)<br> \
at org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)<br> \
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> \
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> \
at java.lang.Thread.run(Thread.java:745)<br>[2016-06-30 06:28:48,855] ERROR - \
AsyncIndexer Error while \
indexing.<br>org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to \
start new registry transaction.<br> at \
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)<br> \
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)<br> \
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)<br> \
at org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)<br> \
at java.security.AccessController.doPrivileged(Native Method)<br> at \
org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)<br> \
at org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)<br> \
at org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)<br> \
at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)<br> \
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> \
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> \
at java.lang.Thread.run(Thread.java:745)<br>Caused by: \
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-216-thread-7] Timeout: Pool \
empty. Unable to fetch a connection in 60 seconds, none available[size:50; busy:50; \
idle:0; lastwait:60000].<br> at \
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)<br> \
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)<br> \
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)<br> \
at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)<br> \
... 18 more<br><br><br>[1] <br><br><img src="cid:ii_iq1yg45e1_155a015a108ea27e" \
height="245" width="533"><br><br><br><br><br>[2] <a \
href="https://docs.wso2.com/display/AM1100/Tuning+Performance" \
target="_blank">https://docs.wso2.com/display/AM1100/Tuning+Performance</a><br><br><br></div><div>Thank \
you,<span><font color="#888888"><br></font></span></div><span><font \
color="#888888"><div>-- <br><div data-smartmail="gmail_signature"><div \
dir="ltr"><span><font color="#888888"><div dir="ltr"><font \
color="#888888"><div><b>Chalitha Maheshwari</b><br></div><div><div>Software \
Engineer-QA, </div>
<div>WSO2 Inc.</div></div><div><br></div><div><b>E-mail:</b> <a \
href="mailto:chalitha@wso2.com" \
target="_blank"><span>chalitha@wso2.com</span></a><br></div><div><b>Mobile: </b><a \
href="tel:%2B94710%20411%20112" value="+94710411112" target="_blank">+94710 411 \
112</a><br><br><br></div> </font></div></font></span></div></div>
</div></font></span></div></div></div>
<br></div></div>_______________________________________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailman/listinfo/dev</a><br> \
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- \
<br><div data-smartmail="gmail_signature"><div dir="ltr"><div>
<p style="margin-bottom:0in"><span style="font-family:tahoma,sans-serif"><font \
color="#888888">Danesh Kuruppu</font></span></p><span \
style="font-family:tahoma,sans-serif"><font color="#888888">Software Engineer<br>WSO2 \
Inc,</font><font color="#888888"><br></font><font color="#888888">Mobile: <a \
href="tel:%2B94%20%2877%29%201690552" value="+94771690552" target="_blank">+94 (77) \
1690552</a></font></span><span style="font-family:georgia,serif"><font \
color="#888888"><br></font></span></div><span \
style="font-family:georgia,serif"></span><span \
style="font-family:georgia,serif"></span>
</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div \
data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><div \
dir="ltr"><font color="#888888"><div><b>Chalitha \
Maheshwari</b><br></div><div><div>Software Engineer-QA, </div>
<div>WSO2 Inc.</div></div><div><br></div><div><b>E-mail:</b> <a \
href="mailto:chalitha@wso2.com" \
target="_blank"><span>chalitha@wso2.com</span></a><br></div><div><b>Mobile: </b><a \
href="tel:%2B94710%20411%20112" value="+94710411112" target="_blank">+94710 411 \
112</a><br></div> </font></div></font></span></div></div>
</div>
</div></div><br>_______________________________________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailman/listinfo/dev</a><br> \
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br></div></div><div \
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div dir="ltr"><div>Thanks</div><div>Abimaran \
Kugathasan</div><div>Senior Software Engineer</div><div><br></div><div>Email : <a \
href="mailto:abimaran@wso2.com" target="_blank">abimaran@wso2.com</a></div><div><font \
face="comic sans ms, sans-serif">Mobile : <a href="tel:%2B94%C2%A0773922820" \
value="+94773922820" target="_blank">+94 \
773922820</a></font></div><div><br></div><div><a \
href="http://stackoverflow.com/users/515034" target="_blank"><img \
src="https://cdn1.iconfinder.com/data/icons/D_Social_Icon_Set/32/stackoverflowpx.png"></a> \
<a href="http://lk.linkedin.com/in/abimaran" target="_blank"><img \
src="https://cdn3.iconfinder.com/data/icons/free-social-icons/67/linkedin_square_color-32.png"></a> \
<a href="http://www.lkabimaran.blogspot.com/" target="_blank"><img \
src="https://cdn1.iconfinder.com/data/icons/google_jfk_icons_by_carlosjj/32/blogger.png"></a> \
<a href="https://github.com/abimarank" target="_blank"><img \
src="https://cdn1.iconfinder.com/data/icons/D_Social_Icon_Set/32/githubpx.png" \
alt=""></a> <a href="https://twitter.com/abimaran" target="_blank"><img \
src="https://cdn3.iconfinder.com/data/icons/free-social-icons/67/twitter_square-32.png \
"></a><br></div><div><br></div></div></div></div></div></div></div></div></div></div> \
</div> </blockquote></div><br><br clear="all"><br>-- <br><div \
data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><div \
dir="ltr"><font color="#888888"><div><b>Chalitha \
Maheshwari</b><br></div><div><div>Software Engineer-QA, </div>
<div>WSO2 Inc.</div></div><div><br></div><div><b>E-mail:</b> <a \
href="mailto:chalitha@wso2.com" \
target="_blank"><span>chalitha@wso2.com</span></a><br></div><div><b>Mobile: </b><a \
href="tel:%2B94710%20411%20112" value="+94710411112" target="_blank">+94710 411 \
112</a><br></div> </font></div></font></span></div></div>
</div>
</div></div><br>_______________________________________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org" target="_blank">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailman/listinfo/dev</a><br> \
<br></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br \
clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div \
dir="ltr"><div><div><div><b>Chandana Napagoda</b><br>Senior Software Engineer<br>WSO2 \
Inc. - <a href="http://wso2.org" target="_blank">http://wso2.org</a><br><i>Email : \
<a href="mailto:chandana@wso2.com" \
target="_blank">chandana@wso2.com</a><br></i><i>Mobile : <a href="tel:%2B94718169299" \
value="+94718169299" target="_blank">+94718169299</a></i><br></div><i>Blog : \
<a href="http://cnapagoda.blogspot.com" \
target="_blank">http://cnapagoda.blogspot.com</a><br></i></div></div><div><div><div><br></div></div></div></div></div>
</font></span></div>
<br>_______________________________________________<br>
Dev mailing list<br>
<a href="mailto:Dev@wso2.org">Dev@wso2.org</a><br>
<a href="http://wso2.org/cgi-bin/mailman/listinfo/dev" rel="noreferrer" \
target="_blank">http://wso2.org/cgi-bin/mailman/listinfo/dev</a><br> \
<br></blockquote></div><br></div></div>
--001a11492afc18882605367fbef1--
["oracledbconnections.png" (image/png)]
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic