'Re: Errors under high-load (ActiveMQ 5.3 + JBoss 5.1)'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-users
Subject:    Re: Errors under high-load (ActiveMQ 5.3 + JBoss 5.1)
From:       Gary Tully <gary.tully () gmail ! com>
Date:       2009-12-21 16:38:21
Message-ID: 3a73c17c0912210838jb7c274am22e52b23e678d6b0 () mail ! gmail ! com
[Download RAW message or body]


please file issues but the key component will be some sort of test case that
makes it easy to reproduce.

2009/12/21 Leonardo Lima <leomrlima@gmail.com>

> Hello.
> 
> As I continue to try and debug my problem, I see that both errors are
> because of ActiveMQ.
> 
> There are three kinds of stacktraces from ActiveMQ Logs:
> 
> 2009-12-21 12:19:38,296 | WARN  | POST COMMIT FAILED:  |
> org.apache.activemq.transaction.XATransaction | NIO Transport Thread
> java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> 
> org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
> at
> 
> org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
> at
> org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
> at
> 
> org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
>  at
> 
> org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
> at
> 
> org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
>  at
> 
> org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
>  at
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
> at
> 
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>  at
> 
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>  at
> 
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>  at
> 
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>  at
> 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
> at
> 
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>  at
> 
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 
> 2009-12-21 12:17:10,281 | WARN  | Async error occurred:
> javax.transaction.xa.XAException: Transaction
> 
> 'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
>  has not been started. |
> org.apache.activemq.broker.TransportConnection.Service | NIO Transport
> Thread
> javax.transaction.xa.XAException: Transaction
> 
> 'XID:131075:312d2d33663537633766663a663839353a34623266326262613a313663653763:2d33663537633766663a663839353a34623266326262613a313663653764'
>  has not been started.
> at
> 
> org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:266)
>  at
> 
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:208)
> at
> 
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>  at
> 
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> at
> 
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:443)
>  at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
> at
> 
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>  at
> 
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>  at
> 
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>  at
> 
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>  at
> 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
> at
> 
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>  at
> 
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 
> 2009-12-20 19:45:50,406 | WARN  | Async error occurred:
> java.lang.ArrayIndexOutOfBoundsException: -1 |
> org.apache.activemq.broker.TransportConnection.Service | NIO Transport
> Thread
> java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.get(ArrayList.java:324)
> at
> 
> org.apache.activemq.store.kahadb.KahaDBStore.createTransactionInfo(KahaDBStore.java:552)
>  at
> 
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:152)
>  at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:609)
> at
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1237)
> at
> 
> org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:60)
>  at
> 
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:250)
>  at
> 
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:369)
>  at
> 
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:470)
> at
> 
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>  at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> at
> 
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
>  at
> 
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
>  at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> at
> 
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
>  at
> 
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
>  at
> 
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>  at
> 
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>  at
> 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
> at
> 
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
> at
> 
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
>  at
> 
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 
> I'm thinking about filing bugs for each of one, should I?
> 
> Thank you very much.
> Leonardo.
> 
> On Mon, Dec 21, 2009 at 7:35 AM, Leonardo Lima <leomrlima@gmail.com>
> wrote:
> 
> > Hello.
> > 
> > I'm stress-testing this scenario: about 15000 messages are sent in about
> 10
> > min (~25 msg/sec) to a standalone broker (using nio transport). Messages
> are
> > consumed via MDB in a Jboss 5.1, configured with the activemq-rar
> > instructions found on the website (
> > http://activemq.apache.org/integrating-apache-activemq-with-jboss.html).
> > The MDBs are also inserting data in a database, and everything is under
> XA
> > (or so it seems from the errors...). I'm running everything in a Windows
> > 2008 R2 and Sun's 64-bit JVM.
> > 
> > Everything seems to run fine, but after the messages are sent, some (~50
> > messages) about are "lingering" in the queue (Number Of Pending Messages
> is
> > bigger than 0). But no one consumer is getting messages anymore and I
> can't
> > browse them.
> > 
> > Checking for errors, I see two issues. Sometimes, I get a
> > "javax.transaction.xa.XAException: Transaction has not been started.".
> That
> > shows at both JBoss and ActiveMQ. I've searched about it and found little
> > info on how to debug it. I think it must be a configuration mismatch
> about
> > quantities of avaiable MDBs/Connections/Transaction/Pool or something
> like
> > that, but I don't know. And I believe ActiveMQ would retry to dispatch
> this
> > message again, so this wouldn't be a big deal.
> > 
> > And sometimes I get a "POST COMMIT FAILED"
> > java.util.ConcurrentModificationException at ActiveMQ. I couldn't find
> this
> > error anywhere else, and I don't know what else can I do about it. Is it
> a
> > bug? Maybe it has something to do with this pending messages. Here goes
> the
> > stacktrace:
> > 
> > 2009-12-21 07:05:31,522 | WARN  | POST COMMIT FAILED:  |
> > org.apache.activemq.transaction.XATransaction | NIO Transport Thread
> > java.util.ConcurrentModificationException
> > at
> > java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> > at java.util.AbstractList$Itr.next(AbstractList.java:343)
> > at
> > 
> org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
> > at
> > 
> org.apache.activemq.transaction.XATransaction.doPostCommit(XATransaction.java:113)
> > at
> > 
> org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:68)
> > at
> > 
> org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
> 
> > at
> > 
> org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:94)
> > at
> > 
> org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:101)
> 
> > at
> > 
> org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:406)
> 
> > at
> > 
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
> > at
> > 
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
> 
> > at
> > 
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
> 
> > at
> > 
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> > at
> > 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> 
> > at
> > 
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> 
> > at
> > 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> > at
> > 
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:128)
> > at
> > 
> org.apache.activemq.transport.nio.NIOTransport.access$000(NIOTransport.java:44)
> > at
> > 
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:68)
> > at
> > 
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:78)
> 
> > at
> > 
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:103)
> > at
> > 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> > 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > 
> > My main concern is that I don't know if there really is something pending
> > and how should I avoid this.
> > 
> > Could you please take a look at this and give me pointers on how to
> > proceed?
> > 
> > Thank you very much.
> > Leo
> > 
> 



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com



[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic