[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-dev
Subject:    [jira] Commented: (AMQ-1918) AbstractStoreCursor.size gets out of
From:       "Richard Yarger (JIRA)" <jira () apache ! org>
Date:       2008-08-28 22:14:52
Message-ID: 2039464022.1219961692885.JavaMail.jira () brutus
[Download RAW message or body]


    [ https://issues.apache.org/activemq/browse/AMQ-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=45226#action_45226 \
] 

Richard Yarger commented on AMQ-1918:
-------------------------------------

I have a reproducible scenario for this issue.
I cannot guarantee that this is how it always happens, but AbstractStoreCursor.size \
does go negative everytime I follow this procedure. I am sorry I could not make it \
into a nice little test case. I tried to break it down as best I could.

I wrapped up my eclipse project. It has 2 consumers, and 2 producers.
producer_flood sends a larger number of messages to test.queue.1 every 3 seconds
producer_steady sends a small number of messages to test.queue.1 every 3 seconds
consumer1 consumes messages off test.queue.1 and forwards them to test.queue.2
consumer2 consumes messages off test.queue.2 and prints text to system.out
My broker config file is included in the amq_broker_config dir. It sets the memory \
limits low so that they can be reached faster. The queues are setup for 1MB each and \
the broker is setup for 3MB.

Here is the script:
1) Start a clean broker
2) Start consumer1 and consumer2
3) Start producer_flood - this will push messages in faster than consumer1 can \
handle.  Let the queue size build up to 1000 messages. This should be 100% of the 1MB \
allowed for that queue. 4) Stop consumer2 - this will allow messages to build up in \
queue2.  If you watch this queue in a jconsole, you will notice that the percent \
memory used does not rise, even after you pass the memory limit. Let the queue2 size \
grow to around 2000. This will put you near the memory limit for the broker.
5) Start consumer2 - if you look at jconsole now, the percent memory used is updated \
and > 100%. 6) Wait for queue2 to fall below 1000 messages and stop the producer. Let \
the messages drain and one or both of the queues should now have negative counts.  7) \
If you start the producer_steady now you'll notice that messages do not reach \
consumer2 at the rate that they go in.  If you debug the broker now and look at \
AbstractStoreCursor.size, it will be negative.   
Please let me know if you need more info.
Thanks.

> AbstractStoreCursor.size gets out of synch with Store size and blocks consumers
> -------------------------------------------------------------------------------
> 
> Key: AMQ-1918
> URL: https://issues.apache.org/activemq/browse/AMQ-1918
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.1.0
> Reporter: Richard Yarger
> Priority: Critical
> Attachments: activemq.xml
> 
> 
> In version 5.1.0, we are seeing our queue consumers stop consuming for no reason.
> We have a staged queue environment and we occasionally see one queue display \
> negative pending message counts that hang around -x, rise to -x+n gradually and \
> then fall back to -x abruptly. The messages are building up and being processed in \
> bunches but its not easy to see because the counts are negative. We see this \
> behavior in the messages coming out of the system. Outbound messages come out in \
> bunches and are synchronized with the queue pending count dropping to -x. This \
> issue does not happen ALL of the time. It happens about once a week and the only \
> way to fix it is to bounce the broker. It doesn't happen to the same queue \
> everytime, so it is not our consuming code. Although we don't have a reproducible \
> scenario, we have been able to debug the issue in our test environment. We traced \
> the problem to the cached store size in the AbstractStoreCursor. This value becomes \
> 0 or negative and prevents the AbstractStoreCursor from retrieving more messages \
> from the store. (see AbstractStoreCursor.fillBatch() ) We have seen size value go \
> lower than -1000. We have also forced it to fix itself by sending in n+1 messages. \
> Once the size goes above zero, the cached value is refreshed and things work ok \
> again. Unfortunately, during low volume times, it could be hours before n+1 \
> messages are received, so our message latency can rise during low volume times.... \
> :( I have attached our broker config.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[prev in list] [next in list] [prev in thread] [next in thread]