[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-dev
Subject:    [jira] [Updated] (AMQ-5202) Default cursor exceeds allocated memory on broker restart
From:       "Jakub Korab (JIRA)" <jira () apache ! org>
Date:       2014-05-28 14:03:02
Message-ID: JIRA.12717048.1401285516736.31236.1401285782210 () arcas
[Download RAW message or body]


     [ https://issues.apache.org/jira/browse/AMQ-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel \
]

Jakub Korab updated AMQ-5202:
-----------------------------

    Description: 
This seems to be related to solution that was implemented in AMQ-4495.

Background:
Publishing to queue:one at a high rate (loop, 10 producers), no consumers
Publishing to queue:two at a low rate (2 msg/s), 1 consumer

When using a fresh broker (no data directory) with LevelDB/KahaDB:
Memory usage + cursor usage rise together on queue:one until they hit 69%; this \
corresponds to disk usage of 90mb (69% of tempUsage - under 70% default memory \
watermark for the cursor); queue:one keeps accepting messages.
queue:two keeps accepting messages, and handing them off to the consumer
This is reached when ~10000 * 4kb messages are sent. All OK so far.

Issue:
Restart the broker.
Memory usage + cursor usage rise together very slowly on queue:one until they hit 70% \
(default policyEntry@cursorMemoryHighWaterMark) queue:one keeps accepting messages.
queue:two keeps accepting messages, but _stops handing them off to the consumer_

It doesn't matter whether the number of messages in the store is under or over the \
cursor watermark when the broker is restarted - the cursor usage hits the limit and \
the messages stop flowing.

Consuming some messages from queue:one at this point will free up space up on the \
cursor, and messages start flowing through to the queue:two consumers. The cursor \
will eventually hit the watermark again and the consumers will again block, so it is \
not a self-healing issue.

Deleting the queues does not change that situation, and neither consumers on \
queue:one nor queue:two will receive messages. The cursor seems to keep a handle on \
the old queue:one's reference.

  was:
This seems to be related to solution that was implemented in AMQ-4495.

Background:
Publishing to queue:one at a high rate (loop, 10 producers), no consumers
Publishing to queue:two at a low rate (2 msg/s), 1 consumer

When using a fresh broker (no data directory) with LevelDB/KahaDB:
Memory usage + cursor usage rise together on queue:one until they hit 69%; this \
corresponds to disk usage of 90mb (69% of tempUsage - under 70% default memory \
watermark for the cursor); queue:one keeps accepting messages.
queue:two keeps accepting messages, and handing them off to the consumer
This is reached when ~10000 * 4kb messages are sent. All OK so far.

Issue:
Restart the broker.
Memory usage + cursor usage rise together very slowly on queue:one until they hit 70% \
(default policyEntry@cursorMemoryHighWaterMark) queue:one keeps accepting messages.
queue:two keeps accepting messages, but _stops handing them off to the consumer_

It doesn't matter whether the number of messages in the store is under or over the \
cursor watermark when the broker is restarted - the cursor usage hits the limit and \
the messages stop flowing.

Consuming some messages from queue:one at this point will free up space up on the \
cursor, and messages start flowing through to the queue:two consumers. The cursor \
will eventually hit the watermark again and the consumers will again block, so it is \
not a self-healing issue.


> Default cursor exceeds allocated memory on broker restart
> ---------------------------------------------------------
> 
> Key: AMQ-5202
> URL: https://issues.apache.org/jira/browse/AMQ-5202
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.0, 5.9.1
> Environment: Linux, OSX
> Reporter: Jakub Korab
> 
> This seems to be related to solution that was implemented in AMQ-4495.
> Background:
> Publishing to queue:one at a high rate (loop, 10 producers), no consumers
> Publishing to queue:two at a low rate (2 msg/s), 1 consumer
> When using a fresh broker (no data directory) with LevelDB/KahaDB:
> Memory usage + cursor usage rise together on queue:one until they hit 69%; this \
> corresponds to disk usage of 90mb (69% of tempUsage - under 70% default memory \
> watermark for the cursor); queue:one keeps accepting messages.
> queue:two keeps accepting messages, and handing them off to the consumer
> This is reached when ~10000 * 4kb messages are sent. All OK so far.
> Issue:
> Restart the broker.
> Memory usage + cursor usage rise together very slowly on queue:one until they hit \
> 70% (default policyEntry@cursorMemoryHighWaterMark) queue:one keeps accepting \
> messages. queue:two keeps accepting messages, but _stops handing them off to the \
> consumer_ It doesn't matter whether the number of messages in the store is under or \
> over the cursor watermark when the broker is restarted - the cursor usage hits the \
> limit and the messages stop flowing. Consuming some messages from queue:one at this \
> point will free up space up on the cursor, and messages start flowing through to \
> the queue:two consumers. The cursor will eventually hit the watermark again and the \
> consumers will again block, so it is not a self-healing issue. Deleting the queues \
> does not change that situation, and neither consumers on queue:one nor queue:two \
> will receive messages. The cursor seems to keep a handle on the old queue:one's \
> reference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[prev in list] [next in list] [prev in thread] [next in thread]