[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-dev
Subject:    [jira] [Created] (AMQ-4151) Duplicate non-persistent messages that are sent to a queue are either di
From:       "Stirling Chow (JIRA)" <jira () apache ! org>
Date:       2012-10-31 23:21:11
Message-ID: 493500818.53679.1351725672177.JavaMail.jiratomcat () arcas
[Download RAW message or body]

Stirling Chow created AMQ-4151:
----------------------------------

             Summary: Duplicate non-persistent messages that are sent to a queue are \
either dispatched multiple times (i.e., not detected as duplicates) or cause the \
queue size to be miscalculated.  Key: AMQ-4151
                 URL: https://issues.apache.org/jira/browse/AMQ-4151
             Project: ActiveMQ
          Issue Type: Bug
            Reporter: Stirling Chow


Symptom
=======
We have a virtual topic that is shared by a network of brokers.  Each broker has \
consumers that process the corresponding Consumer.*.VirtualTopic.> queue.

While testing the effects of memory limits on our system, we encountered AMQ-4148, \
which resulted in duplicate subscriptions being made to the virtual topic.  These \
duplicate subcriptions were attempting to enqueue the same topic message multiple \
times to the corresponding Consumer.*.VirtualTopic.>.

Although logic exists in queues to handle duplicate messages, we noticed lots of \
strange behaviour with duplicate handling.  We decided to investigate further and \
found a race condition whereby duplicate non-persistent messages that were correctly \
ignored resulted in the queue size appearing to be non-empty when in fact there were \
no messages.

While our original investigation was prompted by AMQ-4148, the test case we attached \
to this ticket does not rely on the misbehaviour caused by AMQ-4148, but can occur \
during normal operation of network bridges when there are multiple consumers for a \
virtual topic (as is the case, e.g., when conduit subscriptions are disabled).

Cause
=====
If a virtual topic has multiple consumers, then sending a message to the topic \
results in multiple dispatches of the same message (one to each consumer).  If the \
multiple consumers originate from the same remote broker (e.g., because conduit \
subscriptions are disabled), then a single message sent to the virtual topic on \
broker1 will result in multiple duplicate messages being sent to the virtual topic on \
broker2.

If broker2 has corresponding Consumer.*.VirtualTopic.> queue, then the multiple \
duplicate messages sent to the virtual topic on broker2 will result in multiple \
duplicate messages being sent to the queue.  

If the messages are non-persistent, the only logic that prevents each duplicate \
message from being dispatched to a consumer is as follows:

{code:title=Queue.java}
private PendingList doPageInForDispatch(boolean force) throws Exception {
...
        // Only add new messages, not already pagedIn to avoid multiple
        // dispatch attempts
        pagedInMessagesLock.writeLock().lock();
        try {
            if(isPrioritizedMessages()) {
                resultList = new PrioritizedPendingList();
            } else {
                resultList = new OrderedPendingList();
            }
            for (QueueMessageReference ref : result) {
                if (!pagedInMessages.containsKey(ref.getMessageId())) {
                    pagedInMessages.put(ref.getMessageId(), ref);
                    resultList.addMessageLast(ref);
                } else {
                    ref.decrementReferenceCount();
                }
            }
        } finally {
            pagedInMessagesLock.writeLock().unlock();
        }
...        
{code}

If the consumers are fast and acknowledge the initial message dispatch before the \
next duplicate message is sent to the queue, then the \
{{pagedInMessages.constainsKey(...)}} check will *not* prevent the duplicate message \
from being dispatched since the message ID will have already been removed as part of \
the acknowledgement.

This is problem 1: duplicate detection fails if acknowledgements are quick

If the consumers are slow and don't acknowledge the initial message dispatch before \
the next duplicate message is sent to the queue, then duplicate detection will \
correctly ignore the message.  However, by this time, the duplicate message has \
already incremented the queue size:

{code:title=Queue.java}
final void messageSent(final ConnectionContext context, final Message msg) throws \
Exception {  destinationStatistics.getEnqueues().increment();
    destinationStatistics.getMessages().increment();
    messageDelivered(context, msg);
{code}

The call to {{messageSent}} is made by the thread sending the message to the queue.  \
This thread is completely unaware that the message was ignored since the call to \
{{doPageInForDispatch}} is done by a separate thread (e.g., the taskrunner calling \
{{iterate()}}.

Since the queue size is incremented, but the duplicate message is never dispatched, \
there is no subsequent acknowledgement ot reduce the queue size.  As a result, each \
duplicate message that is ignored remains counted in the queue size even though it's \
no longer in the queue.  

This is problem 2: if duplication detection succeeds, the queue size incorrectly \
counts the duplicate/ignored message.

Which problem gets exhibited depends on a race condition between the thread(s) \
enqueueing the duplicate virtual topic subscriptions and consumer threads \
acknowledging the dispatches.

While the unit test's methodology of creating multiple virtual topic consumers is \
somewhat contrived, there must be other circumstances under which duplicate messages \
are sent to queues (otherwise, why would there be logic to handle this case?)  In \
this context, the test case reveals a problem with the handling of duplicate \
messages.

Solution
========
None at this time.  We worked around the problem by patching AMQ-4148 to prevent the \
duplicate subscriptions that cause this bug.  However, our concern has been raised \
about AMQ's generally handling of duplicate messages sent to a queue.





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[prev in list] [next in list] [prev in thread] [next in thread]