[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-users
Subject:    Re: Producer flow control
From:       Torsten Mielke <torsten () fusesource ! com>
Date:       2012-02-20 9:25:56
Message-ID: B5B421A0-85CB-4C11-B2A7-CD458D05A88E () fusesource ! com
[Download RAW message or body]

Hi,

You could use the vmQueueCursor as well. 
In both cases (assuming producerFlowControl=true) the cursors will only dispatch msg \
from memory and not from the store. 

Regards,
Torsten Mielke


On Feb 15, 2012, at 3:48 PM, Hervé BARRAULT wrote:

> Hi,
> If i set cursorMemoryHighWaterMark to 100, is it a good idea to use
> vmCursor instead of File Based Cursor (the default one) ?
> 
> Regards
> Hervé
> 
> On Wed, Feb 15, 2012 at 10:50 AM, Torsten Mielke <torsten@fusesource.com>wrote:
> 
> > Hello,
> > 
> > > So when using persistence, i can't rely on producerFlowControl to
> > > limit the Flow.
> > 
> > 
> > Yes you can. However when using persistent queue messages,
> > producerFlowControl does not kick-in that early. Which is actually a good
> > thing in general.
> > 
> > All this is configurable as well. You can specify
> > cursorMemoryHighWaterMark="100" on your destination policy configuration.
> > With that the cursor will take up to 100% of the configured queue memory
> > limit. When that limit is reached, producer flow control will also kick in
> > (as now 100% of the memory are in use).
> > 
> > 
> > > What is the best way to add a mechanism to constrain the queue size
> > > (handling slow consumer when using persistence) ?
> > 
> > ProducerFlowControl and perhaps in your case also setting
> > cursorMemoryHighWaterMark="100" on the destination policy.
> > 
> > 
> > Best Regards,
> > 
> > Torsten Mielke
> > torsten@fusesource.com
> > tmielke@blogspot.com
> > 
> > 
> > On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
> > 
> > > Hi,
> > > thanks for the answer.
> > > 
> > > So when using persistence, i can't rely on producerFlowControl to
> > > limit the Flow.
> > > 
> > > If i have a consumer which is able to process only 10 messages by
> > > seconds and my queue is filled with 36000 messages (due to faster
> > > producers), i know that it will take 1 hour to process all my
> > > messages. I would reject new messages as my processing time shall be
> > > less than one hour.
> > > 
> > > I would also keep the order of incoming messages (small and big) so i
> > > can't easily manage two queues without adding a complex
> > > synchronization mechanism.
> > > 
> > > I see http://activemq.apache.org/slow-consumer-handling.html for
> > > non-persistent messages but it should not apply for persistent ones.
> > > (Prefetch is configurable for persistence :
> > > 
> > http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
> > 
> > > but nothing for Pending Message Limit Strategy).
> > > 
> > > What is the best way to add a mechanism to constrain the queue size
> > > (handling slow consumer when using persistence) ?
> > > 
> > > Modifying the cursor, producer flow control, another thing ?
> > > 
> > > Regards
> > > 
> > > Hervé
> > > 
> > > 
> > > On 2/14/12, Torsten Mielke <torsten@fusesource.com> wrote:
> > > > Hello Herve,
> > > > 
> > > > You cannot configure producer flow control based on the various message
> > > > sizes.
> > > > However you could consider using different queues for these types of
> > msgs
> > > > and configure
> > > > the limits for each queue differently. That way your small msgs won't be
> > > > blocked just because of a small amount of large msgs on the same queue
> > > > (assuming different producers for both kinds of msgs).
> > > > 
> > > > Also, when sending persistent queue messages, you rarely see producer
> > flow
> > > > control happening when using the default store cursor. That is because
> > the
> > > > store cursor is configured to hold messages in memory up to only 70% of
> > the
> > > > configured destination limit. If this 70% limit is reached, it won't
> > keep
> > > > new msgs in memory but reload them from the store (kahadb) later (when
> > its
> > > > time to dispatch the msg). Remember a persistent msg is first written
> > to the
> > > > store when received by the broker.
> > > > Producer flow control however kicks in when 100% of the available
> > memory for
> > > > each destination are used. This limit however you generally don't reach.
> > > > Things are different for non-persistent msgs. The default file cursor
> > keeps
> > > > msgs in memory until 100% of the configured destination limit are
> > reached.
> > > > Thereafter it swaps msgs to temp storage (which is different from
> > kahadb)
> > > > and producer flow control would kick in.
> > > > 
> > > > Hope this helps.
> > > > 
> > > > 
> > > > Torsten Mielke
> > > > torsten@fusesource.com
> > > > tmielke@blogspot.com
> > > > 
> > > > 
> > > > On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > i have looked to the documentation of the producer flow control
> > > > > functionality (http://activemq.apache.org/producer-flow-control.html).
> > > > > 
> > > > > I see that it is based on the storage size but in my case, i think not
> > > > > being able to use the size.
> > > > > I have a queue where i put messages two kind of messages : Big 10MB and
> > > > > Small 1kB.
> > > > > 
> > > > > As Big messages are rare (compared to small ones), i would limit the
> > > > > Memory
> > > > > to 100MB (10 messages).
> > > > > But for small messages i would limit the number of small messages to
> > 5000
> > > > > (so about 5MB) . We have noted that with JDBC persistence, the number
> > of
> > > > > pending messages in the broker (not by queue) is important for
> > > > > "performances".
> > > > > 
> > > > > As i would keep the order of Big and Small messages, i put them in the
> > > > > same
> > > > > queue.
> > > > > 
> > > > > Is there a way to define a strategy for the flow control which is 100MB
> > > > > and
> > > > > 5k messages ?
> > > > > 
> > > > > 
> > > > > Regards
> > > > > Hervé


[prev in list] [next in list] [prev in thread] [next in thread]