[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-dev
Subject:    Re: Database access and prefetch considerations
From:       Manuel Teira Paz <mteira () tid ! es>
Date:       2008-10-30 15:45:20
Message-ID: 4909D690.8090504 () tid ! es
[Download RAW message or body]

Thanks a lot, James, for your fast answer. Taking advantage of your 
kindness, I will like to elaborate on some of the topics. Please, find 
my questions below:


James Strachan escribió:
> 2008/10/30 Manuel Teira Paz <mteira@tid.es>:
>   
>> Hi, activemq guys.
>>
>>
>> We are suffering some bad service times on a production system embedding
>> ActiveMQ as message broker. We are not sure whether the problem is the
>> broker itself, but I would like to get some insights in the broker
>> internals, to be able to understand the situation better.
>>
>> We are using an activemq 4.1 release (we are not allowed to upgrade it) ,
>> with pure JDBC persistence. We are using only queues and persistent
>> messages.
>>
>> The first concern is about database access. Even when the number of messages
>> in queues are high, we are observing no growth in the number of active
>> database connections. Furthermore, it seems that only a thread per
>> connection is responsible of fetching and deleting messages in the
>> persistence.
>>     
>
> Yes thats how it works.
>
>
>   
>> Given the fact that we have 30 consumers on a given queue,
>> sharing the same connection, does this mean that a single thread is managing
>> all the persistence needs? If true, wouldn't it be a potential bottleneck,
>> avoiding us to improve the performance adding more consumers?
>>     
>
> ActiveMQ is tuned for loads of concurrent connections. If you only
> really have one connection, just don't share your connection across
> each producer/consumer - or use async dispatch
>
>   
Actually, we are not using a single connection for the whole system, but 
a single connection for every one of our components, handling message 
from a given queue. So, what is your recommendation, to use a single 
connection for every one of our 30 consumers, some ratio between 
consumers and connections?

About the async dispatch possibility. What do actually involves, 
performance and reliably wise? I mean, how the database workload is 
delivered using async dispatch? Or, for example, should we be aware of 
asynchronous exceptions if we switch to that mode?
>   
>> The second question is about the consumer prefetch. We did disable it in the
>> past, trying to avoid situations where messages got stuck in queues, due to
>> slow consumption of previous messages, producing unexpected and apparently
>> random delays in the consumption of some messages.
>>
>> We wonder if prefetch could in some way lighten the load on the database
>> side. We guess that perhaps messages prefetched for a given queue are kept
>> in memory, avoiding the need to read them from the database. On the other
>> side, does prefetch activate a particular revover strategy for database
>> message restoring, or are they fetched in a one by one basis?
>>     
>
> They are fetched one by one - prefetch only affects how quickly they
> are streamed to a consumer
>
>
>   
>> This also triggers the question of mesage caching. Even when prefetch is
>> disabled, is there any strategy, perhaps configurable, to cache messages in
>> memory?
>>     
>
> Not in 4.x though there are improvements in 5.x
>   
Does that mean that there isn't any cache feature in 4.x or that it's 
not configurable?
Could you please give more details about those improvements in 5.x?

> BTW pure JDBC is always gonna be kinda slow - its why we developed AMQ
> Store which is often 10X to 100X faster
> http://activemq.apache.org/amq-message-store.html
>
>   
The problem here is that we need the system to work in a master-slave 
configuration. I think that having some kind of short term fast 
persistence solution (as I guess amq store is) is not an option, since 
on a machine failure, database information is not guaranteed to be 
updated. Or am I missing something?


Thanks a lot. Looking forward for your wise answers.

Best regards.


> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
>   

[prev in list] [next in list] [prev in thread] [next in thread]