[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-users
Subject:    Re: ActiveMQ Classic HA without SAN
From:       Matt Pavlovich <mattrpav () gmail ! com>
Date:       2022-06-10 16:34:03
Message-ID: 9645824A-A44B-4D09-A31F-B23C55530562 () gmail ! com
[Download RAW message or body]


Hello Yannick-

When it comes to distributed systems and HA it generally boils down to a set of \
benefits and trade-offs. There is no ‘easy button' to get perfect uptime. When you \
use replicated systems (filesystems, databases, messaging systems, etc) , you suffer \
from having to take a full-shutdown to fix 'split brain' in the event the two nodes \
get out of sync— creating a generally undesirable amount of downtime to resolve.

Using well established and stable infrastructure is always wise, and leveraging ESX \
to do fault tolerant volumes is a strong option. I know of lots of messaging \
infrastructure (ActiveMQ, IBM MQ, and others) that use a lot of single broker \
architectures to reduce the overall amount of tech involved. Then focus energies on \
application performance to keep the queues as close to empty as possible vs making \
the messaging broker HA. Remember— if the queues are empty, there is no need to \
replicate anything!

Echoing what Justin said his follow-up email, if the broker can't write to disk the \
safest thing for the broker to do is to error to the clients— this way clients can \
decide whether to bubble up the error message to the sender, or take some sort of \
compensating action.

Thanks,
Matt

> On Jun 10, 2022, at 1:38 AM, Carbonneaux Yannick <yannick.carbonneaux@skyguide.ch> \
> wrote: 
> We were actually discussing this possibility yesterday, but wondered what would be \
>                 the behavior of the broker in case the shared FS fails, and have \
>                 two options there:
> - let the VM exposing the NFS fail and be restarted on another ESX host by the \
>                 cluster
> - or set the VM as fault tolerant on the ESX so that it is readily available shall \
> the primary host fail.  
> But I'm especially wondering what would be the best practice in case of failure of \
> the shared storage with KahaDB ? Just piles up in memory until KahaDB comes back ? \
> Or plainly fail ?  Seems to be configurable via LeaseLockerIOExceptionHandler if I \
> understood well.  
> Yannick
> 
> 
> -----Original Message-----
> From: Matt Pavlovich <mattrpav@gmail.com> 
> Sent: Thursday, 9 June 2022 22:18
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Classic HA without SAN
> 
> A NFSv4 server running on a dedicated NAS (or other Linux server running on ESX) \
> can provide the shared-storage option for HA data storage using KahaDB 
> Two (or more) ActiveMQ servers can mount the same NFSv4 file share and the first \
> one to obtain the lock "wins" and becomes the primary (aka master) broker. 
> -Matt
> 
> > On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick < \
> > yannick.carbonneaux@skyguide.ch> wrote: 
> > > Hi,
> > > 
> > > We are planning the deployment of an ActiveMQ Classic cluster and I 
> > > am looking for the best way to implement HA in a VMWare environment 
> > > on RHEL OS without a SAN.
> > > It seems I'm down to Master/Slave with either something like GFS2 (so 
> > > a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that 
> > > no messages are lost if we lose an ESX in our VMWare cluster.
> > > 
> > > Are there any recommendations on how to implement HA in this context?
> > > Performance is not really problematic today... but might in a near future.
> > > 
> > > Thanks for your help,
> > > 
> > > ___
> > > Yannick Carbonneaux
> > > 
> > > 
> 



[prev in list] [next in list] [prev in thread] [next in thread]