'Re: Is it possible to route message to a specific consumer?'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-users
Subject:    Re: Is it possible to route message to a specific consumer?
From:       Endre_Stølsvik <Endre () stolsvik ! com>
Date:       2023-12-10 10:11:39
Message-ID: CABbot+sdbRckpZrqqFyaN71DahvGStFvDcZPowDwnvepEGrKvg () mail ! gmail ! com
[Download RAW message or body]


I've written the following little article in conjunction with the library
Mats3, which does not exactly describe your problem, but I believe an
aspect of it might be relevant: https://mats3.io/patterns/work-queues/

The core idea there, and the aspect which might be relevant, is to not use
actual Message Queues, but instead explicit "work queues", i.e. a table in
a database.

You have a state machine that these jobs go through. You can use the MQ (or
any other means like calling out to a service, or calculating something
"right there") to get the pieces of work done - and then transition the job
through its states.

Thus, you gain lots of introspection/monitoring abilities - e.g. a REST
query for how it is going with a particular job - and you also gain the
ability to control "pacing" of lots of jobs - which is what that article is
about. And you gain the ability to reset a job back one or multiple states
(via some UI you cook up) if something went wrong.

Wrt. the message queue use, you'd use, as Justin points out, correlationIds
to know which message is working on which job (to transition the job along
when the "done with piece of work" comes back) - or you could use Mats3,
whose central point is to *not* have to think about correlationIds, but
instead get an "async RPC" functionality, with a callstack! ;-)

Kind regards,
Endre.


On Sun, Dec 10, 2023 at 6:49 AM Justin Bertram <jbertram@apache.org> wrote:

> I wouldn't expect message grouping to help given the information about
> your use-case that you've provided so far. I outlined the reasons for this
> in my previous email. However, if you say that it's working for you then
> there's almost certainly something about your use-case that I don't
> understand.
> 
> Typically in request/response use-cases the requester specifies a
> "correlation ID" to allow it to keep track of the exchange of messages.
> Also, in cases with multiple brokers it's typical to cluster those brokers
> together so that messages can be shared transparently across the different
> brokers or perhaps to use connection routers [1] to "shard" applications to
> particular brokers or sets of brokers so that the applications can get
> reconnected back to the proper broker(s) in case they get disconnected
> assuming that the brokers contain application-related state information. I
> assume you're using something like this already, but if not it may be worth
> looking into.
> 
> 
> Justin
> 
> [1]
> https://activemq.apache.org/components/artemis/documentation/latest/connection-routers.html#connection-routers
>  
> 
> Justin
> 
> On Fri, Dec 8, 2023 at 5:26 PM John Lilley
> <john.lilley@redpointglobal.com.invalid> wrote:
> 
> > Hi Justin,
> > 
> > 
> > 
> > Thanks for the response!
> > 
> > 
> > 
> > Our use case is something like this.  Suppose we have a nice RPC client
> > wrapped around message queues and their services.  Some methods take a long
> > time, and we want to report status along the way, so we've arranged for
> > such methods to return a "task ID":
> > 
> > 
> > 
> > String taskID = client.startLongThing();
> > 
> > while (!client.isLongThingFinished(taskID)) { wait, or print progress… }
> > 
> > client.getLongThingResult(taskID);
> > 
> > 
> > 
> > The problem we hit is that the long-running tasks are not stateless.
> > They start, execute and finish within the same server process.  So the
> > calls to startLongThing(), isLongThingFinished(), and getLongThingResult()
> > must all be serviced by the same process.  If we have multiple HA instances
> > of a service running in a Kubernetes cluster, the isLongThingFinished()
> > requests will be handle by a random server that may not know about the task.
> > 
> > 
> > 
> > Do you think that the "message group" attribute is a reasonable
> > approach?  In JMS:
> > 
> > message.setStringProperty("JMSXGroupID", "grouplabel");
> > 
> > 
> > 
> > I've implemented and tested this and it does seem to work.  Do you know
> > of any caveats or cautions?
> > 
> > 
> > 
> > Thanks
> > 
> > John
> > 
> > 
> > 
> > 
> > 
> > 
> > [image: rg] <https://www.redpointglobal.com/>
> > 
> > John Lilley
> > 
> > Data Management Chief Architect, Redpoint Global Inc.
> > 
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> > 
> > *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
> > 
> > *From:* Justin Bertram <jbertram@apache.org>
> > *Sent:* Thursday, December 7, 2023 10:56 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Re: Is it possible to route message to a specific consumer?
> > 
> > 
> > 
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> > 
> > 
> > 
> > > Is there any kind of "consumer affinity" that can be attached to a
> > message, which directs the broker to select a specific consumer?
> > 
> > 
> > 
> > I may be forgetting something, but I believe the answer to this question
> > is no. The broker is broadly designed to support decoupled, asynchronous
> > messaging where consumers and producers know essentially nothing about each
> > other and the broker just receives messages and makes them available for
> > consumption. Message consumption is fundamentally consumer-driven. The
> > broker itself doesn't choose where the messages go except in some limited
> > sense in particular use-cases. Even in the case of message grouping, the
> > messages are not directed to a specific consumer as much as they are just
> > directed at _just one consumer (i.e. any consumer) at a time_ since the
> > goal is just for the messages to be processed in order.
> > 
> > 
> > 
> > You can add specific properties to specific messages and then JMS
> > consumers can use selectors to either ignore or choose those messages for
> > consumption and in this way you can ensure that certain consumers get
> > certain messages, but both the producer and consumer need to know about the
> > properties. Furthermore, if all the applications use a single queue then
> > they can start interfering with each other. For example, if one application
> > fails to consume its messages then messages will begin accumulating and may
> > prevent other applications from getting their messages since queues are by
> > nature first-in-first-out data structures.
> > 
> > 
> > 
> > I'd need to know more about the specifics of your use-case before I could
> > make further recommendations.
> > 
> > 
> > 
> > 
> > 
> > Justin
> > 
> > 
> > 
> > On Thu, Dec 7, 2023 at 1:26 PM John Lilley <
> > john.lilley@redpointglobal.com.invalid> wrote:
> > 
> > Oooh, maybe I answered my own question:
> > 
> > 
> > https://activemq.apache.org/components/artemis/documentation/1.0.0/message-grouping.html
> >  
> > This *looks* like what I'm after.  Any advice around its use?
> > 
> > 
> > 
> > John
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > [image: rg]
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,RLCz7uLdUVWU-4PdO0mzCFR75gph7wgFAJpvnoG5yg55RbBDZxZfAUVZs-ERB9VGZHVUZK6w6h0r3FI5bjkHA50Th_z9pS5huRB5AZIUhxN5nTIQlTbVa2sLLA,,&typo=1>
> >  
> > *John Lilley *
> > 
> > *Data Management Chief Architect, Redpoint Global Inc. *
> > 
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> > 
> > *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
> > 
> > *From:* John Lilley <john.lilley@redpointglobal.com.INVALID>
> > *Sent:* Thursday, December 7, 2023 11:58 AM
> > *To:* users@activemq.apache.org
> > *Subject:* Is it possible to route message to a specific consumer?
> > 
> > 
> > 
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> > 
> > 
> > 
> > Greetings,
> > 
> > 
> > 
> > We are in the process of transforming our Java-based services into
> > Kubernetes-based microservice swarms with HA topology.  So every service
> > will have multiple instances.  The good news is, AMQ supports this
> > splendidly, because the message broker is very good about load-balancing
> > between consumers and handling failed consumers.
> > 
> > 
> > 
> > However, we have a few cases of "long running tasks" that a single
> > service instance is responsible for.  In those cases, we need to be able to
> > route a request to a specific consumer instance.  We could use
> > specially-named queues for each consumer instance, but this seems awkward
> > and hopefully not necessary.
> > 
> > 
> > 
> > Is there any kind of "consumer affinity" that can be attached to a
> > message, which directs the broker to select a specific consumer?
> > 
> > If not, other ideas are certainly welcome.
> > 
> > 
> > 
> > We are using OpenJDK 17
> > 
> > 
> > 
> > This is our maven dependency
> > 
> > <dependency>
> > <groupId>org.apache.activemq</groupId>
> > <artifactId>artemis-jms-client-all</artifactId>
> > <version>2.30.0</version>
> > </dependency>
> > 
> > 
> > 
> > 
> > 
> > Thanks
> > 
> > John
> > 
> > 
> > 
> > 
> > 
> > 
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
> >  
> > *John Lilley
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>*
> >  
> > *Data Management Chief Architect, Redpoint Global Inc.
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>*
> >  
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
> >  
> > *M: +1 7209385761 | john.lilley@redpointglobal.com
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>*
> >  
> > 
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
> >  
> > 
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E \
> > ,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>
> >  
> > 
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> > 
> 



[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic