[prev in list] [next in list] [prev in thread] [next in thread] 

List:       zeromq-dev
Subject:    Re: [zeromq-dev] content based routing for a thread pool
From:       Brett Viren via zeromq-dev <zeromq-dev () lists ! zeromq ! org>
Date:       2023-10-30 12:59:45
Message-ID: 867cn4e05a.fsf () hierocles ! phy ! bnl ! gov
[Download RAW message or body]

[Attachment #2 (multipart/signed)]


Lei Jiang <lei.jiang.29@gmail.com> writes:

> While=C2=A0putting together my code, I find it a bit hard to keep the imp=
lementation simple and efficient. For
> example, the routing IDs are separated by empty frames as delimiters. Acc=
ording to some charts in the guide,
> multiple IDs could be separated by multiple delimiters. This makes it imp=
ossible to identify message body
> efficiently until all frames have been received

This is a feature.  Multiple layers of routing can be accommodated by a
sequence of these routing parts.    IIRC, the guide describes this as
nested "envelopes".  The first ROUTER need only care about the first
parts and should (must) leave any subsequent parts un-recv'ed.

>  (BTW=C2=A0there does not seem to be a formal documentation on the
> message formats?).

The "ZMQ_ROUTER" section of the zmq_socket(3) man page documents the
message schema that pertains to routing.  It is also defined by the
REPREQ RFC:

  https://rfc.zeromq.org/spec/28/

> Also it seems the send() calls will close the message being sent,
> which I find a bit odd, too.

zmq_msg_send(3) man page states this behavior and suggests to copy the
message object if you wish to send it multiple times.  (In general,
ZeroMQ is very good at stating these details in man pages and RFCs).

I don't know this internal detail myself but I have always assumed this
is done so that the socket internals may avoid a copy in the more common
case of sending a message to a single socket.

> Another thing I find is that it's quite hard to make good use of features=
 like zero-copy and multi-part
> messages. When having hops, every hop will at least require an additional=
 copy. For the multipart,
> originally I thought I could use it for responses with unknown lengths. B=
ut then I realized because the
> proxy code can't serve any other peer until current request or response i=
s finished, it will for sure block
> everyone else. The only remedy I can think of is for every packet to carr=
y ID or routing data instead of
> using the "more" flag. So in reality the best use of multipart is probabl=
y to carry different parts/fields
> of requests/responses.

Except for the requirements of ROUTER, the application is free to do
what it wants in defining messages schema.

In making such definitions one contends with how explicit or implicit to
get.  Eg, should the message schema have some amount of self-description
or are endpoints more trusted to "do the right thing".

To give you a flavor, here is an example of an "opinionated" message
format I've used in the past:

  https://brettviren.github.io/zio/messages.html

Warning: this format is NOT at all as well thought out as it should be.
It also has some application-specific parts at levels that makes it not
generally useful.

> Lastly, regarding the proxy being single threaded, I think that's a
> must? As the stable versions don't support accessing sockets from
> different threads, the bridging code of any 2 sockets must run in a
> single thread.

Yes.

But a "proxy" is typically not doing much work other than:

  recv() -> [marshal] -> send()

If the "[marshal]" code becomes so computationally expensive as to
require multiple threads in order to keep up with the message rate from
the proxy's clients then perhaps the architecture needs rethinking.

For example, the "[marshal]" code can become a gateway to multi-threaded
(or even multi-process) workers.  Eg, "[marshal]" could be made to be
also a "client" in a MDP/PPP pattern.  Such "[marshal]" code would best
to use a poller watching both its own client socket and the socket used
to talk to the MDP/PPP "broker".  Then, the code inside the [marshal]'s
main loop that is servicing that poller will become a rather fast path
that can run in a single thread.


=2DBrett.

["signature.asc" (application/pgp-signature)]

_______________________________________________
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev


[prev in list] [next in list] [prev in thread] [next in thread]