[prev in list] [next in list] [prev in thread] [next in thread]
List: zeromq-dev
Subject: Re: [zeromq-dev] NetMQ: is PUB/SUB filtering done at publisher side?
From: Giacomo Tesio <giacomo () tesio ! it>
Date: 2013-06-27 13:12:04
Message-ID: CAHL7psEF3mptPhKRAaneroKJ0Z6gK1m9UnuMDBT==xYvuL1cDA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thanks Doron.
I'll try this way: since I can't overload the process that recieve data
from the external source, I'll use it with a XSUB/XPUB proxy in the
subscribers process; XPUB will be inproc with SendWatermark to zero so that
the connected subscribers will receive all the messages. I can accept high
memory usage on the subscriber side, but not on the publisher one.
Looks reasonable?
Giacomo
On Thu, Jun 27, 2013 at 11:46 AM, Doron Somech <somdoron@gmail.com> wrote:
> Hi Giacomo,
>
> I'm pretty sure subscribers cannot block the publisher, I'm not sure the
> breakpoint is a good way to test that.
>
> Anyway regarding your other questions, NetMQ can handle slow subscribers,
> you just need to set the SendWatermark to zero and NetMQ will queue all
> messages, you should however know that this can cause to a memory problem.
>
> I don't think you need any queues at any side, ZeroMQ and NetMQ are
> working with queues, when you send a message or receive a message you
> actually queue and dequeue, there is another thread that actually do the
> network stuff. So I think you should publish the message to NetMQ socket
> directly from the thread that listen to the source and on the subscriber
> side just consume the messages directly from the queue.
>
> Regarding the timeout, you can also do it with NetMQ using a poller or
> Poll method on a Socket.
>
>
>
>
> On Thu, Jun 27, 2013 at 11:45 AM, Giacomo Tesio <giacomo@tesio.it> wrote:
>
>> Hi Doron!
>>
>> Thanks for your help.
>> I have two (windows7) processes that communicate via netmq pub/sub (the
>> publisher runs as x86, the subscriber as x64). While debugging (in visual
>> studio), I've set a breakpoint just after the send() (in the publisher) and
>> just after the Recieve (in the subscriber) and I noted that the publisher
>> send the message but hang while the subscriber handle it.
>>
>> The publisher forwards data from another async source that is very fast
>> and I must ensure that:
>> 1) no message from the source is lost (thus I can't block the publisher
>> process, because the async source requires very low latency and do not
>> provide any cache: if no one is ready to read the message they send, the
>> message is lost)
>> 2) no message from the publisher is lost by the subscribers, even if
>> they have different processing speeds.
>>
>> To ensure 2 I'm using an unbounded BlockingCollection<T> as a queue (that
>> btw, enable me to TryTake messages with a timeout, that is useful to
>> control the consuming thread when the publisher don't send nothing)
>>
>> In case of slow subscribers, does NetMQ alone (without the unbounded
>> BlockingCollection) ensure that I won't neither loose any message nor block
>> the publisher or other subscribers?
>>
>>
>> Giacomo
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 27, 2013 at 9:53 AM, Doron Somech <somdoron@gmail.com> wrote:
>>
>>> Hi Giacomo,
>>>
>>> I'm happy to hear you choose netmq, answers to your questions:
>>>
>>> 1. There are no functionality differences between NetMQ and
>>> CLRZMQ/ZeroMQ, there are some small API differences between NetMQ and
>>> CLRZMQ.
>>> 2. Same as ZeroMQ, so publisher side (it's based on ZeroMQ 3.2.2)
>>> 3. What to do you mean blocked? how did you noted that? I don't think
>>> you should queue because NetMQ / ZeroMQ is already doing exactly that. Just
>>> consume the messages from the socket.
>>>
>>> Doron
>>>
>>>
>>> On Thu, Jun 27, 2013 at 1:04 AM, Giacomo Tesio <giacomo@tesio.it> wrote:
>>>
>>>> Hello everybody, I'm working to my first ZeroMQ based real-world
>>>> application and I have a few questions:
>>>>
>>>> 1) I'm working on .NET and I choosed NetMQ instead of CLRZMQ just on
>>>> the base of github activity. I've noted there are some differences between
>>>> the two API (for example I had to Subscribe("") instead of SubscribeAll())
>>>> but they seem minor differences at a first look. Is there any difference in
>>>> functionality that I should be aware of?
>>>> 2) I have to use a topic based PUB tcp socket: is PUB/SUB filtering
>>>> done at publisher side?
>>>> 3) I noted (in debug) that the publisher blocks while the subscribers
>>>> handle each message: I've tried to minimize the impact of this with a queue
>>>> of messages of subscriber side, with the subscriber that fills the queue
>>>> and another thread consuming it. Is there a better approach?
>>>>
>>>> Thanks a lot!
>>>>
>>>>
>>>> Giacomo
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev@lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
[Attachment #5 (text/html)]
<div dir="ltr">Thanks Doron.<div><br></div><div>I'll try this way: since I \
can't overload the process that recieve data from the external source, I'll \
use it with a XSUB/XPUB proxy in the subscribers process; XPUB will be inproc with \
SendWatermark to zero so that the connected subscribers will receive all the \
messages. I can accept high memory usage on the subscriber side, but not on the \
publisher one.</div> <div><br></div><div>Looks \
reasonable?</div><div><br></div><div><br></div><div>Giacomo</div></div><div \
class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 27, 2013 at 11:46 \
AM, Doron Somech <span dir="ltr"><<a href="mailto:somdoron@gmail.com" \
target="_blank">somdoron@gmail.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Giacomo,<div><br></div><div>I'm pretty \
sure subscribers cannot block the publisher, I'm not sure the breakpoint is a \
good way to test that.</div> <div><br></div><div>Anyway regarding your other \
questions, NetMQ can handle slow subscribers, you just need to set the SendWatermark \
to zero and NetMQ will queue all messages, you should however know that this can \
cause to a memory problem. </div>
<div><br></div><div>I don't think you need any queues at any side, ZeroMQ and \
NetMQ are working with queues, when you send a message or receive a message you \
actually queue and dequeue, there is another thread that actually do the network \
stuff. So I think you should publish the message to NetMQ socket directly from the \
thread that listen to the source and on the subscriber side just consume the messages \
directly from the queue.</div>
<div><br></div><div>Regarding the timeout, you can also do it with NetMQ using a \
poller or Poll method on a Socket. </div><div><br></div><div><br></div></div><div \
class="HOEnZb"><div class="h5"><div class="gmail_extra"><br> <br><div \
class="gmail_quote"> On Thu, Jun 27, 2013 at 11:45 AM, Giacomo Tesio <span \
dir="ltr"><<a href="mailto:giacomo@tesio.it" \
target="_blank">giacomo@tesio.it</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div dir="ltr">Hi Doron!<div><br></div><div>Thanks for your help. </div><div>I have \
two (windows7) processes that communicate via netmq pub/sub (the publisher runs as \
x86, the subscriber as x64). While debugging (in visual studio), I've set a \
breakpoint just after the send() (in the publisher) and just after the Recieve (in \
the subscriber) and I noted that the publisher send the message but hang while the \
subscriber handle it.</div>
<div><br></div><div>The publisher forwards data from another async source that is \
very fast and I must ensure that:</div><div> 1) no message from the source is lost \
(thus I can't block the publisher process, because the async source requires very \
low latency and do not provide any cache: if no one is ready to read the message they \
send, the message is lost)</div>
<div> 2) no message from the publisher is lost by the subscribers, even if they have \
different processing speeds.</div><div><br></div><div>To ensure 2 I'm using an \
unbounded BlockingCollection<T> as a queue (that btw, enable me to TryTake \
messages with a timeout, that is useful to control the consuming thread when the \
publisher don't send nothing)</div>
<div><br></div><div>In case of slow subscribers, does NetMQ alone (without the \
unbounded BlockingCollection) ensure that I won't neither loose any message nor \
block the publisher or other subscribers?</div><span><font color="#888888"><div>
<br></div>
<div><br></div><div>Giacomo</div></font></span><div><div><div><br></div><div><br></div><div><br></div><div><br></div><div \
class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 27, 2013 at 9:53 AM, \
Doron Somech <span dir="ltr"><<a href="mailto:somdoron@gmail.com" \
target="_blank">somdoron@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Giacomo,<div><br></div><div>I'm happy \
to hear you choose netmq, answers to your questions:<br><div>
<br></div><div>1. There are no functionality differences between NetMQ and \
CLRZMQ/ZeroMQ, there are some small API differences between NetMQ and CLRZMQ.</div> \
</div><div>2. Same as ZeroMQ, so publisher side (it's based on ZeroMQ \
3.2.2)</div><div>3. What to do you mean blocked? how did you noted that? I don't \
think you should queue because NetMQ / ZeroMQ is already doing exactly that. Just \
consume the messages from the socket.</div>
<div><br></div><div>Doron</div></div><div class="gmail_extra"><br><br><div \
class="gmail_quote"><div><div>On Thu, Jun 27, 2013 at 1:04 AM, Giacomo Tesio <span \
dir="ltr"><<a href="mailto:giacomo@tesio.it" \
target="_blank">giacomo@tesio.it</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div>Hello everybody, \
I'm working to my first ZeroMQ based real-world application and I have a few \
questions:</div>
<div><br></div><div>1) I'm working on .NET and I choosed NetMQ instead of CLRZMQ \
just on the base of github activity. I've noted there are some differences \
between the two API (for example I had to Subscribe("") instead of \
SubscribeAll()) but they seem minor differences at a first look. Is there any \
difference in functionality that I should be aware of?</div>
<div>2) I have to use a topic based PUB tcp socket: is PUB/SUB filtering done at \
publisher side?</div><div>3) I noted (in debug) that the publisher blocks while the \
subscribers handle each message: I've tried to minimize the impact of this with a \
queue of messages of subscriber side, with the subscriber that fills the queue and \
another thread consuming it. Is there a better approach?</div>
</div><div><br></div><div>Thanks a lot!</div><span><font \
color="#888888"><div><br></div><div><br></div><div>Giacomo</div></font></span></div> \
<br></div></div>_______________________________________________<br> zeromq-dev \
mailing list<br> <a href="mailto:zeromq-dev@lists.zeromq.org" \
target="_blank">zeromq-dev@lists.zeromq.org</a><br> <a \
href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br> \
<br></blockquote></div><br></div> \
<br>_______________________________________________<br> zeromq-dev mailing list<br>
<a href="mailto:zeromq-dev@lists.zeromq.org" \
target="_blank">zeromq-dev@lists.zeromq.org</a><br> <a \
href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br> \
<br></blockquote></div><br></div></div></div></div> \
<br>_______________________________________________<br> zeromq-dev mailing list<br>
<a href="mailto:zeromq-dev@lists.zeromq.org" \
target="_blank">zeromq-dev@lists.zeromq.org</a><br> <a \
href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br> \
<br></blockquote></div><br></div> \
</div></div><br>_______________________________________________<br> zeromq-dev \
mailing list<br> <a href="mailto:zeromq-dev@lists.zeromq.org">zeromq-dev@lists.zeromq.org</a><br>
<a href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br> \
<br></blockquote></div><br></div>
_______________________________________________
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic