[prev in list] [next in list] [prev in thread] [next in thread]
List: zeromq-dev
Subject: Re: [zeromq-dev] ZeroMQ STREAM Socket Type
From: Doron Somech <somdoron () gmail ! com>
Date: 2016-05-25 9:41:02
Message-ID: CAB814uFwGyqbkQ4bmwLs9eBfskKOznxjiOrevAXmUdO6M7Utbg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
1. Doesn't matter
2. Yes, first part is the identity, then the data.
3. I'm not sure multipart is even working with STREAM socket.
On Tue, May 24, 2016 at 6:43 PM, Peter Witkowski <pwitkowski@gmail.com>
wrote:
> Thanks for the reply.
>
> First, I think the diagram below (please forgive my poor ASCII art) shows
> what I was trying to explain.
>
> Original (working code):
> PUSH (three part message) -----------TCP--------------> PULL (three part
> message)
>
> Currently Planned Implementation:
> STREAM Client (PUSH replacement)------TCP--------> Data Link Server
> ---------MAGIC?-------> Data Link Client------TCP------>STREAM Server (PULL
> replacement)
>
> Each STREAM socket must talk to the data link's TCP service (and this is
> "regular" TCP) which then forwards the sender's data across the link. No
> data from the PULL replacement will be able to make it back due to
> limitations with the physical layer on the data link (labeled "MAGIC?").
>
> I still have a few questions left:
> 1. Does it matter what I set the ZMQ_CONNECT_RID to? I'm assuming the
> STREAM Server isn't looking for this.
> 2. How does the receive side look like? Do I get a identity from the Data
> Link Client after I bind? Do I read this out each time before I read the
> actual data?
> 3. I'm assuming it's best to now NOT use the three part message (have to
> check each zmq_send() call for the number of bytes sent). Instead, I'm
> planning on serializing and ensuring that all the bytes make it across.
>
> Thanks again for the help.
>
> On Tue, May 24, 2016 at 3:00 AM, Doron Somech <somdoron@gmail.com> wrote:
>
>> Ok, you have an option called ZMQ_CONNECT_RID, when you set it in a
>> STREAM socket is set the identity of the first peer, so instead of what you
>> are doing now, do:
>>
>> 1. Set the socket CONNECT RID to "C" (or whatever)
>> 2. Connect the socket (RID MUST BE BEFORE CONNECT)
>> 3. Send identity"C"
>> 4. Send message
>>
>> Regarding atomic send, it wont, STREAM will send any bytes arrive, but it
>> should not matter as TCP is stream protocol snd don't have the notion of
>> messages.
>>
>> Zero length message is not flushing the data, is just closing the tcp
>> socket. You can also just close the socket.
>> On May 24, 2016 09:52, "Doron Somech" <somdoron@gmail.com> wrote:
>>
>>> It seems what you are doing won't work as currently it is hard to use
>>> STREAM as the first sender, as you don't know the identity if the peer. I
>>> think someone added notify option, worth to check it out. Are you using
>>> STREAM at both sides?
>>> On May 24, 2016 00:04, "Peter Witkowski" <pwitkowski@gmail.com> wrote:
>>>
>>>> Sorry to spam everyone, but just to clarify, when I say that PUSH-PULL
>>>> doesn't work because of handshaking, I'm speaking about the ZeroMQ
>>>> handshake (greetings, etc.). The data link handles the TCP connection
>>>> stuff.
>>>>
>>>> On Mon, May 23, 2016 at 4:58 PM, Peter Witkowski <pwitkowski@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Long story short, I have code that works that I now need to refactor.
>>>>> The networking for my application has changed and I need to push my ZeroMQ
>>>>> messages over a half-duplex (i.e., one way) wireless data link. The data
>>>>> link is looking for raw TCP on either end and specifies what side is the
>>>>> client and server. You can think of this data link as a one way bridge,
>>>>> with a TCP endpoint on either end that I need to talk to. The data link
>>>>> then forwards the TCP traffic in some weird protocol to the other side.
>>>>>
>>>>> The code I have written is a PUSH-PULL pattern, which doesn't work due
>>>>> to the handshaking involved at the socket's start-up (I'm assuming there's
>>>>> no way to disable this). The code sends and receives a three-part message
>>>>> (the third message part is large, about 65KB). I need to refactor this
>>>>> into two STREAM sockets, but I'm having problems (conceptually) with
>>>>> sending and receiving.
>>>>>
>>>>> Here's some pseudo code for the sender (which is the client per the
>>>>> data link):
>>>>>
>>>>> - Set-up context, open socket, call connect.
>>>>> - Get ZMQ_IDENTITY using zmq_getsockopt()
>>>>> - Send identity
>>>>> - Send message part 1 (assuming that this is an atomic send and
>>>>> won't only send N bytes)
>>>>> - Send identity
>>>>> - Send message part 2 (see previous assumption about atomic)
>>>>> - Send identity
>>>>> - Send message part 3 (see previous assumption about atomic)
>>>>> - When I'm ready to close, I send the identity followed by a zero
>>>>> length packet
>>>>>
>>>>> The receiver (TCP server per the data link) is basically the opposite,
>>>>> but here I'm a bit confused. Namely, is each receive call atomic, or is
>>>>> there a chance that the data gets chunked up into multiple messages? Also,
>>>>> do I need to read off the identifier each time or does the library only
>>>>> forward the message parts?
>>>>>
>>>>> Also, in general, are there any issues with using ZeroMQ (even STREAM
>>>>> sockets) with this set-up? Note that my assumption is that the receiver
>>>>> never needs to send data and the sender never needs to receive data.
>>>>>
>>>>> Thanks in advance for the help.
>>>>> --
>>>>> Peter Witkowski
>>>>> pwitkowski@gmail.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Peter Witkowski
>>>> pwitkowski@gmail.com
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev@lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
>
> --
> Peter Witkowski
> pwitkowski@gmail.com
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
[Attachment #5 (text/html)]
<div dir="ltr">1. Doesn't matter<div>2. Yes, first part is the identity, then the \
data.</div><div>3. I'm not sure multipart is even working with STREAM \
socket.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May \
24, 2016 at 6:43 PM, Peter Witkowski <span dir="ltr"><<a \
href="mailto:pwitkowski@gmail.com" \
target="_blank">pwitkowski@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div \
dir="ltr"><div><div><div><div><div><div><div><div><div><div>Thanks for the \
reply.<br><br></div>First, I think the diagram below (please forgive my poor ASCII \
art) shows what I was trying to explain.<br><br></div>Original (working \
code):<br></div>PUSH (three part message) -----------TCP--------------> PULL \
(three part message)<br><br></div>Currently Planned Implementation:<br></div>STREAM \
Client (PUSH replacement)------TCP--------> Data Link Server \
---------MAGIC?-------> Data Link Client------TCP------>STREAM Server (PULL \
replacement)<br><br></div>Each STREAM socket must talk to the data link's TCP \
service (and this is "regular" TCP) which then forwards the sender's \
data across the link. No data from the PULL replacement will be able to make it \
back due to limitations with the physical layer on the data link (labeled \
"MAGIC?").<br><br></div>I still have a few questions left:<br></div>1. Does \
it matter what I set the ZMQ_CONNECT_RID to? I'm assuming the STREAM Server \
isn't looking for this.<br></div>2. How does the receive side look like? Do I \
get a identity from the Data Link Client after I bind? Do I read this out each time \
before I read the actual data?<br></div><div>3. I'm assuming it's best to now \
NOT use the three part message (have to check each zmq_send() call for the number of \
bytes sent). Instead, I'm planning on serializing and ensuring that all the \
bytes make it across.<br></div><div><br></div><div>Thanks again for the \
help.<br></div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div \
class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, May 24, \
2016 at 3:00 AM, Doron Somech <span dir="ltr"><<a href="mailto:somdoron@gmail.com" \
target="_blank">somdoron@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><p dir="ltr">Ok, you have an option called ZMQ_CONNECT_RID, \
when you set it in a STREAM socket is set the identity of the first peer, so instead \
of what you are doing now, do:</p> <p dir="ltr">1. Set the socket CONNECT RID to \
"C" (or whatever)<br> 2. Connect the socket (RID MUST BE BEFORE \
CONNECT)<br> 3. Send identity"C"<br>
4. Send message</p>
<p dir="ltr">Regarding atomic send, it wont, STREAM will send any bytes arrive, but \
it should not matter as TCP is stream protocol snd don't have the notion of \
messages.</p> <p dir="ltr">Zero length message is not flushing the data, is just \
closing the tcp socket. You can also just close the socket.<br> </p><div><div>
<div class="gmail_quote">On May 24, 2016 09:52, "Doron Somech" <<a \
href="mailto:somdoron@gmail.com" target="_blank">somdoron@gmail.com</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">It seems what you are \
doing won't work as currently it is hard to use STREAM as the first sender, as \
you don't know the identity if the peer. I think someone added notify option, \
worth to check it out. Are you using STREAM at both sides?</p> <div \
class="gmail_quote">On May 24, 2016 00:04, "Peter Witkowski" <<a \
href="mailto:pwitkowski@gmail.com" target="_blank">pwitkowski@gmail.com</a>> \
wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry to spam \
everyone, but just to clarify, when I say that PUSH-PULL doesn't work because of \
handshaking, I'm speaking about the ZeroMQ handshake (greetings, etc.). The \
data link handles the TCP connection stuff.<br><div class="gmail_extra"><br><div \
class="gmail_quote">On Mon, May 23, 2016 at 4:58 PM, Peter Witkowski <span \
dir="ltr"><<a href="mailto:pwitkowski@gmail.com" \
target="_blank">pwitkowski@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello,<br><br></div>Long story \
short, I have code that works that I now need to refactor. The networking for my \
application has changed and I need to push my ZeroMQ messages over a half-duplex \
(i.e., one way) wireless data link. The data link is looking for raw TCP on either \
end and specifies what side is the client and server. You can think of this data \
link as a one way bridge, with a TCP endpoint on either end that I need to talk to. \
The data link then forwards the TCP traffic in some weird protocol to the other \
side.<br><br></div>The code I have written is a PUSH-PULL pattern, which doesn't \
work due to the handshaking involved at the socket's start-up (I'm assuming \
there's no way to disable this). The code sends and receives a three-part \
message (the third message part is large, about 65KB). I need to refactor this into \
two STREAM sockets, but I'm having problems (conceptually) with sending and \
receiving.<br><br></div>Here's some pseudo code for the sender (which is the \
client per the data link):<br><ul><li>Set-up context, open socket, call \
connect.</li><li>Get ZMQ_IDENTITY using zmq_getsockopt()</li><li>Send \
identity</li><li>Send message part 1 (assuming that this is an atomic send and \
won't only send N bytes)<br></li><li>Send identity</li><li>Send message part 2 \
(see previous assumption about atomic)<br></li><li>Send identity</li><li>Send message \
part 3 (see previous assumption about atomic)</li><li>When I'm ready to close, I \
send the identity followed by a zero length packet</li></ul><p>The receiver (TCP \
server per the data link) is basically the opposite, but here I'm a bit confused. \
Namely, is each receive call atomic, or is there a chance that the data gets chunked \
up into multiple messages? Also, do I need to read off the identifier each time or \
does the library only forward the message parts?</p><p>Also, in general, are there \
any issues with using ZeroMQ (even STREAM sockets) with this set-up? Note that my \
assumption is that the receiver never needs to send data and the sender never needs \
to receive data.</p><p>Thanks in advance for the help.<span><font \
color="#888888"><br></font></span></p><span><font \
color="#888888"><div><div><div><div><div><div><div><div>-- <br><div>Peter \
Witkowski<br><a href="mailto:pwitkowski@gmail.com" \
target="_blank">pwitkowski@gmail.com</a></div> \
</div></div></div></div></div></div></div></div></font></span></div> \
</blockquote></div><br><br clear="all"><br>-- <br><div>Peter Witkowski<br><a \
href="mailto:pwitkowski@gmail.com" target="_blank">pwitkowski@gmail.com</a></div> \
</div></div> <br>_______________________________________________<br>
zeromq-dev mailing list<br>
<a href="mailto:zeromq-dev@lists.zeromq.org" \
target="_blank">zeromq-dev@lists.zeromq.org</a><br> <a \
href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" rel="noreferrer" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br></blockquote></div>
</blockquote></div>
</div></div><br>_______________________________________________<br>
zeromq-dev mailing list<br>
<a href="mailto:zeromq-dev@lists.zeromq.org" \
target="_blank">zeromq-dev@lists.zeromq.org</a><br> <a \
href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" rel="noreferrer" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br></blockquote></div><br><br \
clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- \
<br><div>Peter Witkowski<br><a href="mailto:pwitkowski@gmail.com" \
target="_blank">pwitkowski@gmail.com</a></div> \
</font></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br>_______________________________________________<br>
zeromq-dev mailing list<br>
<a href="mailto:zeromq-dev@lists.zeromq.org">zeromq-dev@lists.zeromq.org</a><br>
<a href="http://lists.zeromq.org/mailman/listinfo/zeromq-dev" rel="noreferrer" \
target="_blank">http://lists.zeromq.org/mailman/listinfo/zeromq-dev</a><br></blockquote></div><br></div>
_______________________________________________
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic