[prev in list] [next in list] [prev in thread] [next in thread] 

List:       pdns-users
Subject:    Re: [Pdns-users] [EXT] Re:  remote backend
From:       Alexis Fidalgo via Pdns-users <pdns-users () mailman ! powerdns ! com>
Date:       2023-11-30 9:36:14
Message-ID: 670A5181-52AB-4717-ACC4-05F87B7302DF () gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


Check this, i removed the code from my responder by now but it was something like \
this

func main() {
    l, err := net.ListenUnix("unix", &UnixAddr{Name: "unix",Net :"/tmp/echo.sock",})
    if err != nil {
        panic(err)
    }

    for {
        fd, err := l.Accept()
        if err != nil {
            panic(err)
        }

    go func(fd net.Conn) {  
        var buf [1024]byte
        n, err := fd.Read(buf)
        if err!=nil{
            panic(err)
        }
        data := buf[0:netip]
        println("Server got:", string(data))
        _, err = c.Write(data)
        if err != nil {
            panic(err)
        }
    }(fd)}
}

Using this (tested with pdns as client, a python cliend pasted below, a golang client \
and also netcat) causes

1. i can connect
2. i can send a message and it's printed
3. answer is received
4. complete lock, im not able to send any other message (yes, starting a different \
instance of any client sends another message and locks and so on)

If i add (right before }(fd)} (inside the for loop) an "fd.Close()" what happens is, \
there's no lock anymore, i can send messages over and over again, but (the famous \
but) doesn't matter which client i use (python, pdns, golang, java) i need to start \
the client everytime.

Example

import socket
import sys
import time
import json

with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
    client.connect("/tmp/pra.sock")

    while True:
        client.send(json.dumps({'method':'initialize'}).encode('utf-8'))
        v = client.recv(1024)
        print(f"--> {v}")
        time.sleep(1)

    client.close()

Scenario 1 (without the fd.Close())
Starting this, only sends and receive first message and locks

Scenario 2 (with fd.Close())
Starting this, sends and receives first message and throws an Exception because \
Broken Pipe

You can see im moving the read/answer to a different coroutine on every call to avoid \
blocking the socket in anyway but it doesnt help.

Note:
Using SOCKET_DGRAM i was able to loop for ever with no problem, i tried to set that \
in unixconnector.cc <http://unixconnector.cc/> changing the socket type but is not \
that simple.

Note 2:
Helped a lot what you told me, i wrote a simple python script to stress the responder \
(putting pdns aside, only pcap'd real traffic to have real queries for my script) and \
i found something interesting, as more threads i open pushing the responder, there's \
queries that takes more and more time to be answered, a good reason for the http \
connection between pdns > backend fails with timeout and DNS client see no responses. \
im working on that right now. nevertheless i would like to have the option of \
unixsockets if possible for different small scenarios (example, a single pod \
containing pdns+backend+redis+mongo) able to be replicated a few times in a small K8 \
cluster.

Hope it helps





> On 30 Nov 2023, at 05:18, Remi Gacogne via Pdns-users \
> <pdns-users@mailman.powerdns.com> wrote: 
> On 29/11/2023 01:07, Alexis Fidalgo wrote:
> > Problem is (and i've testing with golang and python) after the answer the \
> > "initialize" message, the socket is closed, so, getAllDomains message is being \
> > sent using a closed socket and that's why i don't see it on the responder side \
> > and pdns does not receive and answer, polls 2 times and reaches timeout.
> 
> Why do you think the socket is closed? It doesn't show up in your previous strace \
> log, and poll() wouldn't not time out but immediately return an error if the socket \
> had been closed. 
> > i can see there's no test for unixsocket in the source tree.
> 
> There is such a test in test-remotebackend-unix.cc
> 
> -- 
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
> 
> _______________________________________________
> Pdns-users mailing list
> Pdns-users@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users


[Attachment #5 (unknown)]

<html><head><meta http-equiv="content-type" content="text/html; \
charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: \
space; line-break: after-white-space;">Check this, i removed the code from my \
responder by now but it was something like this<div><br></div><div><div><div \
style="color: rgb(54, 54, 54); background-color: rgb(255, 255, 255); font-family: \
Menlo, Monaco, Consolas, &quot;Lucida Console&quot;, &quot;Roboto Mono&quot;, \
&quot;Courier New&quot;, monospace, Menlo, Monaco, &quot;Courier New&quot;, \
monospace; font-size: 15.6px; line-height: 23px; white-space: pre;"><div><span \
style="color: rgb(63, 151, 223);">func</span> main<span style="color: rgb(44, 44, \
44);">()</span> <span style="color: rgb(44, 44, 44);">{</span></div><div>    l<span \
style="color: rgb(44, 44, 44);">,</span> err <span style="color: rgb(44, 44, \
44);">:=</span> net<span style="color: rgb(44, 44, 44);">.</span>ListenUnix<span \
style="color: rgb(44, 44, 44);">(</span><span style="color: rgb(162, 86, \
55);">"unix"</span><span style="color: rgb(44, 44, 44);">,</span> <span style="color: \
rgb(44, 44, 44);">&amp;</span>UnixAddr<span style="color: rgb(44, 44, \
44);">{</span>Name<span style="color: rgb(44, 44, 44);">:</span> <span style="color: \
rgb(162, 86, 55);">"unix"</span><span style="color: rgb(44, 44, 44);">,</span>Net \
<span style="color: rgb(44, 44, 44);">:</span><span style="color: rgb(162, 86, \
55);">"/tmp/echo.sock"</span><span style="color: rgb(44, 44, \
44);">,})</span></div><div>    <span style="color: rgb(63, 151, 223);">if</span> err \
<span style="color: rgb(44, 44, 44);">!=</span> <span style="color: rgb(63, 151, \
223);">nil</span> <span style="color: rgb(44, 44, 44);">{</span></div><div>        \
panic<span style="color: rgb(44, 44, 44);">(</span>err<span style="color: rgb(44, 44, \
44);">)</span></div><div>    <span style="color: rgb(44, 44, \
44);">}</span></div><br><div>    <span style="color: rgb(63, 151, 223);">for</span> \
<span style="color: rgb(44, 44, 44);">{</span></div><div>        fd<span \
style="color: rgb(44, 44, 44);">,</span> err <span style="color: rgb(44, 44, \
44);">:=</span> l<span style="color: rgb(44, 44, 44);">.</span>Accept<span \
style="color: rgb(44, 44, 44);">()</span></div><div>        <span style="color: \
rgb(63, 151, 223);">if</span> err <span style="color: rgb(44, 44, 44);">!=</span> \
<span style="color: rgb(63, 151, 223);">nil</span> <span style="color: rgb(44, 44, \
44);">{</span></div><div>            panic<span style="color: rgb(44, 44, \
44);">(</span>err<span style="color: rgb(44, 44, 44);">)</span></div><div>        \
<span style="color: rgb(44, 44, 44);">}</span></div><br><div>    <span style="color: \
rgb(63, 151, 223);">go</span> <span style="color: rgb(63, 151, \
223);">func</span><span style="color: rgb(44, 44, 44);">(</span>fd net<span \
style="color: rgb(44, 44, 44);">.</span>Conn<span style="color: rgb(44, 44, \
44);">)</span> <span style="color: rgb(44, 44, 44);">{</span>  </div><div>        \
<span style="color: rgb(63, 151, 223);">var</span> buf <span style="color: rgb(44, \
44, 44);">[</span><span style="color: rgb(73, 104, 57);">1024</span><span \
style="color: rgb(44, 44, 44);">]</span><span style="color: rgb(63, 151, \
223);">byte</span></div><div>        n<span style="color: rgb(44, 44, 44);">,</span> \
err <span style="color: rgb(44, 44, 44);">:=</span> fd<span style="color: rgb(44, 44, \
44);">.</span>Read<span style="color: rgb(44, 44, 44);">(</span>buf<span \
style="color: rgb(44, 44, 44);">)</span></div><div>        <span style="color: \
rgb(63, 151, 223);">if</span> err<span style="color: rgb(44, 44, 44);">!=</span><span \
style="color: rgb(63, 151, 223);">nil</span><span style="color: rgb(44, 44, \
44);">{</span></div><div>            panic<span style="color: rgb(44, 44, \
44);">(</span>err<span style="color: rgb(44, 44, 44);">)</span></div><div>        \
<span style="color: rgb(44, 44, 44);">}</span></div><div>        data <span \
style="color: rgb(44, 44, 44);">:=</span> buf<span style="color: rgb(44, 44, \
44);">[</span><span style="color: rgb(73, 104, 57);">0</span><span style="color: \
rgb(44, 44, 44);">:</span>netip<span style="color: rgb(44, 44, \
44);">]</span></div><div>        println<span style="color: rgb(44, 44, \
44);">(</span><span style="color: rgb(162, 86, 55);">"Server got:"</span><span \
style="color: rgb(44, 44, 44);">,</span> <span style="color: rgb(63, 151, \
223);">string</span><span style="color: rgb(44, 44, 44);">(</span>data<span \
style="color: rgb(44, 44, 44);">))</span></div><div>        _<span style="color: \
rgb(44, 44, 44);">,</span> err <span style="color: rgb(44, 44, 44);">=</span> c<span \
style="color: rgb(44, 44, 44);">.</span>Write<span style="color: rgb(44, 44, \
44);">(</span>data<span style="color: rgb(44, 44, 44);">)</span></div><div>        \
<span style="color: rgb(63, 151, 223);">if</span> err <span style="color: rgb(44, 44, \
44);">!=</span> <span style="color: rgb(63, 151, 223);">nil</span> <span \
style="color: rgb(44, 44, 44);">{</span></div><div>            panic<span \
style="color: rgb(44, 44, 44);">(</span>err<span style="color: rgb(44, 44, \
44);">)</span></div><div>        <span style="color: rgb(44, 44, \
44);">}</span></div><div>    <span style="color: rgb(44, 44, 44);">}(</span>fd<span \
style="color: rgb(44, 44, 44);">)</span><span style="color: rgb(44, 44, 44); \
font-size: 15.6px;">}</span></div><div><span style="color: rgb(44, 44, \
44);">}</span></div></div></div><div><br></div><div>Using this (tested with pdns as \
client, a python cliend pasted below, a golang client and also netcat) \
causes</div><div><br></div><div>1. i can connect</div><div>2. i can send a message \
and it's printed</div><div>3. answer is received</div><div>4. complete lock, im not \
able to send any other message (yes, starting a different instance of any client \
sends another message and locks and so on)</div><div><br></div><div>If i add (right \
before }(fd)} (inside the for loop) an "fd.Close()" what happens is, there's no lock \
anymore, i can send messages over and over again, but (the famous but) doesn't matter \
which client i use (python, pdns, golang, java) i need to start the client \
everytime.</div><div><br></div><div>Example</div><div><br></div><div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">import socket</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">import sys</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">import time</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">import json</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br></div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">with \
socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">&nbsp; &nbsp; \
client.connect("/tmp/pra.sock")</div><div style="caret-color: rgb(0, 0, 0); color: \
rgb(0, 0, 0);"><br></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, \
0);">&nbsp; &nbsp; while True:</div><div style="caret-color: rgb(0, 0, 0); color: \
rgb(0, 0, 0);">&nbsp; &nbsp; &nbsp; &nbsp; \
client.send(json.dumps({'method':'initialize'}).encode('utf-8'))</div><div \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">&nbsp; &nbsp; &nbsp; &nbsp; v \
= client.recv(1024)</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, \
0);">&nbsp; &nbsp; &nbsp; &nbsp; print(f"--&gt; {v}")</div><div style="caret-color: \
rgb(0, 0, 0); color: rgb(0, 0, 0);">&nbsp; &nbsp; &nbsp; &nbsp; \
time.sleep(1)</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, \
0);"><br></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">&nbsp; \
&nbsp; client.close()</div></div><div><br></div><div>Scenario 1 (without the \
fd.Close())</div><div>Starting this, only sends and receive first message and \
locks</div><div><br></div><div>Scenario 2 (with fd.Close())</div><div>Starting this, \
sends and receives first message and throws an Exception because Broken \
Pipe</div><div><br></div><div>You can see im moving the read/answer to a different \
coroutine on every call to avoid blocking the socket in anyway but it doesnt \
help.</div><div><br></div><div>Note:</div><div>Using SOCKET_DGRAM i was able to loop \
for ever with no problem, i tried to set that in <a \
href="http://unixconnector.cc">unixconnector.cc</a>&nbsp;changing the socket type but \
is not that simple.</div><div><br></div><div>Note 2:</div><div>Helped a lot what you \
told me, i wrote a simple python script to stress the responder (putting pdns aside, \
only pcap'd real traffic to have real queries for my script) and i found something \
interesting, as more threads i open pushing the responder, there's queries that takes \
more and more time to be answered, a good reason for the http connection between pdns \
&gt; backend fails with timeout and DNS client see no responses. im working on that \
right now. nevertheless i would like to have the option of unixsockets if possible \
for different small scenarios (example, a single pod containing \
pdns+backend+redis+mongo) able to be replicated a few times in a small K8 \
cluster.</div><div><br></div><div>Hope it \
helps</div><div><br></div><div><br></div><div><br></div><div><br \
id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 30 Nov \
2023, at 05:18, Remi Gacogne via Pdns-users &lt;pdns-users@mailman.powerdns.com&gt; \
wrote:</div><br class="Apple-interchange-newline"><div><div>On 29/11/2023 01:07, \
Alexis Fidalgo wrote:<br><blockquote type="cite">Problem is (and i've testing with \
golang and python) after the answer the "initialize" message, the socket is closed, \
so, getAllDomains message is being sent using a closed socket and that's why i don't \
see it on the responder side and pdns does not receive and answer, polls 2 times and \
reaches timeout.<br></blockquote><br>Why do you think the socket is closed? It \
doesn't show up in your previous strace log, and poll() wouldn't not time out but \
immediately return an error if the socket had been closed.<br><br><blockquote \
type="cite">i can see there's no test for unixsocket in the source \
tree.<br></blockquote><br>There is such a test in \
test-remotebackend-unix.cc<br><br>-- <br>Remi Gacogne<br>PowerDNS.COM BV - \
https://www.powerdns.com/<br><br>_______________________________________________<br>Pdns-users \
mailing list<br>Pdns-users@mailman.powerdns.com<br>https://mailman.powerdns.com/mailma \
n/listinfo/pdns-users<br></div></div></blockquote></div><br></div></div></body></html>




_______________________________________________
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users


[prev in list] [next in list] [prev in thread] [next in thread]