[prev in list] [next in list] [prev in thread] [next in thread]
List: nmap-dev
Subject: Tudor's Status Report - #9 of 17
From: Tudor-Emil COMAN <tudor_emil.coman () cti ! pub ! ro>
Date: 2016-06-28 5:50:33
Message-ID: AM3PR01MB0694491D442CD2B66AE4535F94220 () AM3PR01MB0694 ! eurprd01 ! prod ! exchangelabs ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi folks,
I spent the last week solving some bugs and crashes in the IOCP engine afte=
r I noticed some bad accuracy results as I kept increasing the parallelism =
level.
That Amazon Instance Brandon gave me access was really helpful in identifyi=
ng some bad situations.
Firstly there was an issue with the error handling in the sense that when a=
n asynchronous I/O operation was initiated if the operation failed the stat=
e of the event wouldn't be updated.
ConnectEx has been giving me troubles with the fact that it's not reporting=
the proper WSA errors with the WSAGetLastError function.
For example instead of WSAECONNREFUSED (10061) it would return ERROR_CONNEC=
TION_REFUSED (1225) which I think it's the same error. The problem with tha=
t is that the error wouldn't be caught by the connect handler and it would =
exit with an unexpected error. Simply transforming the error into it's WSA =
counterpart seemed to do the trick.
This bug was documented here: https://svn.boost.org/trac/boost/ticket/10744
Also further attention was needed when handling expired events. For the loo=
p I chose GetQueuedCompletionStatus which takes just one completed event fr=
om the queue so we are processing just one event per loop (with the expired=
events also). Now it may be the case that a completed event finishes befor=
e it is ever taken out and it is marked as expired which is would impact pe=
rformance and even accuracy a lot. I manage to solve this one by writing my=
own process_expired_events in engine_iocp so that my code doesn't interfer=
e with that same function in nsock_core.
As per last week I made some graphs but they should be taken with a grain o=
f salt as it is not finished and the results may be flawed.
I did look at the actual scan results almost every single time and they com=
pare pretty good between each other with a similar number of tcpwrapped por=
ts, unrecognized probes and detected services but still, the results are wa=
y too good in favor of IOCP with the Sunday one in particular not looking r=
ight, so I'll look into why is that this week.
This week's both graphs are made on the same 5000 live hosts.
I also was working on creating my own script to have more control of the te=
sting due to the fact that Nmap is too complex for this.
The scripts in nsock/examples and nsock/tests where all made for Linux so I=
had to start from scratch and so far I've nailed down the compilation part=
and made a program that just makes a connection to a target IP. I would e=
xtend it to send a Get request and use it on a big group of http servers.
Other cases:
Localhost Scanning: I tested if localhost scanning works by doing a service=
scan on my PC and it worked.
IPv6 Scanning: I scanned the link-local IPv6 address of my old PC with the =
-6 option and it worked.
I didn't really have an aim this week but I will focus on finishing the eng=
ine by the next status report.
Accomplishments:
- Solved a bunch of issues with engine_iocp.
- Started working on a proper example script for it.
Priorities:
- Concentrate on the accuracy of the scans to spot other potential errors.
- Start using SVN instead of git and make a branch or something.
- Finish the example script.
- Investigate possible registry tweaks.
Time really flies doesn't it,
Tudor
[Attachment #5 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} \
--></style> </head>
<body dir="ltr">
<div id="divtagdefaultwrapper" \
style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi folks,</p>
<p><br>
</p>
<p><br>
</p>
<p>I spent the last week solving some bugs and crashes in the IOCP engine after \
I noticed some bad accuracy results as I kept increasing the parallelism level.</p> \
<p>That Amazon Instance Brandon gave me access was really helpful in identifying some \
bad situations.</p> <p><br>
</p>
<p>Firstly there was an issue with the error handling in the sense that when \
an asynchronous I/O operation was initiated if the operation failed \
the state of the event wouldn't be updated.</p> <p><br>
</p>
<p>ConnectEx has been giving me troubles with the fact that it's not reporting the \
proper WSA errors with the WSAGetLastError function.</p> <p>For example instead \
of <span>WSAECONNREFUSED (10061) it would \
return <span>ERROR_CONNECTION_REFUSED (1225) which I think it's the same error. \
The problem with that is that the error wouldn't be caught by the connect handler and \
it would exit with an unexpected error. Simply transforming the error into it's WSA \
counterpart seemed to do the trick.</span></span></p> <p><span><span>This bug was \
documented here: <a href="https://svn.boost.org/trac/boost/ticket/10744" \
class="OWAAutoLink" id="LPlnk475024" \
title="https://svn.boost.org/trac/boost/ticket/10744 Ctrl+Click or tap to follow \
the link">https://svn.boost.org/trac/boost/ticket/10744</a></span><br> </span></p>
<p><span><br>
</span></p>
<p><span>Also further attention was needed when handling expired events. For the loop \
I chose <span>GetQueuedCompletionStatus which takes just one completed event \
from the queue so we are processing just one event per loop (with the expired events \
also). Now it may be the case that a completed event finishes before it is ever \
taken out and it is marked as expired which is would impact performance and even \
accuracy a lot. I manage to solve this one by writing my own process_expired_events \
in engine_iocp so that my code doesn't interfere with that same function in \
nsock_core.</span></span></p> <p><span><span><br>
</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>As per last week I made some graphs but they should be taken with a \
grain of salt as it is not finished and the results may be flawed.</span></span></p> \
<p><span><span>I did look at the actual scan results almost every single time and \
they compare pretty good between each other with a similar number of tcpwrapped \
ports, unrecognized probes and detected services but still, the results are way too \
good in favor of IOCP with the Sunday one in particular not looking \
right, <span style="font-family: Calibri, Arial, Helvetica, sans-serif, \
"Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, \
"Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: \
16px;">so I'll look into why is that this week</span>.</span></span></p>
<p><span><span>This week's both graphs are made on the same 5000 live \
hosts.</span></span></p> <p><span><span><br>
</span></span></p>
<p><span><span>I also was working on creating my own script to have more control of \
the testing due to the fact that Nmap is too complex for this.</span></span></p> \
<p><span><span>The scripts in nsock/examples and nsock/tests where all made for \
Linux so I had to start from scratch and so far I've nailed down the compilation part \
and made a program that just makes a connection to a target IP. I would extend \
it to send a Get request and use it on a big group of http \
servers.</span></span></p> <p><span><span><br>
</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Other cases:</span></span></p>
<p><span><span>Localhost Scanning: I tested if localhost scanning works by doing \
a service scan on my PC and it worked.</span></span></p> <p><span><span>IPv6 \
Scanning: I scanned the link-local IPv6 address of my old PC with the -6 \
option and it worked.</span></span></p> <p><span><span></span></span></p>
<p><br>
</p>
<p><span><span><br>
</span></span></p>
<p><span><span>I didn't really have an aim this week but I will focus on finishing \
the engine by the next status report.</span></span></p> <p><span><span><br>
</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Accomplishments:</span></span></p>
<p><span><span>- Solved a bunch of issues with engine_iocp.</span></span></p>
<p><span><span>- Started working on a proper example script for it.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Priorities:</span></span></p>
<p><span><span>- Concentrate on the accuracy of the scans to spot other potential \
errors.</span></span></p> <p><span><span>- Start using SVN instead of git and make a \
branch or something.</span></span></p> <p><span><span>- Finish the example \
script.</span></span></p> <p></p>
<p>- Investigate possible registry tweaks.</p>
<p><br>
</p>
<p></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Time really flies doesn't it,</span></span></p>
<p><span><span>Tudor</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span><br>
</span></span></p>
<p><br>
</p>
<p><span><br>
</span></p>
</div>
</body>
</html>
["Saturday-comparison.png" (image/png)]
["Sunday-Comparison.png" (image/png)]
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic