[prev in list] [next in list] [prev in thread] [next in thread] 

List:       nmap-dev
Subject:    [NSE] ideas for http library
From:       Patrick Donnelly <batrick () batbytes ! com>
Date:       2011-10-27 6:42:53
Message-ID: CACh33FrF4TPZXVXZm-v7Pe-8EptYbWgpS07n5wK5f36mGC3D9Q () mail ! gmail ! com
[Download RAW message or body]

I promised a while back I'd make a post about this earlier this Summer
and just now remembered:

We were discussing pipelining for the http library and how it doesn't
currently use the caching mechanism of http.get. I brought up how it'd
be useful if pipelining were transparent and automatic via http.get
and the rest of the current pipelining API were thrown out.

Essentially, the http library would have separate worker threads (one
thread for each host) which would concatenate requests for URIs to a
single host via pipelining and give back the response to the script
which requested it via some sort of callback (or similar design). The
advantages of this is that we have only one active connection to an
HTTP server which reduces load, improves overall performance, and
increases parallelism (more scripts can do HTTP requests at once).

Some things to consider:

o The worker thread would "belong" to the first script which made the
request. I believe currently that worker thread would function
normally despite that script possibly finishing before its work is
done. Perhaps there should be a separate "library thread" function
which would have separate ownership mechanisms.

o http.get should maybe block in certain circumstances e.g. when the
pipeline "queue" is full. That way a script doesn't make thousands of
requests and then wait for callbacks to return results.

o Other http functions should also work in the same pipeline, e.g. http.head.

o Obviously, the cache should function in the new system.

Comments welcome.

-- 
- Patrick Donnelly
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/
[prev in list] [next in list] [prev in thread] [next in thread]