[prev in list] [next in list] [prev in thread] [next in thread] 

List:       pywikipediabot-users
Subject:    Re: [Pywikipedia-l] SVN:  [4774] branches/rewrite/pywikibot
From:       "Merlijn van Deen" <valhallasw () arctus ! nl>
Date:       2007-12-29 1:10:09
Message-ID: 1480.10.0.0.4.1198890609.squirrel () elladan
[Download RAW message or body]

On Sat, December 29, 2007 1:08 am, valhallasw@svn.wikimedia.org wrote:
> Revision: 4774
> Author:   valhallasw
> Date:     2007-12-29 00:08:32 +0000 (Sat, 29 Dec 2007)
>
> Log Message:
> -----------
> Added epydoc configuration
> Added threaded cookie-eating HTTP library, based on httplib2
>
> Added Paths:
> -----------
>     branches/rewrite/pywikibot/data/threadedhttp.py
>     branches/rewrite/pywikibot/epydoc.cfg

epydoc.cfg contains a simple configuration to create epydoc documentation.
Threadedhttp.py is a wrapper around httplib2, which implements a threaded
system and cookie support.

Example code:

import logging
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s
[%(threadName)s] %(message)s')

import threadedhttp
cj = threadedhttp.LockableCookieJar()
cp = threadedhttp.ConnectionPool()
import Queue
q = Queue.Queue()

for i in range(10):
  threadedhttp.HttpProcessor(q,cj,cp).start()

for i in range(10):
  q.put(threadedhttp.HttpRequest('http://google.com'))
  q.put(threadedhttp.HttpRequest('http://en.wikipedia.org'))

for i in range(10):
  q.put(None)


I have not yet come up with a good set of unit tests as most of the
functions are hard to unit-test... any suggestions on that part are very
welcome. I have implemented epydoc-style comments for all external
functions.

For the ConnectionPool: I have now implemented a connection pool with a
maximum of 5 saved concurrent connections to each host. This means that it
is possible to have up to about N(threads) connections to one host open.
We may want to change this to a maximum of 5 concurrent connections, and
locking if a connection is required that is unavailable at that time.
Secondly, no timeouts are implemented, while these probably should be
implemented: keeping connections to rarely-used servers open is useless.
Thirdly, we may want to limit the total number of connections for those
who do not have the luxury of \infty outbound connections.
Any suggestions are, as always, very welcome!

--valhallasw


_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l@lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
[prev in list] [next in list] [prev in thread] [next in thread]