[prev in list] [next in list] [prev in thread] [next in thread] 

List:       httpclient-users
Subject:    Re: Custom Connection Management
From:       "Sam Berlin" <sberlin () gmail ! com>
Date:       2008-06-12 21:44:34
Message-ID: 19196d860806121444v2212e1aema664bd5c8b16f856 () mail ! gmail ! com
[Download RAW message or body]

The internal connection manager, in a nutshell, is there to reduce the
amount of times new connections have to be created & prevent a server
from being overloaded with too many connections.  It exposes a lot of
bells & whistles (and HttpClient makes use of them) so that a single
connection can be 'managed' by the connection manager.  The premise is
that if you make one HTTP request and later make another, it should
reuse the existing HTTP connection instead of creating a new one.
There's also checks to make sure that connections to a given host stay
under a certain size (to play nicely with the server and be in
accordance with some RFC recommendations).

To accomplish this, there's a *lot* of layers to enabling hooks into
knowing when a connection is in use, or idle... or when a connection
requires different routes (such as proxies)... or when different
states can't be mixed (such as TLS, HTTPS, HTTP).  Really the best way
into learning is by stepping through what happens in
HttpClient.execute, how it gets a connection & all that.  Still, it's
incredibly complicated.  If you do step through it and learn things,
please feel free to share the learnings here or on the HttpComponents
wiki -- it would help out a lot of people.

Sam

On 6/12/08, Quintin Beukes <quintin@last.za.net> wrote:
> On 6/12/08, Oleg Kalnichevski <olegk@apache.org> wrote:
> > Quintin Beukes wrote:
> >
> > > Hey,
> > >
> > >
> >
> >  Hi Quintin
> >
> >
> >
> > > HttpClient is great. Funny enough HttpClient is faster than HttpCore.
> > >
> >
> >  This is not really plausible as HttpClient uses HttpCore as its low level
> > transport layer.
> >
>
> Yes, like I mentioned in my other e-mail. I basically didn't explain
> my whole situation in this line.
>
> >
> >
> > > I couldn't compare it to NIO, because the NIO examples didn't want to
> > > work with me. It's like the connections only paused and didn't
> > > continue, eventually just timeing out. I didn't research it further
> > > since the throughputs on HttpClient was sufficient.
> > >
> > >
> >
> >  The use of NIO will not achieve a higher throughput compared ti the classic
> > IO unless the number of concurrent connections is much greater than a
> > thousand. NIO has its advantages, but the raw throughput is not one of
> > those.
> >
> >
> >
> > > Eitherhow, let me get to the point.
> > >
> > > I am using HttpClient to fetch some of my resources of backend
> > > servers. So far it's been perfect for this, but I want to do some
> > > optimizations. I have made two separate connection managers and then
> > > have maps to select which I want to use, based on the resource I want
> > > to fetch. I was hoping to combine this all into a single connection
> > > manager that selects the resource based on the URL and previous
> > > history (which it builds up over time).
> > >
> > > These smart selections are complex, so I'm not going to describe them
> > > here. They are irrelevant to my question anyhow.
> > >
> > > My question is basically how I would approach this. I assume I have to
> > > extend connection manager and supply it to the constructor of
> > > DefaultHttpClient. But what now? What do I have to do to manage
> > > persistent connections? How do I store/retrieve information on
> > > connections? How do I return such connections? How does all this work?
> > >
> > >
> >
> >  Management of persistent connections is a fairly complex subject. I
> > personally do not see a lot of valid use cases for writing a completely new
> > connection manager. One may want to extend the existing implementation and
> > override certain aspects of it, but this is about it.
> >
> >  We currently do not have any documentation of the internal design of the
> > existing connection managers, so studying the source code is your only
> > choice.
> >
> >  I am sorry but this is all I can do for you at this point.
>
> Would it be possible to do it with relatively little effort. Little
> effort here is compared to an extensive and large amount of code, so
> in itself I don't mean like 10/20 lines, just not thousands of changes
> and multiple classes and so forth.
>
> Could you perhaps just explain to me how the internal connection
> manager works. Just a general overview, so I sort of know what I'm
> looking at when I read the code.
>
> >
> >  Oleg
> >
> >
> >
> > > If anyone could guide me, or point me in the right direction (docs,
> > > examples, etc) it would all be greatly appreciated.
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> >  To unsubscribe, e-mail:
> > httpclient-users-unsubscribe@hc.apache.org
> >  For additional commands, e-mail:
> > httpclient-users-help@hc.apache.org
> >
> >
>
>
> --
> Quintin Beukes
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

[prev in list] [next in list] [prev in thread] [next in thread]