[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-core-devel
Subject:    Re: kio_http, persistent connections
From:       Dawit Alemayehu <adawit () kde ! org>
Date:       2001-06-04 23:16:52
[Download RAW message or body]

On Monday 04 June 2001 14:09, Lars Knoll wrote:
> On Monday 04 June 2001 19:55, Dirk Mueller wrote:
> > On Mon, 04 Jun 2001, Rik Hemsley wrote:
> > > When requesting a web page which includes quite a few (about 10) small
> > > images, it appears that kio_http creates about 4 connections to the
> > > server. This, I would guess, is to allow 'parallel' retrieval of the
> > > various components that make up the page.
> >
> > Yes. the image retrieval algorithm is currently using 4 connections in
> > parallel (head of khtml/misc/loader.cpp).
> >
> > > This is fine, if kio_http will send "Connection: close" in a future
> > > request on the same connection. Unfortunately, it doesn't. Instead, it
> > > seems try to open another connection to the server instead.
> >
> > euhm. you mean when there is a connection: keep-alive you're only allowed
> > to open one connection to the server ?
>
> 2 if I remember the http specs correctly. I'm pretty sure 4 are not
> allowed. This makes it possible to already retrieve style sheets and images
> while the main page is still loading.
>
> > > According to RFC2616, clients should simply pipeline requests without
> > > waiting for a response from the server.
> >
> > so it should keep sending request headers without waiting for the first
> > request to be finished ?
>
> It is at least allowed to do so, and should be a lot faster than waiting
> for each request to be finished.
>
> > that would explain why KIO is still a few magnitudes slower in retrieving
> > webpages than i.e. IE/Netscape.
>
> If pipelining is not implemented, that would explain a lot.
>
> > > If kio_http were to limit itself to 4 connections and, when it has
> > > reached this limit, send further requests (pipelined) on the existing
> > > 4 connections, with "Connection: close" as a header in the final
> > > request, there would be no problem.
> >
> > Well, it can't do a connection:close because it never really knows when
> > there is no further request. I guess we have to do that via a timeout. I
> > don't understand why it has to be done via a Connection: close instead of
> > simply closing the socket.
>
> The http server send us a timeout anyway. We can just keep it open up to
> MAX(server_timeout, our_max_timeout).
>
> > > With the current implementation of kio_http, I need to set quite a
> > > high simultaneous connection limit in the server, to handle the case
> > > when kio_http fires off what appears to be an unlimited number of
> > > connections.
> >
> > unlimited number ? huh?? that is a bug then.
>
> As said above, it should be maximal two connections to the same server.
>
> > > will be requested. Even with this knowledge, a perfect implementation
> > > would be difficult, as the user may quickly select a link on the
> > > page, so some requests are cancelled and new ones are added.
> >
> > Well, it seems that you understand the issue. Make suggestions on how
> > khtml/kio_http should improve.
> >
> > > I think that the best solution would simply be to ensure that no
> > > more than 4 connections are made to one host.
> >
> > why ?
>
> Because of the http specs? Because opening 20 connections to one host
> thrashes the server if every client does it.
>
> > > I would still need a large simultaneous limit in my server, because
> > > kio_http does not close the connections when finished - it just leaves
> > > them to timeout.
> >
> > but then the slaves will be reused on a further request (click on a
> > link). what do you mean by timeout ? the kio timeout or the server
> > timeout ?
> >
> > > Perhaps I will have to decide that persistent connections are a bad
> > > idea, due to the way that kio_http makes connections (and fails to
> > > disconnect when it should.)
> >
> > Huh? Why that ?
> >
> > I still fail to understand the problem.
>
> The problem is, that a lot of open connections are bad for the server. We
> should really stick to the http specs. Here's the relevant quote btw:

It is not only a problem for the servers, but also "possibly" the client because of
the way TCP works.  If the requests are too small some TCP stack implementations
might hold off on sending the request until they have the minimum number of enough
packets to trigger transmission.  This of course contributes to the degrading of page
retrieval speed.

Regards,
Dawit A.

[prev in list] [next in list] [prev in thread] [next in thread]