From kde-core-devel  Mon Jun 04 18:09:53 2001
From: Lars Knoll <knoll () kde ! org>
Date: Mon, 04 Jun 2001 18:09:53 +0000
To: kde-core-devel
Subject: Re: kio_http, persistent connections
X-MARC-Message: https://marc.info/?l=kde-core-devel&m=99167833909688

On Monday 04 June 2001 19:55, Dirk Mueller wrote:
> On Mon, 04 Jun 2001, Rik Hemsley wrote:
> > When requesting a web page which includes quite a few (about 10) small
> > images, it appears that kio_http creates about 4 connections to the
> > server. This, I would guess, is to allow 'parallel' retrieval of the
> > various components that make up the page.
>
> Yes. the image retrieval algorithm is currently using 4 connections in
> parallel (head of khtml/misc/loader.cpp).
>
> > This is fine, if kio_http will send "Connection: close" in a future
> > request on the same connection. Unfortunately, it doesn't. Instead, it
> > seems try to open another connection to the server instead.
>
> euhm. you mean when there is a connection: keep-alive you're only allowed
> to open one connection to the server ?

2 if I remember the http specs correctly. I'm pretty sure 4 are not allowed. 
This makes it possible to already retrieve style sheets and images while the 
main page is still loading.

> > According to RFC2616, clients should simply pipeline requests without
> > waiting for a response from the server.
>
> so it should keep sending request headers without waiting for the first
> request to be finished ?

It is at least allowed to do so, and should be a lot faster than waiting for 
each request to be finished.

> that would explain why KIO is still a few magnitudes slower in retrieving
> webpages than i.e. IE/Netscape.

If pipelining is not implemented, that would explain a lot. 

> > If kio_http were to limit itself to 4 connections and, when it has
> > reached this limit, send further requests (pipelined) on the existing
> > 4 connections, with "Connection: close" as a header in the final
> > request, there would be no problem.
>
> Well, it can't do a connection:close because it never really knows when
> there is no further request. I guess we have to do that via a timeout. I
> don't understand why it has to be done via a Connection: close instead of
> simply closing the socket.

The http server send us a timeout anyway. We can just keep it open up to 
MAX(server_timeout, our_max_timeout).

> > With the current implementation of kio_http, I need to set quite a
> > high simultaneous connection limit in the server, to handle the case
> > when kio_http fires off what appears to be an unlimited number of
> > connections.
>
> unlimited number ? huh?? that is a bug then.

As said above, it should be maximal two connections to the same server.

> > will be requested. Even with this knowledge, a perfect implementation
> > would be difficult, as the user may quickly select a link on the
> > page, so some requests are cancelled and new ones are added.
>
> Well, it seems that you understand the issue. Make suggestions on how
> khtml/kio_http should improve.
>
> > I think that the best solution would simply be to ensure that no
> > more than 4 connections are made to one host.
>
> why ?

Because of the http specs? Because opening 20 connections to one host 
thrashes the server if every client does it.

> > I would still need a large simultaneous limit in my server, because
> > kio_http does not close the connections when finished - it just leaves
> > them to timeout.
>
> but then the slaves will be reused on a further request (click on a link).
> what do you mean by timeout ? the kio timeout or the server timeout ?
>
> > Perhaps I will have to decide that persistent connections are a bad
> > idea, due to the way that kio_http makes connections (and fails to
> > disconnect when it should.)
>
> Huh? Why that ?
>
> I still fail to understand the problem.

The problem is, that a lot of open connections are bad for the server. We 
should really stick to the http specs. Here's the relevant quote btw:

"Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD maintain AT MOST 2 connections with any
   server or proxy. A proxy SHOULD use up to 2*N connections to another
   server or proxy, where N is the number of simultaneously active
   users. These guidelines are intended to improve HTTP response times
   and avoid congestion of the Internet or other networks."

Lars