From kde-core-devel  Mon Jun 04 19:57:28 2001
From: Rik Hemsley <rik () kde ! org>
Date: Mon, 04 Jun 2001 19:57:28 +0000
To: kde-core-devel
Subject: Re: kio_http, persistent connections
X-MARC-Message: https://marc.info/?l=kde-core-devel&m=99168700624119

#if Dirk Mueller
> > This is fine, if kio_http will send "Connection: close" in a future
> > request on the same connection. Unfortunately, it doesn't. Instead, it
> > seems try to open another connection to the server instead.
> 
> euhm. you mean when there is a connection: keep-alive you're only allowed to 
> open one connection to the server ?

No, you're allowed to open 2 according to the RFC. But you should re-use
them :)

> > According to RFC2616, clients should simply pipeline requests without
> > waiting for a response from the server.
> 
> so it should keep sending request headers without waiting for the first 
> request to be finished ?

Yes.

> that would explain why KIO is still a few magnitudes slower in retrieving 
> webpages than i.e. IE/Netscape. 

It would certainly be one explanation.

> > If kio_http were to limit itself to 4 connections and, when it has
> > reached this limit, send further requests (pipelined) on the existing
> > 4 connections, with "Connection: close" as a header in the final
> > request, there would be no problem.
> 
> Well, it can't do a connection:close because it never really knows when 
> there is no further request. I guess we have to do that via a timeout. I 
> don't understand why it has to be done via a Connection: close instead of 
> simply closing the socket. 

Yes, I understand that it's very difficult to know when there will be
no more requests - we can't predict the future.

It doesn't have to be done with Connection: close. Closing the socket
is fine.

> > With the current implementation of kio_http, I need to set quite a
> > high simultaneous connection limit in the server, to handle the case
> > when kio_http fires off what appears to be an unlimited number of
> > connections.
> 
> unlimited number ? huh?? that is a bug then. 

I say unlimited, because it seems to just keep making connections when
some are busy. The fact that it uses more than 4 is a problem, anyway,
when it should be using 2 in theory.

> > will be requested. Even with this knowledge, a perfect implementation
> > would be difficult, as the user may quickly select a link on the
> > page, so some requests are cancelled and new ones are added.
> 
> Well, it seems that you understand the issue. Make suggestions on how 
> khtml/kio_http should improve. 
> 
> > I think that the best solution would simply be to ensure that no
> > more than 4 connections are made to one host.
> 
> why ?

Actually, it should be 2, because that's what the RFC specifies as
the maximum.

Also, I expect other servers have simultaneous connect limits too. It's
quite important to do this, because otherwise you could end up storing a
_lot_ of data in memory.

Of course, I try not to simply drop incoming connections when I hit
the limit. Instead, I queue up the fd of the connection in a (large) list
of ints and go back to it later.
 
> > I would still need a large simultaneous limit in my server, because
> > kio_http does not close the connections when finished - it just leaves
> > them to timeout.
> 
> but then the slaves will be reused on a further request (click on a link). 
> what do you mean by timeout ? the kio timeout or the server timeout ?

The slaves are re-used, but new slaves are created too, if the existing
slaves are 'busy'. That's the problem.

> > Perhaps I will have to decide that persistent connections are a bad
> > idea, due to the way that kio_http makes connections (and fails to
> > disconnect when it should.)
> 
> Huh? Why that ? 
> 
> I still fail to understand the problem. 

Hopefully this has clarified a little.

Rik