[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-core-devel
Subject:    Re: kio_http, persistent connections
From:       Waldo Bastian <bastian () kde ! org>
Date:       2001-06-04 17:59:25
[Download RAW message or body]

On Monday 04 June 2001 08:35, Rik Hemsley wrote:
> I'm busy implementing persistent connection support in my web server
> (kpf).
>
> My server supports simultaneous-connection limiting, intended to keep
> resource usage down.
>
> While testing persistent connection support using konqueror, I noticed
> that if I set a low value (e.g. 4) for the simultaneous connection
> limit, there is a problem with kio_http.
>
> When requesting a web page which includes quite a few (about 10) small
> images, it appears that kio_http creates about 4 connections to the
> server. This, I would guess, is to allow 'parallel' retrieval of the
> various components that make up the page.
>
> kio_http tells the server that it wants to keep the connection open (it
> doesn't send "Connection: close" and it explicitly sends "Connection:
> Keep-Alive").
>
> This is fine, if kio_http will send "Connection: close" in a future
> request on the same connection. Unfortunately, it doesn't. Instead, it
> seems try to open another connection to the server instead.

Each kio_http process will have its own connection.

> This happens when kio_http has not yet retrieved the full response from
> the server.

> According to RFC2616, clients should simply pipeline requests without
> waiting for a response from the server.

No, they _may_ pipeline requests. It doesn't say they _should_. Pipelining is 
currently not supported by the io-slave architecture which handles requests 
one by one.

"
8.1.2.2 Pipelining
 
   A client that supports persistent connections MAY "pipeline" its
   requests (i.e., send multiple requests without waiting for each
   response).
"

> I can see that pipelining all the requests necessary for one page, on
> one connection, would lose the benefit of parallelism, so it makes sense
> that kio_http would use more than one connection.
>
> If kio_http were to limit itself to 4 connections and, when it has
> reached this limit, send further requests (pipelined) on the existing
> 4 connections, with "Connection: close" as a header in the final
> request, there would be no problem.

io-slaves do not know if more requests will follow.

> With the current implementation of kio_http, I need to set quite a
> high simultaneous connection limit in the server, to handle the case
> when kio_http fires off what appears to be an unlimited number of
> connections.

There should be no more than about 4 concurrent open connections to the same 
host. How many kio_http processes do you have running? How many open network 
connection do you have?

> I can see that generating "Connection: close" intelligently must be
> difficult, as kio_http would have to know some information about
> the web page it is accessing, i.e. the number of images (etc.) that
> will be requested. Even with this knowledge, a perfect implementation
> would be difficult, as the user may quickly select a link on the
> page, so some requests are cancelled and new ones are added.

When a request is canceled, the slave is killed which closes the connection.

> I think that the best solution would simply be to ensure that no
> more than 4 connections are made to one host.

That should already be the case. Are you sure your server correctly detects 
connections being closed?

"Clients and servers SHOULD both constantly watch for the other side of the 
transport close, and respond to it as appropriate."

> I would still need a large simultaneous limit in my server, because
> kio_http does not close the connections when finished - it just leaves
> them to timeout.

> After a page with a few graphics has been accessed, I will have 4
> connections waiting 60 seconds to timeout, so the connection pool will
> still be artificially filled, but this is better than > 4 connections,
> at least.

You could reduce your timeout.

> Perhaps I will have to decide that persistent connections are a bad
> idea, due to the way that kio_http makes connections (and fails to
> disconnect when it should.)

I am not aware that they _should_ disconnect at any specific time.

> In that case, I will simply go back to sending "Connection: close"
> with each response, though I would prefer not to do this, as
> persistent connections can save a lot of time and resources.

Cheers,
Waldo
-- 
bastian@kde.org | SuSE Labs KDE Developer | bastian@suse.com

[prev in list] [next in list] [prev in thread] [next in thread]