[prev in list] [next in list] [prev in thread] [next in thread] 

List:       imap
Subject:    Re: nntp-extensions Re: ietf-nntp NNTP SEARCHextensioninternet-draft available
From:       Jeff Coffler  <jeffc () Netmanage ! COM>
Date:       1996-10-22 10:58:12
[Download RAW message or body]

Hi Ben,

--- On Mon, 21 Oct 1996 16:22:17 -0700  Ben Polk <bpolk@netscape.com> wrote:

Ben, I'm afraid I disagree with you.  My reasoning follows:

[Snip]

> Thanks for your comments Jeff.
> 
> As you might suspect from me bringing up the option of returning
> an XOVER-like response, we considered this.  For the reasons 
> below, we chose not to do it.
> 
> 1. I'd rather keep a few thousand bytes on disk for the client for
> a short time than receive many fetch by message-id commands.  This
> information is in the form of an .overview file for the virtual
> newsgroup, which the server is already adept at handling.

Current, context maintained by the server is really minimal.  We have
a current group, a current article, and a handful of other items.

With your suggestion, the server will now keep a whole lot of context,
particularly for a lot of clients.  Consider: What if somebody does a
SEARCH that returns a LOT of data.  You may have some number of megabytes
of OVERVIEW data - or more.  That's a lot of stuff for the server to keep
around.  Now multiply that by a few dozen clients that may concurrently
be doing some searches.  It's a LOT of context!

> 3. Your statement in passing that the server keeps "minimal" 
> state about the client is not true in most servers.  The nnrpd 
> process in INN is a big bad boy.  In the Netscape server the
> corresponding per-user threads use considerable resources.  And 
> lord help you if you do enough fetches by message-id to cause the 
> history file to get mapped in.  The additional state we're adding 
> here isn't significant.  Actually executing the searches and 
> maintaining the full-text indexes both dwarf the resource 
> requirements of keeping this state.  And these are just the
> price that has to be paid to support real full-text searching.

Don't mistake INN arihetecture with what a server actually needs.
NNRPD is a big bad boy because it's written for the lowest common
denominator UNIX.  It doesn't use threading, or shared memory, or
any of those sorts of things.  Heck, things like threading weren't
even around (for any significant number of platforms, anyway) when
NNRPD was written.  This is an INN architecture issue issue, not a
news server issue per se.

There do exist NNTP servers (I know - I wrote one) that runs on a
subset of what INN runs on, but uses things like threads, shared
memory, etc.  As a result, a client connection takes about 1000
bytes of context, and that's because my command buffer is in the
context.  If I wanted to make that shared (on an as-needed basis),
my context overhead would drop to about 488 bytes (or less) per
client.  For me, a connection is a connection - I have no architectural
reason to separate "servers" from "clients".  Indeed, I don't even
know (for sure) if a connection is for a server or a client until I
start getting commands (if I get a server command, like TAKETHIS or
IHAVE, then I know it's a server for sure and check for appropriate
access rights).

Fetching by message ID is also an INN issue, not a generic news server
issue.  It's possible to write code so that fetching by message ID does
NOT require mapping in the entire history file (or even a significant
portion of it).  Heck, look at databases today: they can have billions
of records (history databases look paltry in comparison).  A good
database can fetch MANY MANY records per second with minimal disk I/O
and minimal memory overhead.

I don't mean to imply that INN should use a database - it shouldn't (in
my opinion, anyway) for a variety of reasons.  What I'm saying, though,
is that fetching by message ID could by VERY cheap.  You shouldn't shy
away from that, particularly if it significantly simplifies the news
server implementation, just because INN, today, doesn't handle it well.

> 4. A new XOVER-like search response is more work for clients.  
> Clients currently know how to read groups.  So once they get the 
> name of the search result group, they don't have to do anything new 
> to display the results.

Clients, today, generally know how to read XOVER responses too!  At
least I'm not aware of any clients that do NOT know how to deal with
this (it's pretty crutial for performance purposes).

Basically, a client would need to keep a flag that stated that the
current XOVER database was from a SEARCH.  Then, when a client wanted
an article, fetch via message ID (because the flag is set), and voila!
This is better, anyway - if you fetch via Message ID, and one of the
articles (of a cross-posting) has expired, you'll still get a hit from
another article (as long as one of the cross-posted articles has not
expired yet).

The additional work on the client for all this is very easy.  More work
would likely go into nice GUI UIs for search criteria (in my opinion,
anyway).

> Would returning an XOVER-like search response work?  Absolutely.
> It's just not as good as the alternative.

This is where we disagree.  XOVER can be a great alternative, and much
easier for the server to deal with (no state needs to be saved after
the XOVER search response is returned).

Thoughts?  Opinions?

	-- Jeff

[prev in list] [next in list] [prev in thread] [next in thread]