[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kmail-devel
Subject:    Re: CIA proposal (was: ClientInterface)
From:       Don Sanders <sanders () kde ! org>
Date:       2003-07-25 6:30:53
[Download RAW message or body]

On Friday 25 July 2003 00:15, Marc Mutz wrote:
> Hi Don!
>
> I'm in the process of going home, so I'll address some points on
> shortly. I apologize.
>
> On Wednesday 23 July 2003 06:58, Don Sanders wrote:
> > I do wonder if multiple processes ro mmap the same file whether
> > the memory used to back the mmap is shared by all those
> > processes.
>
> It is (if you use MAP_SHARED).

That's nice.

> > My concerns:
> >
> > 1) This doesn't actually address the issue of external mail
> > clients modifying ~/Mail does it? I mean not even for MailDir as
> > this approach requires KMail to be running to detect changes in
> > ~/Mail.
>
> New mail is delivered into the new/ subdir for maildir. The first
> instance of KMail to close the folder or have a user click on the
> new message will move the mails from new/ to cur/, updating the
> index along the way.
>
> Of course, MUAs wishing to modify ~/Mail need to conform to CIA/4,
> but maildir-compliant MDAs work without modifications.

Ok, but what about the case of another (non-KMail) mail client moving 
mail from new/ to cur/ while KMail is not running. It seems to me 
that in this case the KMail index for that folder will be out of sync 
with the cur/ directory. And hence currently in the KMail code the 
index would be completely recreated, losing status information and 
causing new sernums for all messages in that folder to be assigned, 
hence invalidating all search folders.

Wouldn't traditional integration with other maildir clients require 
being able to handle this situation without these detrimental 
affects?

> > 2) I'm concerned that this approach could be fragile. A client
> > might die after deleting/inserting a Maildir message but before
> > updating the index, or vice versa. Maybe this can be handled
> > robustly so this is just a concern not a concrete criticism. But
> > clearly this method does rely on mbox style locking rather than
> > maildir style elimination of the need for locking.
>
> Locking is only necessary for the index files, not the mails
> themselves. If a MUA dies while adding a message to the maildir, it
> will stay incomplete in tmp/ until cleaned up. tmp/ is not
> monitored, so nobody cares.
>
> If a MUA dies moving a message from tmp/ to new/ or cur/, either
> the move happens completely or not at all. In the first case,
> another KMail instance can try to add the message after waiting a
> certain amount of time for the index to be updated. If the update
> doesn't arrive, the addition is performed as if it was delivered.
> For this to work, the sernum has to be added as a x-KMail header to
> the mail.

How will you set the timeout for the lock? Some operations in KMail 
take a long time so perhaps there's a danger of mistakenly thinking 
the lock is stale when really a KMail client is busy, right?

> > 3) If this approach works then it does address the criticism of
> > having multiple clients modify index files but there's still the
> > memory costs associated with this approach. Each client will need
> > there own kmmsgdict, IIRC each entry in the kmmsgdict maps a
> > sernum -> (folder , index ) which is an int -> (int, int) mapping
> > or 12 bytes. I have >500K messages currently, so that's >6MB per
> > client instance.
>
> You typically won't run that many KMail instances that these few MB
> make any difference. Once a lib is split off of KMail for other
> apps to use, I don't think most other apps will have a great need
> for the dictionary.
>
> > And then there's the full text index. A full text index is by
> > it's nature a large object as it's space/time tradeoff. For me
> > it's > 24MB.
>
> See above. Can be loaded on demand and most clients won't want to
> use it anyway.

Half a year ago I spent some time trying to delay the loading of the 
dictionary to make startup of KMail faster.

But personally I couldn't manage it. We use serial numbers quite a lot 
now as they provide a safe way of keeping a reference to a message, 
even at startup the dictionary is used (in KMReaderWin) when the 
initial message is displayed.

It seems to me that any client that wants to display messages will use 
the dict. I would have thought that this is a pretty common case, 
would you not agree?

Unfortunately the case of the full text index seems to be an even more 
difficult one. As in order to keep the full text index up to date it 
must be running whenever mail is discovered, altered, copied, or 
moved. Wouldn't this be a pretty typical kind of thing for clients 
todo?

I wonder if a client can't efficiently handle either of these cases 
what useful task could a client do efficiently?

> > Besides folder files there are also other files than need to be
> > handled. The config file is a problem if it is desirable to have
> > each client keep their config dialogs and general configuration
> > info in sync.
>
> I'm cool with having that break. If a user is dumb enough to keep
> the config dialogs of two KMail's open at the same time, then he
> shouldn't complain if the second-to-be-closed one overwrites the
> settings of the first-to-be-closed one. I don't see a need to make
> that work.

Ok. But even if that was acceptable wouldn't there be a problem on say 
session exit with all the clients updating the config file at exit? 

I mean say a new account was created in one client, couldn't that new 
account be lost if another client closed after the client where the 
new account was created?

> > Another set of files is the list of mails that are kept on the
> > server for each account.
>
> These are different. Locking is the std answer I'd give here ;-)
> You have to have locking for accounts anyway. Two instances
> concurrently checking mail will else break.

Right, but isn't this extra work something that only has to be done in 
the CIA approach? Wouldn't this extra work be something unnecessary 
with the client/server approach?

Isn't this also true for the problem of preventing duplicate serial 
numbers being given out? And also for the problem of overwriting 
config entries given above? Oh, and also for the problem of 
determining when index lock files are stale. Isn't it true that all 
of these problems only exist for the client only approach?

If it's true that in the client/sever approach these kinds of problems 
don't exist couldn't you consider that as an argument for the 
client/server approach? I mean given that there don't seem to be any 
outstanding arguments against the client/server approach, are there?

And wouldn't it also be nice if the composer wouldn't freeze up when 
fsyncing new messages to disk when checking for mail? If the 
client/server model had that advantage wouldn't that be a 
characterisitic that was favorable about the client/server model?

I'm also still a bit concerned about the folder index mmaps. Or rather 
about giving up the internal representation of index file entries. I 
need to double check but I'm concerned that new entries added to a 
folder after it is opened aren't covered by the mmap instead those 
entries are only covered by the in memory representation of the index 
entry.

Don.
_______________________________________________
KMail Developers mailing list
kmail@mail.kde.org
http://mail.kde.org/mailman/listinfo/kmail
[prev in list] [next in list] [prev in thread] [next in thread]