From kmail-devel Fri Jul 25 06:30:53 2003 From: Don Sanders Date: Fri, 25 Jul 2003 06:30:53 +0000 To: kmail-devel Subject: Re: CIA proposal (was: ClientInterface) X-MARC-Message: https://marc.info/?l=kmail-devel&m=105911380428235 On Friday 25 July 2003 00:15, Marc Mutz wrote: > Hi Don! > > I'm in the process of going home, so I'll address some points on > shortly. I apologize. > > On Wednesday 23 July 2003 06:58, Don Sanders wrote: > > I do wonder if multiple processes ro mmap the same file whether > > the memory used to back the mmap is shared by all those > > processes. > > It is (if you use MAP_SHARED). That's nice. > > My concerns: > > > > 1) This doesn't actually address the issue of external mail > > clients modifying ~/Mail does it? I mean not even for MailDir as > > this approach requires KMail to be running to detect changes in > > ~/Mail. > > New mail is delivered into the new/ subdir for maildir. The first > instance of KMail to close the folder or have a user click on the > new message will move the mails from new/ to cur/, updating the > index along the way. > > Of course, MUAs wishing to modify ~/Mail need to conform to CIA/4, > but maildir-compliant MDAs work without modifications. Ok, but what about the case of another (non-KMail) mail client moving mail from new/ to cur/ while KMail is not running. It seems to me that in this case the KMail index for that folder will be out of sync with the cur/ directory. And hence currently in the KMail code the index would be completely recreated, losing status information and causing new sernums for all messages in that folder to be assigned, hence invalidating all search folders. Wouldn't traditional integration with other maildir clients require being able to handle this situation without these detrimental affects? > > 2) I'm concerned that this approach could be fragile. A client > > might die after deleting/inserting a Maildir message but before > > updating the index, or vice versa. Maybe this can be handled > > robustly so this is just a concern not a concrete criticism. But > > clearly this method does rely on mbox style locking rather than > > maildir style elimination of the need for locking. > > Locking is only necessary for the index files, not the mails > themselves. If a MUA dies while adding a message to the maildir, it > will stay incomplete in tmp/ until cleaned up. tmp/ is not > monitored, so nobody cares. > > If a MUA dies moving a message from tmp/ to new/ or cur/, either > the move happens completely or not at all. In the first case, > another KMail instance can try to add the message after waiting a > certain amount of time for the index to be updated. If the update > doesn't arrive, the addition is performed as if it was delivered. > For this to work, the sernum has to be added as a x-KMail header to > the mail. How will you set the timeout for the lock? Some operations in KMail take a long time so perhaps there's a danger of mistakenly thinking the lock is stale when really a KMail client is busy, right? > > 3) If this approach works then it does address the criticism of > > having multiple clients modify index files but there's still the > > memory costs associated with this approach. Each client will need > > there own kmmsgdict, IIRC each entry in the kmmsgdict maps a > > sernum -> (folder , index ) which is an int -> (int, int) mapping > > or 12 bytes. I have >500K messages currently, so that's >6MB per > > client instance. > > You typically won't run that many KMail instances that these few MB > make any difference. Once a lib is split off of KMail for other > apps to use, I don't think most other apps will have a great need > for the dictionary. > > > And then there's the full text index. A full text index is by > > it's nature a large object as it's space/time tradeoff. For me > > it's > 24MB. > > See above. Can be loaded on demand and most clients won't want to > use it anyway. Half a year ago I spent some time trying to delay the loading of the dictionary to make startup of KMail faster. But personally I couldn't manage it. We use serial numbers quite a lot now as they provide a safe way of keeping a reference to a message, even at startup the dictionary is used (in KMReaderWin) when the initial message is displayed. It seems to me that any client that wants to display messages will use the dict. I would have thought that this is a pretty common case, would you not agree? Unfortunately the case of the full text index seems to be an even more difficult one. As in order to keep the full text index up to date it must be running whenever mail is discovered, altered, copied, or moved. Wouldn't this be a pretty typical kind of thing for clients todo? I wonder if a client can't efficiently handle either of these cases what useful task could a client do efficiently? > > Besides folder files there are also other files than need to be > > handled. The config file is a problem if it is desirable to have > > each client keep their config dialogs and general configuration > > info in sync. > > I'm cool with having that break. If a user is dumb enough to keep > the config dialogs of two KMail's open at the same time, then he > shouldn't complain if the second-to-be-closed one overwrites the > settings of the first-to-be-closed one. I don't see a need to make > that work. Ok. But even if that was acceptable wouldn't there be a problem on say session exit with all the clients updating the config file at exit? I mean say a new account was created in one client, couldn't that new account be lost if another client closed after the client where the new account was created? > > Another set of files is the list of mails that are kept on the > > server for each account. > > These are different. Locking is the std answer I'd give here ;-) > You have to have locking for accounts anyway. Two instances > concurrently checking mail will else break. Right, but isn't this extra work something that only has to be done in the CIA approach? Wouldn't this extra work be something unnecessary with the client/server approach? Isn't this also true for the problem of preventing duplicate serial numbers being given out? And also for the problem of overwriting config entries given above? Oh, and also for the problem of determining when index lock files are stale. Isn't it true that all of these problems only exist for the client only approach? If it's true that in the client/sever approach these kinds of problems don't exist couldn't you consider that as an argument for the client/server approach? I mean given that there don't seem to be any outstanding arguments against the client/server approach, are there? And wouldn't it also be nice if the composer wouldn't freeze up when fsyncing new messages to disk when checking for mail? If the client/server model had that advantage wouldn't that be a characterisitic that was favorable about the client/server model? I'm also still a bit concerned about the folder index mmaps. Or rather about giving up the internal representation of index file entries. I need to double check but I'm concerned that new entries added to a folder after it is opened aren't covered by the mmap instead those entries are only covered by the in memory representation of the index entry. Don. _______________________________________________ KMail Developers mailing list kmail@mail.kde.org http://mail.kde.org/mailman/listinfo/kmail