Hi!

> It's not watching the files that is the main problem. The problem is 
> retaining the integrity of objects that point into files which can be 
> changed by other processes.

I was asked to provide a concept for a server-less solution to the CIA 
(Concurrent Index Access) problem. Here is it. It's based on the idea 
from LinuxTag that's described here as Step 3. It solves the memory 
load problems with index files by having them shared between KMail 
instances running on the same machine.

Step 1: Require Maildir
Rationale:
  - Changes in maildir folders can be efficiently monitored with
    KDirWatch (which can use DNotify/FAM). In particular, changes
    are monitored per-file. This is not the case for mbox.
  - In particular, changes to stati result in renaming the file.

Step 2: Observe that the Index file format allows most changes (esp. all 
status updates) to be performed in-place.

Step 3: Don't create an in-memory representation of index entries 
("KMMsgInfo::kd") anymore. Instead, let every entry just be a pointer 
to the beginning of it's mmaped region in the index file. For this to 
be efficient, use Don's idea of storing those offsets in a file of 
their own. Every access to information stored in the index accesses the 
index file data directly.
Rationale:
  - This allows most operations on exisiting messages to be
    transparently shared across all KMail instances; and in an extremely
    efficient way (memory-, cpu- and network-load-wise). This holds esp.
    for the common operations of changing the various stati of messages.

Step 4: Define a protocol by which a given Kmail instance can obtain 
write access to a given folder/index file.

Sketch of the CIA/4 protocol:

KMail instance A wants to obtain write access to folder F from B:

1. A tries to create folder-lockfile L_F
   Success: Write own DCOP id into the lock file.
            A has now (exclusive) write access to F.
2. A tries to read a DCOP ID of B from L_F
   Success: A sends B the message
            "Please release folder F"
            Wait a random amount of time, then goto 1.
   Failure: Wait a random amount of time, then goto 1.
   Repeated failure: Tell user.

Observe how B is not required to close the folder or to unmmap the index 
file. It marks the folder read-only internally (or remaps the index to 
be ro).

An instance holding the write lock is required to

a. Release it as soon as practicable
b. Release it as soon as possible if the above DCOP message arrives
c. never alter existing index entries' length

If a change in entry length is necessary, mark the old one as deleted, 
rename the maildir file and add a new entry at the end. Other instances 
will see a delete and add through the dirwatching and can react 
accordingly.

CIA/4 can be extended with a DCOP broadcast call to all KMail instances 
to close F, so that A can perform (index file) compaction after all 
instances replied with a "folder F closed" message.

Marc

-- 
Military justice is to justice what military music is to music.
                                                  -- Groucho Marx
_______________________________________________
KMail Developers mailing list
kmail@mail.kde.org
http://mail.kde.org/mailman/listinfo/kmail