[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kmail-devel
Subject:    [PATCH] speed up status setting
From:       Till Adam <till () adam-lilienthal ! de>
Date:       2003-12-21 9:00:17
[Download RAW message or body]

[Attachment #2 (multipart/signed)]

[Attachment #4 (multipart/mixed)]


Hey folks, 

the attached patch speeds up status setting for local and especially imap 
folders by:

o accumulating all jobs for each folder in a separate list (could be several 
   when marking messages in search folders, for example) and send that 
   list off to each folder when done instead of processing each message
   individually in KMSetStatusCommand.

o in the imap folder case, taking the list of uids handed over by the 
   KMSetStatusCommand and chunking it into groups that will be processed 
   as one list of uid sets depending on what the result status flags of the
   mails is. So if there are some mails in the folder that are important and
   some that are not, there will be two groups in the case of for example 
   "Mark all as read", one with result flags "\SEEN \FLAG" and one with result 
   flags just "\SEEN". Build sets of uids for each of those, a la 
   ";UID=3:6,16:32" and send of each of those sets to the server for 
   processing

With this scheme the number of imap transactions goes from 20000 to a handfull  
when marking all mails in a for a folder with 20000 mails in it as read. 

Needless to say that speeds things up several orders of magnitude. The 
limiting factor for both local and imap status setting is now disk i/o from 
index accessing.

Please review.

Till

["KMail-statusSettingSpeedup.diff" (text/x-diff)]

Index: imapjob.cpp
===================================================================
RCS file: /home/kde/kdepim/kmail/imapjob.cpp,v
retrieving revision 1.33
diff -u -3 -p -r1.33 imapjob.cpp
--- imapjob.cpp	18 Dec 2003 21:10:39 -0000	1.33
+++ imapjob.cpp	21 Dec 2003 12:55:07 -0000
@@ -97,8 +97,8 @@ void ImapJob::init( JobType jt, QString 
   {
     // transfers the complete message to the server
     KURL url = account->getUrl();
-    url.setPath( folder->imapPath() + ";SECTION="
-      + QString::fromLatin1( KMFolderImap::statusToFlags( msg->status() ) ) );
+    QString flags = KMFolderImap::statusToFlags( msg->status() ); 
+    url.setPath( folder->imapPath() + ";SECTION=" + flags );
     ImapAccountBase::jobData jd;
     jd.parent = 0; jd.offset = 0;
     jd.total = 1; jd.done = 0;
Index: kmcommands.cpp
===================================================================
RCS file: /home/kde/kdepim/kmail/kmcommands.cpp,v
retrieving revision 1.94
diff -u -3 -p -r1.94 kmcommands.cpp
--- kmcommands.cpp	16 Dec 2003 19:19:54 -0000	1.94
+++ kmcommands.cpp	21 Dec 2003 12:55:10 -0000
@@ -1098,6 +1098,7 @@ void KMSetStatusCommand::execute()
         parentStatus = false;
     }
   }
+  QMap< KMFolder*, QValueList<int> > folderMap;
   for ( it = mSerNums.begin(); it != mSerNums.end(); ++it ) {
     kmkernel->msgDict()->getLocation( *it, &folder, &idx );
     if (folder) {
@@ -1114,9 +1115,17 @@ void KMSetStatusCommand::execute()
             continue;
         }
       }
-      folder->setStatus( idx, mStatus, mToggle );
+      /* Collect the ids for each folder in a separate list and
+         send them off in one go at the end. */
+      folderMap[folder].append(idx);
     }
   }
+  QMapIterator< KMFolder*, QValueList<int> > it2 = folderMap.begin();
+  while ( it2 != folderMap.end() ) {
+     KMFolder *f = it2.key();
+     f->setStatus( (*it2), mStatus, mToggle );
+     ++it2;
+  }
 }
 
 
Index: kmfolderimap.cpp
===================================================================
RCS file: /home/kde/kdepim/kmail/kmfolderimap.cpp,v
retrieving revision 1.156
diff -u -3 -p -r1.156 kmfolderimap.cpp
--- kmfolderimap.cpp	20 Dec 2003 11:09:00 -0000	1.156
+++ kmfolderimap.cpp	21 Dec 2003 12:55:14 -0000
@@ -920,19 +920,18 @@ void KMFolderImap::flagsToStatus(KMMsgBa
 
 
 //-----------------------------------------------------------------------------
-QCString KMFolderImap::statusToFlags(KMMsgStatus status)
+QString KMFolderImap::statusToFlags(KMMsgStatus status)
 {
-  QCString flags = "";
-  if (status & KMMsgStatusNew || status & KMMsgStatusUnread) 
-    return flags;
+  QString flags;
   if (status & KMMsgStatusDeleted) 
     flags = "\\DELETED";
   else {
-    flags = "\\SEEN";
+    if (status & KMMsgStatusOld || status & KMMsgStatusRead) 
+      flags = "\\SEEN ";
     if (status & KMMsgStatusReplied) 
-      flags += " \\ANSWERED";
+      flags += "\\ANSWERED ";
     if (status & KMMsgStatusFlag) 
-      flags += " \\FLAGGED";
+      flags += "\\FLAGGED";
   }
   return flags;
 }
@@ -1237,19 +1236,38 @@ void KMFolderImap::setStatus(QValueList<
 {
   KMFolder::setStatus(ids, status, toggle);
   if (mReadOnly) return;
-  
+
   /* The status has been already set in the local index. Update the flags on
-   * the server. Needs to be done for each message individually. */
-  for ( QValueList<int>::Iterator it = ids.begin(); it != ids.end(); ++it )
-  {
+   * the server. To avoid doing that for each message individually, group them
+   * by the status string they will be assigned and make sets for each of those
+   * groups of mails. This is necessary because the imap kio_slave status job
+   * does not append flags but overwrites them. Example:
+   * 
+   * 2 important mails and 3 unimportant mail, all unread. Mark all as read calls
+   * this method with a list of uids. The 2 important mails need to get the string
+   * \SEEN \FLAGGED while the others need to get just \SEEN. Build sets for each
+   * of those and sort them, so the server can handle them efficiently. */
+  QMap< QString, QStringList > groups;
+  for ( QValueList<int>::Iterator it = ids.begin(); it != ids.end(); ++it ) {
     KMMessage *msg = 0;
     bool unget = !isMessage(*it);
     msg = getMsg(*it);
     if (!msg) continue;
-    QCString flags = statusToFlags(msg->status());
-    setImapStatus(imapPath() + ";UID=" + msg->headerField("X-UID"), flags);
+    QString flags = statusToFlags(msg->status());
+    // Collect uids for each type of flags.
+    groups[flags].append(msg->headerField("X-UID"));
     if (unget) unGetMsg(*it);
   }
+  QMapIterator< QString, QStringList > dit;
+  for ( dit = groups.begin(); dit != groups.end(); ++dit ) {
+     QCString flags = dit.key().latin1();
+     QStringList sets = makeSets( (*dit), true );
+     // Send off a status setting job for each set.
+     for (  QStringList::Iterator slit = sets.begin(); slit != sets.end(); ++slit ) {
+       QString imappath = imapPath() + ";UID=" + ( *slit );
+       setImapStatus(imappath, flags);
+     }
+  }
   mAccount->displayProgress();
 }
 
Index: kmfolderimap.h
===================================================================
RCS file: /home/kde/kdepim/kmail/kmfolderimap.h,v
retrieving revision 1.54
diff -u -3 -p -r1.54 kmfolderimap.h
--- kmfolderimap.h	22 Nov 2003 18:09:11 -0000	1.54
+++ kmfolderimap.h	21 Dec 2003 12:55:15 -0000
@@ -216,7 +216,7 @@ public:
   /**
    * Convert message status to a list of IMAP flags
    */
-  static QCString statusToFlags(KMMsgStatus status);
+  static QString statusToFlags(KMMsgStatus status);
 
   /**
    * Return the filename of the folder (reimplemented from KFolder)

[Attachment #8 (application/pgp-signature)]

_______________________________________________
KMail developers mailing list
KMail-devel@kde.org
https://mail.kde.org/mailman/listinfo/kmail-devel


[prev in list] [next in list] [prev in thread] [next in thread]