[prev in list] [next in list] [prev in thread] [next in thread]
List: kmail-devel
Subject: Patch: For bugs #50997, #59685 and partially #41514
From: Don Sanders <sanders () trolltech ! com>
Date: 2005-07-26 4:39:44
Message-ID: 200507261439.45325.sanders () trolltech ! com
[Download RAW message or body]
Hare Krsna,
I've attached a patch that conditionally enables the action scheduler
for filtering of incoming mails for online imap accounts and for
manual filtering.
To try out the action scheduler apply the patch and add an
action-scheduler=true entry to the [General] section of your kmailrc
file.
I'm sending this patch because there was quite a lot interest
expressed on IRC (by Till, Cornelius) for me to do so, and to commit
the patch before the 3.5 freeze. The problems I've been working on
recently are that sometimes (once every few minutes or hours, it
varies unpredictably) a KMMoveCommand will fail. Similarly once every
few days or so an imap job will fail. I guess it will never be
possible to completely eliminate these problems. So I've been working
on error detection and recovery.
To help me do this I've added a private dcop method to help me monitor
the health of the action scheduler, you can try it by:
dcop kmail KMailIface "debugScheduler()"
At this current point in time I'm unaware of any outstanding bugs.
However I haven't had time to hook up the action scheduler for local,
pop, and dimap accounts, and for sent mail filtering. Also I would
like to use the action scheduler for move, delete, and copy
KMCommands to make them less blocking. Finally I haven't actually
checked that the action scheduler makes spam filtering less blocking
(but I expect it will, I'm using the work Till did in this area).
The action scheduler works by keeping a queue of messages and
filtering them one by one. It does this by sequentially for each
message (sernum) in the queue fetch the message if it is not
complete, then apply filters sequentially, by applying filter actions
sequentially, it uses the new async filter action API to do this. So
actually, conceptually, what the action scheduler does is pretty
simple. But it is also a bit slow. (Oh, there's also a hardcoded 60
second timeout which should at least have a non gui config option).
For more info please see my recent comments on bug #50997. I guess if
there are no objections then I'll commit (this should basically be a
no-op as it's disabled by default).
Don Sanders http://donsanders.org
["as6.diff" (text/x-diff)]
Index: kmacctimap.h
===================================================================
--- kmacctimap.h (revision 438759)
+++ kmacctimap.h (working copy)
@@ -29,6 +29,7 @@
class KMFolderTreeItem;
namespace KMail {
class ImapJob;
+ class ActionScheduler;
}
namespace KIO {
class Job;
@@ -97,6 +98,7 @@
/**
* updates the new-mail-check folderlist
*/
+ void slotFiltered(Q_UINT32 serNum);
void slotUpdateFolderList();
protected:
@@ -144,6 +146,7 @@
int mCountRemainChecks;
QValueList<Q_UINT32> mFilterSerNums;
QDict<int> mFilterSerNumsToSave;
+ KMail::ActionScheduler *mScheduler;
};
#endif /*KMAcctImap_h*/
Index: kmheaders.cpp
===================================================================
--- kmheaders.cpp (revision 438759)
+++ kmheaders.cpp (working copy)
@@ -1314,51 +1314,58 @@
//-----------------------------------------------------------------------------
void KMHeaders::applyFiltersOnMsg()
{
-#if 0 // uses action scheduler
- KMFilterMgr::FilterSet set = KMFilterMgr::All;
- QPtrList<KMFilter> filters;
- filters = *( kmkernel->filterMgr() );
- ActionScheduler *scheduler = new ActionScheduler( set, filters, this );
- scheduler->setAutoDestruct( true );
+ static bool useAs = false;
+ static bool useAsChecked = false;
+ if (!useAsChecked) {
+ useAsChecked = true;
+ KConfig* config = KMKernel::config();
+ KConfigGroupSaver saver(config, "General");
+ useAs = config->readBoolEntry("action-scheduler", false);
+ }
+ if (useAs) { // uses action scheduler
+ KMFilterMgr::FilterSet set = KMFilterMgr::Explicit;
+ QValueList<KMFilter*> filters = kmkernel->filterMgr()->filters();
+ ActionScheduler *scheduler = new ActionScheduler( set, filters, this );
+ scheduler->setAutoDestruct( true );
- int contentX, contentY;
- HeaderItem *nextItem = prepareMove( &contentX, &contentY );
- QPtrList<KMMsgBase> msgList = *selectedMsgs(true);
- finalizeMove( nextItem, contentX, contentY );
+ int contentX, contentY;
+ HeaderItem *nextItem = prepareMove( &contentX, &contentY );
+ QPtrList<KMMsgBase> msgList = *selectedMsgs(true);
+ finalizeMove( nextItem, contentX, contentY );
- for (KMMsgBase *msg = msgList.first(); msg; msg = msgList.next())
- scheduler->execFilters( msg );
-#else
- int contentX, contentY;
- HeaderItem *nextItem = prepareMove( &contentX, &contentY );
+ for (KMMsgBase *msg = msgList.first(); msg; msg = msgList.next())
+ scheduler->execFilters( msg );
+ } else {
+ int contentX, contentY;
+ HeaderItem *nextItem = prepareMove( &contentX, &contentY );
- KMMessageList* msgList = selectedMsgs();
- if (msgList->isEmpty())
- return;
- finalizeMove( nextItem, contentX, contentY );
+ KMMessageList* msgList = selectedMsgs();
+ if (msgList->isEmpty())
+ return;
+ finalizeMove( nextItem, contentX, contentY );
- CREATE_TIMER(filter);
- START_TIMER(filter);
+ CREATE_TIMER(filter);
+ START_TIMER(filter);
- for (KMMsgBase* msgBase=msgList->first(); msgBase; msgBase=msgList->next()) {
- int idx = msgBase->parent()->find(msgBase);
- assert(idx != -1);
- KMMessage * msg = msgBase->parent()->getMsg(idx);
- if (msg->transferInProgress()) continue;
- msg->setTransferInProgress(true);
- if ( !msg->isComplete() )
- {
- FolderJob *job = mFolder->createJob(msg);
- connect(job, SIGNAL(messageRetrieved(KMMessage*)),
- SLOT(slotFilterMsg(KMMessage*)));
- job->start();
- } else {
- if (slotFilterMsg(msg) == 2) break;
+ for (KMMsgBase* msgBase=msgList->first(); msgBase; msgBase=msgList->next()) {
+ int idx = msgBase->parent()->find(msgBase);
+ assert(idx != -1);
+ KMMessage * msg = msgBase->parent()->getMsg(idx);
+ if (msg->transferInProgress()) continue;
+ msg->setTransferInProgress(true);
+ if ( !msg->isComplete() )
+ {
+ FolderJob *job = mFolder->createJob(msg);
+ connect(job, SIGNAL(messageRetrieved(KMMessage*)),
+ SLOT(slotFilterMsg(KMMessage*)));
+ job->start();
+ } else {
+ if (slotFilterMsg(msg) == 2) break;
+ }
}
+ END_TIMER(filter);
+ SHOW_TIMER(filter);
}
- END_TIMER(filter);
- SHOW_TIMER(filter);
-#endif
}
Index: kmkernel.cpp
===================================================================
--- kmkernel.cpp (revision 438759)
+++ kmkernel.cpp (working copy)
@@ -66,6 +66,7 @@
#include <kio/netaccess.h>
#include <kwallet.h>
using KWallet::Wallet;
+#include "actionscheduler.h"
#include <qutf7codec.h>
#include <qvbox.h>
@@ -1083,6 +1084,43 @@
return result;
}
+QString KMKernel::debugScheduler()
+{
+ QString res = KMail::ActionScheduler::debug();
+ return res;
+}
+
+QString KMKernel::debugSernum( Q_UINT32 serialNumber )
+{
+ QString res;
+ if (serialNumber != 0) {
+ int idx = -1;
+ KMFolder *folder = 0;
+ KMMsgBase *msg = 0;
+ KMMsgDict::instance()->getLocation( serialNumber, &folder, &idx );
+ // It's possible that the message has been deleted or moved into a
+ // different folder
+ if (folder && (idx != -1)) {
+ // everything is ok
+ folder->open();
+ msg = folder->getMsgBase( idx );
+ if (msg) {
+ res.append( QString( " subject %s,\n sender %s,\n date %s.\n" )
+ .arg( msg->subject() )
+ .arg( msg->fromStrip() )
+ .arg( msg->dateStr() ) );
+ } else {
+ res.append( QString( "Invalid serial number." ) );
+ }
+ folder->close();
+ } else {
+ res.append( QString( "Invalid serial number." ) );
+ }
+ }
+ return res;
+}
+
+
void KMKernel::pauseBackgroundJobs()
{
mBackgroundTasksTimer->stop();
Index: kmfilteraction.cpp
===================================================================
--- kmfilteraction.cpp (revision 438759)
+++ kmfilteraction.cpp (working copy)
@@ -277,7 +277,16 @@
{
FolderRequester *req = new FolderRequester( parent,
kmkernel->getKMMainWidget()->folderTree() );
- req->setShowImapFolders( false );
+ static bool useAs = false;
+ static bool useAsChecked = false;
+ if (!useAsChecked) {
+ useAsChecked = true;
+ KConfig* config = KMKernel::config();
+ KConfigGroupSaver saver(config, "General");
+ useAs = config->readBoolEntry("action-scheduler", false);
+ }
+ if (!useAs)
+ req->setShowImapFolders( false );
setParamWidgetValue( req );
return req;
}
@@ -313,6 +322,8 @@
mFolder = kmkernel->folderMgr()->findIdString( argsStr );
if (!mFolder)
mFolder = kmkernel->dimapFolderMgr()->findIdString( argsStr );
+ if (!mFolder)
+ mFolder = kmkernel->imapFolderMgr()->findIdString( argsStr );
if (mFolder)
mFolderName = QString::null;
else
Index: kmkernel.h
===================================================================
--- kmkernel.h (revision 438759)
+++ kmkernel.h (working copy)
@@ -176,6 +176,8 @@
int timeOfLastMessageCountChange() const;
virtual bool showMail( Q_UINT32 serialNumber, QString messageId );
virtual QString getFrom( Q_UINT32 serialNumber );
+ virtual QString debugScheduler();
+ virtual QString debugSernum( Q_UINT32 serialNumber );
int viewMessage( const KURL & messageFile );
virtual void pauseBackgroundJobs();
Index: actionscheduler.h
===================================================================
--- actionscheduler.h (revision 438759)
+++ actionscheduler.h (working copy)
@@ -54,7 +54,7 @@
enum ReturnCode { ResultOk, ResultError, ResultCriticalError };
ActionScheduler(KMFilterMgr::FilterSet set,
- QPtrList<KMFilter> filters,
+ QValueList<KMFilter*> filters,
KMHeaders *headers = 0,
KMFolder *srcFolder = 0);
~ActionScheduler();
@@ -75,7 +75,7 @@
/** Set a list of filters to work with
The current list will not be updated until the queue
of messages left to process is empty */
- void setFilterList( QPtrList<KMFilter> filters );
+ void setFilterList( QValueList<KMFilter*> filters );
/* Set the id of the account associated with this scheduler */
void setAccountId( uint id ) { mAccountId = id; mAccount = true; }
@@ -88,10 +88,12 @@
void execFilters(const QPtrList<KMMsgBase> msgList);
void execFilters(KMMsgBase* msgBase);
void execFilters(Q_UINT32 serNum);
+ static QString debug();
signals:
/** Emitted when filtering is completed */
void result(ReturnCode);
+ void filtered(Q_UINT32);
public slots:
/** Called back by asynchronous actions when they have completed */
@@ -118,8 +120,10 @@
void moveMessage();
void moveMessageFinished( KMCommand *command );
void timeOut();
+ void fetchTimeOut();
private:
+ static QValueList<ActionScheduler*> *schedulerList; // for debugging
static KMFolderMgr *tempFolderMgr;
static int refCount, count;
QValueListIterator<Q_UINT32> mMessageIt;
@@ -144,9 +148,10 @@
ReturnCode mResult;
QTimer *finishTimer, *fetchMessageTimer, *tempCloseFoldersTimer;
QTimer *processMessageTimer, *filterMessageTimer;
- QTimer *timeOutTimer;
- QTime timeOutTime;
+ QTimer *timeOutTimer, *fetchTimeOutTimer;
+ QTime timeOutTime, fetchTimeOutTime;
KMCommand *lastCommand;
+ FolderJob *lastJob;
};
}
Index: kmailIface.h
===================================================================
--- kmailIface.h (revision 438759)
+++ kmailIface.h (working copy)
@@ -175,6 +175,8 @@
* DCOP-enabled for use in kaddressbook drop
*/
virtual QString getFrom( Q_UINT32 serialNumber ) = 0;
+ virtual QString debugScheduler() = 0;
+ virtual QString debugSernum( Q_UINT32 serialNumber ) = 0;
/** Does essentially the same as dcopAddMessage except that it doesn't reject
duplicate messages.
Index: kmacctimap.cpp
===================================================================
--- kmacctimap.cpp (revision 438759)
+++ kmacctimap.cpp (working copy)
@@ -38,6 +38,8 @@
#include "kmfiltermgr.h"
#include "folderstorage.h"
#include "imapjob.h"
+#include "actionscheduler.h"
+using KMail::ActionScheduler;
using KMail::ImapJob;
using KMail::ImapAccountBase;
#include "progressmanager.h"
@@ -55,6 +57,7 @@
mCountRemainChecks( 0 )
{
mFolder = 0;
+ mScheduler = 0;
mNoopTimer.start( 60000 ); // // send a noop every minute
mOpenFolders.setAutoDelete(true);
connect(kmkernel->imapFolderMgr(), SIGNAL(changed()),
@@ -395,6 +398,25 @@
QValueListIterator<Q_UINT32> filterIt = mFilterSerNums.begin();
QValueList<Q_UINT32> inTransit;
+ static bool useAs = false;
+ static bool useAsChecked = false;
+ if (!useAsChecked) {
+ useAsChecked = true;
+ KConfig* config = KMKernel::config();
+ KConfigGroupSaver saver(config, "General");
+ useAs = config->readBoolEntry("action-scheduler", false);
+ }
+ if (useAs) {
+ KMFilterMgr::FilterSet set = KMFilterMgr::Inbound;
+ QValueList<KMFilter*> filters = kmkernel->filterMgr()->filters();
+ if (!mScheduler) {
+ mScheduler = new KMail::ActionScheduler( set, filters );
+ connect( mScheduler, SIGNAL(filtered(Q_UINT32)), this, SLOT(slotFiltered(Q_UINT32)) );
+ } else {
+ mScheduler->setFilterList( filters );
+ }
+ }
+
while (filterIt != mFilterSerNums.end()) {
int idx = -1;
KMFolder *folder = 0;
@@ -425,21 +447,25 @@
++filterIt;
continue;
}
-
- if (msg->transferInProgress()) {
- inTransit.append( *filterIt );
- ++filterIt;
- continue;
- }
- msg->setTransferInProgress(true);
- if ( !msg->isComplete() ) {
- FolderJob *job = folder->createJob(msg);
- connect(job, SIGNAL(messageRetrieved(KMMessage*)),
- SLOT(slotFilterMsg(KMMessage*)));
- job->start();
+
+ if (useAs) {
+ mScheduler->execFilters( msg );
} else {
- mFilterSerNumsToSave.remove( QString( "%1" ).arg( *filterIt ) );
- if (slotFilterMsg(msg) == 2) break;
+ if (msg->transferInProgress()) {
+ inTransit.append( *filterIt );
+ ++filterIt;
+ continue;
+ }
+ msg->setTransferInProgress(true);
+ if ( !msg->isComplete() ) {
+ FolderJob *job = folder->createJob(msg);
+ connect(job, SIGNAL(messageRetrieved(KMMessage*)),
+ SLOT(slotFilterMsg(KMMessage*)));
+ job->start();
+ } else {
+ mFilterSerNumsToSave.remove( QString( "%1" ).arg( *filterIt ) );
+ if (slotFilterMsg(msg) == 2) break;
+ }
}
}
++filterIt;
@@ -460,6 +486,12 @@
}
//-----------------------------------------------------------------------------
+void KMAcctImap::slotFiltered(Q_UINT32 serNum)
+{
+ mFilterSerNumsToSave.remove( QString( "%1" ).arg( serNum ) );
+}
+
+//-----------------------------------------------------------------------------
void KMAcctImap::slotUpdateFolderList()
{
if ( !mFolder || !mFolder->folder() || !mFolder->folder()->child() )
Index: actionscheduler.cpp
===================================================================
--- actionscheduler.cpp (revision 438759)
+++ actionscheduler.cpp (working copy)
@@ -41,6 +41,8 @@
#include "kmmsgdict.h"
#include "kmcommands.h"
#include "kmheaders.h"
+#include "accountmanager.h"
+using KMail::AccountManager;
#include <qtimer.h>
#include <kconfig.h>
@@ -53,9 +55,10 @@
KMFolderMgr* ActionScheduler::tempFolderMgr = 0;
int ActionScheduler::refCount = 0;
int ActionScheduler::count = 0;
+QValueList<ActionScheduler*> *ActionScheduler::schedulerList = 0;
ActionScheduler::ActionScheduler(KMFilterMgr::FilterSet set,
- QPtrList<KMFilter> filters,
+ QValueList<KMFilter*> filters,
KMHeaders *headers,
KMFolder *srcFolder)
:mSet( set ), mHeaders( headers )
@@ -72,7 +75,8 @@
mAlwaysMatch = false;
mAccountId = 0;
mAccount = false;
- KMFilter *filter;
+ lastCommand = 0;
+ lastJob = 0;
finishTimer = new QTimer( this );
connect( finishTimer, SIGNAL(timeout()), this, SLOT(finish()));
fetchMessageTimer = new QTimer( this );
@@ -85,9 +89,12 @@
connect( filterMessageTimer, SIGNAL(timeout()), this, SLOT(filterMessage()));
timeOutTimer = new QTimer( this );
connect( timeOutTimer, SIGNAL(timeout()), this, SLOT(timeOut()));
+ fetchTimeOutTimer = new QTimer( this );
+ connect( fetchTimeOutTimer, SIGNAL(timeout()), this, SLOT(fetchTimeOut()));
- for (filter = filters.first(); filter; filter = filters.next())
- mFilters.append( *filter );
+ QValueList<KMFilter*>::Iterator it = filters.begin();
+ for (; it != filters.end(); ++it)
+ mFilters.append( **it );
mDestFolder = 0;
if (srcFolder) {
mDeleteSrcFolder = false;
@@ -102,10 +109,14 @@
mDeleteSrcFolder = true;
setSourceFolder( tempFolder );
}
+ if (!schedulerList)
+ schedulerList = new QValueList<ActionScheduler*>;
+ schedulerList->append( this );
}
ActionScheduler::~ActionScheduler()
{
+ schedulerList->remove( this );
tempCloseFolders();
mSrcFolder->close();
@@ -151,13 +162,14 @@
this, SLOT(msgAdded(KMFolder*, Q_UINT32)) );
}
-void ActionScheduler::setFilterList( QPtrList<KMFilter> filters )
+void ActionScheduler::setFilterList( QValueList<KMFilter*> filters )
{
mFiltersAreQueued = true;
mQueuedFilters.clear();
- KMFilter *filter;
- for (filter = filters.first(); filter; filter = filters.next())
- mQueuedFilters.append( *filter );
+
+ QValueList<KMFilter*>::Iterator it = filters.begin();
+ for (; it != filters.end(); ++it)
+ mQueuedFilters.append( **it );
if (!mExecuting) {
mFilters = mQueuedFilters;
mFiltersAreQueued = false;
@@ -214,9 +226,13 @@
void ActionScheduler::execFilters(Q_UINT32 serNum)
{
- if (mResult != ResultOk)
- return; // An error has already occurred don't even try to process this msg
-
+ if (mResult != ResultOk) {
+ if ((mResult != ResultCriticalError) &&
+ !mExecuting && !mExecutingLock && !mFetchExecuting)
+ mResult = ResultOk; // Recoverable error
+ else
+ return; // An error has already occurred don't even try to process this msg
+ }
if (MessageProperty::filtering( serNum )) {
// Not good someone else is already filtering this msg
mResult = ResultError;
@@ -373,9 +389,12 @@
if (msg && msg->isComplete()) {
messageFetched( msg );
} else if (msg) {
+ fetchTimeOutTime = QTime::currentTime();
+ fetchTimeOutTimer->start( 60 * 1000, true );
FolderJob *job = msg->parent()->createJob( msg );
connect( job, SIGNAL(messageRetrieved( KMMessage* )),
SLOT(messageFetched( KMMessage* )) );
+ lastJob = job;
job->start();
} else {
mFetchExecuting = false;
@@ -387,6 +406,13 @@
void ActionScheduler::messageFetched( KMMessage *msg )
{
+ fetchTimeOutTimer->stop();
+ if (!msg) {
+ // Should never happen, but sometimes does;
+ fetchMessageTimer->start( 0, true );
+ return;
+ }
+
mFetchSerNums.remove( msg->getMsgSerNum() );
// Note: This may not be necessary. What about when it's time to
@@ -618,7 +644,7 @@
// sometimes the move command doesn't complete so time out after a minute
// and move onto the next message
lastCommand = cmd;
- timeOutTimer->start( 6 * 1000, true );
+ timeOutTimer->start( 60 * 1000, true );
}
void ActionScheduler::moveMessageFinished( KMCommand *command )
@@ -635,8 +661,11 @@
mHeaders->clearSelectableAndAboutToBeDeleted( mOriginalSerNum );
KMMessage *msg = 0;
ReturnCode mOldReturnCode = mResult;
- if (mOriginalSerNum)
+ if (mOriginalSerNum) {
msg = message( mOriginalSerNum );
+ emit filtered( mOriginalSerNum );
+ }
+
mResult = mOldReturnCode; // ignore errors in deleting original message
KMCommand *cmd = 0;
if (msg && msg->parent()) {
@@ -676,6 +705,54 @@
mExecutingLock = false;
mExecuting = false;
finishTimer->start( 0, true );
+ if (mOriginalSerNum) // Try again
+ execFilters( mOriginalSerNum );
}
+void ActionScheduler::fetchTimeOut()
+{
+ // Note: This is a good place for a debug statement
+ assert( lastJob );
+ // sometimes imap jobs seem to just stall so give up and move on
+ disconnect( lastJob, SIGNAL(messageRetrieved( KMMessage* )),
+ this, SLOT(messageFetched( KMMessage* )) );
+ lastJob->kill();
+ lastJob = 0;
+ fetchMessageTimer->start( 0, true );
+}
+
+QString ActionScheduler::debug()
+{
+ QString res;
+ QValueList<ActionScheduler*>::iterator it;
+ int i = 1;
+ for ( it = schedulerList->begin(); it != schedulerList->end(); ++it ) {
+ res.append( QString( "ActionScheduler #%1.\n" ).arg( i ) );
+ if ((*it)->mAccount && kmkernel->find( (*it)->mAccountId )) {
+ res.append( QString( "Account %1, Name %2.\n" )
+ .arg( (*it)->mAccountId )
+ .arg( kmkernel->acctMgr()->find( (*it)->mAccountId )->name() ) );
+ }
+ res.append( QString( "mExecuting %1, " ).arg( (*it)->mExecuting ? "true" : "false" ) );
+ res.append( QString( "mExecutingLock %1, " ).arg( (*it)->mExecutingLock ? "true" : "false" ) );
+ res.append( QString( "mFetchExecuting %1.\n" ).arg( (*it)->mFetchExecuting ? "true" : "false" ) );
+ res.append( QString( "mOriginalSerNum %1.\n" ).arg( (*it)->mOriginalSerNum ) );
+ res.append( QString( "mMessageIt %1.\n" ).arg( ((*it)->mMessageIt != 0) ? *(*it)->mMessageIt : 0 ) );
+ res.append( QString( "mSerNums count %1, " ).arg( (*it)->mSerNums.count() ) );
+ res.append( QString( "mFetchSerNums count %1.\n" ).arg( (*it)->mFetchSerNums.count() ) );
+ res.append( QString( "mResult " ) );
+ if ((*it)->mResult == ResultOk)
+ res.append( QString( "ResultOk.\n" ) );
+ else if ((*it)->mResult == ResultError)
+ res.append( QString( "ResultError.\n" ) );
+ else if ((*it)->mResult == ResultCriticalError)
+ res.append( QString( "ResultCriticalError.\n" ) );
+ else
+ res.append( QString( "Unknown.\n" ) );
+
+ ++i;
+ }
+ return res;
+}
+
#include "actionscheduler.moc"
_______________________________________________
KMail developers mailing list
KMail-devel@kde.org
https://mail.kde.org/mailman/listinfo/kmail-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic