[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kdepim-users
Subject:    Re: [kdepim-users] Fwd: KMail + IMAP painfully slow
From:       Ross Boylan <ross () biostat ! ucsf ! edu>
Date:       2007-05-16 22:14:32
Message-ID: 200705161514.32968.ross () biostat ! ucsf ! edu
[Download RAW message or body]

On Wednesday 16 May 2007 14:18, Ingo Klöcker wrote:
> On Wednesday 16 May 2007 21:30, Ross Boylan wrote:
> > > I've spent the day watching KMail be unresponsive (using most of
> > > the CPU, not updating the screen, not responding to the mouse)
> > > using a local (Cyrus) IMAP server.
> > >
> > > Moving messages takes many minutes, sometimes (even when it's not
> > > many messages).
> >
> > Ingo asked (sorry about the single > for all quotes)
> >
> > > Moving messages from one IMAP folder to the other should be quite
> > > fast because it happens on the server.
> >
> > I can confirm this is quite fast when I do it with other programs.
> >
> > > What happens during those many
> > > minutes? Is KMail eating CPU like mad? Or does is just hang and do
> > > nothing?
> >
> > KMail eats CPU like mad.  It does so in all the scenarios I
> > described.  I noticed today that when I moved some messages to a
> > folder the columns on the left for unread and total started updating
> > after I move the messages in.  It appears that after the messages are
> > moved in KMail scans the target folder; I'm guessing kmail builds up
> > its indices.
>
> If the total column is visible then KMail updates it at least on the
> first start to get an initial value. You might try whether disabling
> the total column helps.

disabling just Total, or Total and Unread?  Both are getting updated.

>
> Also, the first time KMail accesses a folder it does indeed create its
> indices.
>
> > While it was doing this I attempted to conduct a new quick search by
> > typing into the search bar.  The program did not respond to this
> > until it finished whatever it was doing with the target box.  The
> > target box was not open in my display (except for showing on the left
> > hand pane).
>
> Yes, there are still lots of blocking operations in KMail.
>
> > I've also noticed some strange behavior.  I had about 7000 messages
> > on a remote server.  After I moved them to the local server I opened
> > the remote folder.  kmail showed the messages that were previously
> > there in the upper right pane (this is with the default setup for
> > panes--not sure what their official name is).  It then started
> > reducing the counts shown on the left, and eliminating message lines
> > shown in the upper right.
> >
> > I got bored waiting for this to finish and I switched to another box.
> >  This made the countdown process stop.  When I switched back to the
> > now-empty box kmail said "no new mail" and left the display as it
> > was, i.e., the left pane showed the  box had quite a few messages,
> > and the upper riight pane also showed quite a few messages.  I was
> > eventually able to force a reread (I think I switched to another box
> > on the remote server and then back to the empty one).
>
> Sounds like an update problem.
Do you mean update of the display, or something else?
>
> > Perhaps consistent with this, I have a lot of boxes which show
> > partial counts in the unread column and nothing at all in the Total
> > column.  When I switch to them these values update.  So perhaps what
> > I'm seeing is the result of other times I switched into and then out
> > of a folder.
>
> KMail caches the last known values.
Maybe it could keep track of whether it was interrupted as well.

>
> > Finally, I once switched into a box after having accessed the
> > contents of the box from another program (I can't recall if it was a
> > read or a read and delete).  This was on the remote server.  When I
> > switched into the box the left hand display started counting down,
> > and the upper right started deleting headers.  This was very
> > alarming, since it went way beyond the number (if any) I had deleted.
> >  I switched out of the box and used another program to copy its
> > entire contents somewhere else.  The I reopened the folder in kmail,
> > which eventually showed the proper counts and contents.  I thought it
> > was deleting messages, but I guess it was again updating its indices,
> > since the mail did remain in the folder.
>
> Yeah, I guess so.
>
> > Since the information in the counts is available from a single call
> > to the IMAP server I conclude that kmail is not getting the
> > information that way, but is instead building its own indices.
>
> I'm not sure how we do get the count. It's quite possible that we just
> count the number of messages marked as unread in our caches instead of
> using the number given to us by the IMAP server so that we can use the
> same code also for local folders (where we have no other choice but to
> count ourselves).

I thought some programs wrote information directly into the message for 
non-IMAP mail.  I think mutt may put some status info there.  Otherwise, if 
you use a different mail client you won't have access to which messages 
are "seen."  

>
> > > Getting the text of a message to display can take forever.
> > >
> > > I disabled automatic checking, and this seemed to help some.  The
> > > manual check remains slow, I guess because it insists on visiting
> > > every folder.
> > >
> > > Am I doing something wrong?  Is my mailstore bigger than KMail was
> > > designed for?  Or is KMail just not that good at IMAP?
> >
Sounds as if the root of the problem is that KMail is not so good at managing 
large messages stores, wherever they reside.  But if the messages are in, for 
example, traditional mbox format, probably accessing the messages causes a 
bottleneck before indexing/caching becomes an issue.

> > Ingo replies
> >
> > > Especially with IMAP the size of the mailstore should be more or
> > > less irrelevant. A possible bottleneck is the fact that we use the
> > > mbox format for caching the message headers.
> >
> > I suspect that the slowness is the result of the local indices,
> > caches, or whatever that kmail is creating.  Perhaps once they are
> > fully created things aren't so bad.
>
> Yes, I think so too. The first import of a large IMAP account does use a
> lot of resources but afterwards it should work okay. It does for me,
> but my largest IMAP folder only holds about 12000 messages.
>
> > I experience kmail as semi-multithreaded.  It does seem to be doing
> > several things at once sometimes (e.g., updating counts on the left;
> > showing a moving progress bar on the lower right; checking mail;
> > displaying messages I've selected).  But often it locks up while
> > doing them.  Since threads are designed to support a responsive user
> > interface while background processing happens (e.g., updating
> > index/cache for a folder), and since the interface is often
> > unresponsive, I'm not sure what's going on.
>
> KMail is single-threaded, but we make extensive use of Qt's signal-slot
> mechanism together with splitting jobs (like moving loads of messages)
> up into many small tasks (like moving a few messages) to keep the GUI
> responsive. As you have noticed this is not used for every operation so
> KMail does still lock up during some operations.
>
> > Also, as I reported above with the folder I emptied, kmail sometimes
> > seems to get confused when I interrupt a task by switching folders or
> > mailboxes.
>
> We do have a few bottlenecks in updating the message list (i.e. the
> upper right pane). This list is often redrawn after each change and
> that's painfully slow if lots of changes happen. When you leave the
> folder then redrawing is obviously no longer necessary. I guess this
> explains some of the behavior you saw.
Are you saying that each folder has an associated graphical view that gets 
updated even if it's not displayed?  In the problem I described above, the 
target folder information was getting updated, but the message list that was 
showing was always that of the source.

>
> > > I have 2 imap accounts; a small remote one and a large local one.
> > > The local one has about 115 folders (with several levels of
> > > nesting), some of which have a lot of messages (several in the 10's
> > > of thousands, the largest around 80k).  KMail's area under .kde is
> > > about 335Mg.  I have yet to visit most of the folders.
> >
> > Ingo asked
> >
> > > Do the problems occur with both accounts?
> >
> > Primarily the large local one, though as noted above I've seen some
> > oddities with the remote one too.
> >
> > > kmail 3.5.5.dfsg.1-6 (the about panel says it's 1.9.5) on Debian
> > > GNU/Linux. 3GHz Pentium4 with a decent SATA disk.
> > >
> > > It's doing better than evolution, which can't define the accounts,
> > > and arguably better than mutt, which hangs every time it exits a
> > > folder.
> >
> > P.S. about the quick search: I keep entering the search term and then
> > hitting enter.  The enter brings up the currently selected message, I
> > think.  You might consider making the enter key a no-op in the search
> > bar.  I'm accustomed to enter meaning "do the search".  I guess that,
> > since the search is incremental, that's redundant in this case.
>
> You might want to file a bug/wish at bugs.kde.org for this.
>
Yes, that's a simple one.  Anything useful I can do about the more general 
performance/responsive issues I encountered?  I'm not even sure how many 
distinct issues they involve.

Thanks.
Ross
_______________________________________________
KDE PIM users mailing list
kdepim-users@kde.org
https://mail.kde.org/mailman/listinfo/kdepim-users

[prev in list] [next in list] [prev in thread] [next in thread]