From kdepim-users Fri Feb 13 09:34:28 2015 From: Martin Steigerwald Date: Fri, 13 Feb 2015 09:34:28 +0000 To: kdepim-users Subject: [kdepim-users] Work-around to issues with Akonadi file based caching (was: Re: rant) Message-Id: <1491494.L1tY1MNmJT () merkaba> X-MARC-Message: https://marc.info/?l=kdepim-users&m=142382010023511 MIME-Version: 1 Content-Type: multipart/mixed; boundary="--===============1504179516908893737==" --===============1504179516908893737== Content-Type: multipart/signed; boundary="nextPart1627211.4rWUC30Dz4"; micalg="pgp-sha1"; protocol="application/pgp-signature" --nextPart1627211.4rWUC30Dz4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" I changed the topic so that I can easily reference it in the future. As= it=20 takes me quite some time to write mails like this. Also others may find= it=20 more easily. Am Donnerstag, 12. Februar 2015, 20:06:50 schrieb E. Hakan Duran: > On Thursday, February 12, 2015 08:20:35 AM Martin Steigerwald wrote: > > ... > > I do think the caching in Akonadi is broke. > >=20 > > It caches to much for too long. > >=20 > > For offline IMAP, sure, download everything. I=C2=B4d prefer a mail= dir for > > that, cause its introspectable and not all in one directory, but > > well. > >=20 > > I wrote enough about this already. Reference my other posts about i= t, > > might have been on kdepim-ml as well. >=20 > Thank you Martin. I just wanted to report one more thing I found. I l= eft > akonadi running overnight. About 24 hours later (now) the size of the= > file_db_data was about 11.5 GBs. I ran akonadi fsck again to find tha= t > the ~7.5 GB addition to this folder was all unlinked files. akonadi > fsck put all that data into the lost+find folder, reducing the size o= f > the file_db_data folder size back to about 4.2 GB. >=20 > Maybe I should setup a cronjob to run akonadi fsck daily, what do you= > think? I recommend you to try the SizeTreshold thing instead. Cause even after= at=20 least a week of using it I still have: martin@merkaba:~/.local/share/akonadi> du -sh file_db_data=20 3,1M file_db_data martin@merkaba:~/.local/share/akonadi> find file_db_data | wc -l 26 martin@merkaba:~/.local/share/akonadi> du -sch db_data/akonadi/* | sort= - rh | head -5 2,0G insgesamt 1,5G db_data/akonadi/parttable.ibd 516M db_data/akonadi/pimitemtable.ibd 23M db_data/akonadi/pimitemflagrelation.ibd 240K db_data/akonadi/collectiontable.ibd Yes, you may get a bit larger database, but it appears to me that in th= ere=20 Akonadi at least cleans up old cached items as it doesn=C2=B4t seem tha= t it got=20 much larger since I setup this. What I use basically is: martin@merkaba:~> cat .config/akonadi/akonadiserverrc=20 [%General] Driver=3DQMYSQL SizeThreshold=3D32768 [=E2=80=A6] Then I did a akonadictl fsck and a akonadictl vacuum *once*. And it sti= ll=20 seems to be good.=20 KMail doesn=C2=B4t usually loose the connection with Akonadi anymore as= well. I do think the file based caching, while I would prefer it, is just bro= ke.=20 And for another reason as well: 1) It caches too much. 2) It caches too many files in *one* directory (it broke a storage=20 appliance standard setup at work) 3) It caches too long. And now in addition to that: 4) It doesn=C2=B4t clean up properly after itself as akonadictl fsck cl= early=20 shows. The fsck action seems to clean up files Akonadi server doesn=C2=B4= t know=20 about anymore, i.e. that have no reference in database. But then, my=20= expectation would be that if Akonadi deletes a cache entry it deletes t= he=20 reference in the *database* and the *file*. Yet, it seems that Akonadi=20= looses track of what its doing. I think thats even another bug. So in my eyes the Akonadi file based code has *at least* four severe bu= gs.=20 In addition to the performance problems it seems to cause. So while I=C2=B4d prefer that Akonadi uses am introspectable directory=20= structure with files to cache things and only keeps references in the=20= database that just doesn=C2=B4t seem to work. Above is now with my POP3 maildir based setup... Additional to that see: Bug 338402 - File system cache is inneficient : too many file per direc= tory https://bugs.kde.org/show_bug.cgi?id=3D338402#c9 Re: Possible akonadi problem? https://lists.debian.org/debian-kde/2015/01/msg00055.html > I already commented on one of the bugs you mentioned earlier Martin. = Do > you think I should report this there or as a separate bug? I think a bug report that clearly shows that akonadictl fsck again and=20= again produces files in lost + found that Akonadi didn=C2=B4t know abou= t it=20 anynmore and just didn=C2=B4t delete it self, i.e. that Akonadi file ba= sed cache=20 doesn=C2=B4t seem to clean up after itself properly might be useful. But I think, spending time to develop Akonadi or maybe better Akonadi N= ext=20 is even more helpful. I currently have other priorities, so while I thi= nk=20 current Akonadi still has quite some serious issues, I also I am also=20= prepared to deal this them for now. The SizeThreshold thing helped me with that. But have caution: This is = no=20 official recommendation of Akonadi developers. Still, it gave me a good= =20 improvement with performance and latency *and* seemed to have solved th= e=20 insane amount of file and data in file_db_data issue for me. Or at leas= t it=20 worked around it successfully. I have switched two IMAP and one POP3 setup to this and upto now have n= o=20 issues that I could attribute to that configuration change. Thanks, =2D-=20 Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 --nextPart1627211.4rWUC30Dz4 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEABECAAYFAlTdxSkACgkQmRvqrKWZhMfqKACfSDPn9kZ7e/fIuQY5YuPPBf48 J44AoJ68kqemV4PSLETzxE1OrinPJaDK =TYfG -----END PGP SIGNATURE----- --nextPart1627211.4rWUC30Dz4-- --===============1504179516908893737== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ KDE PIM users mailing list Subscription management: https://mail.kde.org/mailman/listinfo/kdepim-users --===============1504179516908893737==--