[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-bugs-dist
Subject:    [Akonadi] [Bug 338402] File system cache is inneficient : too many file per directory
From:       Daniel Vrátil <dvratil () kde ! org>
Date:       2015-06-30 20:19:27
Message-ID: bug-338402-17878-cCtdU8rGqD () http ! bugs ! kde ! org/
[Download RAW message or body]

https://bugs.kde.org/show_bug.cgi?id=338402

--- Comment #13 from Daniel Vrátil <dvratil@kde.org> ---
Hi all,

so the problem with files just endlessly piling up in file_db_data should
finally be fixed. Now for the original bug report: the file_db_data containing
too many files.

So I was thinking about how to decrease the file count - obviously the right
solution is the levelled cache as Bastien pointed out. I think in our case one
level should be enough. The filenames of the external payload parts consist of
incremental database unique ID, so I am planning to use the last two digits of
the ID for the folder name to ensure even distribution of files into the cache
folders.

To have some numbers here, using modulo 100 for folder name means 100 folders
in file_db_data (file_db_data/00 - 99/). With 1 million emails in Akonadi
(which is a performance baseline for me) and with average ratio of external vs
internal cache being cca 1:2 we get cca 500 000 external files in file_db_data,
so that means cca 5 000 files per folder. That sounds like a reasonable number
to me. 

With 2 levels of indirection (using last 3rd and 2nd digit for L1 and last
digit for L2 (so file_db_data/00-99/0-9/) we would have 100 folders with 10
folders in each so 1000 folders in total. That would give us cca 500 emails per
folder with 1 000 000 emails in Akonadi. For the baseline of 1 000 000 emails
two levels of indirection seem to be unnecessary.

Regarding migration, we cannot do automatic migration on start, that would take
too much time and resources to perform during start, so only newly created
files would be moved to the cache folders. It would be possible to implement
the full migration as part of akonadictl fsck though, so users could run it
manually.

What are your opinions?

-- 
You are receiving this mail because:
You are watching all bug changes.=
[prev in list] [next in list] [prev in thread] [next in thread]