[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: Review Request 112725: Save memory and time by lazy-loading the item data if possible
From: "Frank Reininghaus" <frank78ac () googlemail ! com>
Date: 2013-09-16 23:02:31
Message-ID: 20130916230231.25510.48216 () vidsolbach ! de
[Download RAW message or body]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/112725/
-----------------------------------------------------------
(Updated Sept. 16, 2013, 11:02 p.m.)
Review request for Dolphin.
Changes
-------
Oops, it turns out that I forgot two places in the code which still assumed that the QHash is always \
filled with data. That caused a crash when grouping by "Name" in some situations. Fixing that is \
straightforward, but I guess I'll better test it a bit more and probably add a unit test to make sure \
that I don't mess anything up before pushing to master.
Description
-------
This patch tries to prevent calling KFileItemModel::retrieveData() when possible in order to save memory \
and time when loading huge directories.
There are changes in the following members of KFileItemModel:
* data() and setData() are modified such that the QHash "values" for an item is initialized with \
retrieveData() if that has not happened already.
* indexForKeyboardSearch() uses the "text" from the KFileItem itself, rather than looking than calling \
data() (which can be expensive if called for many items because data() might call retrieveData()).
* isExpandable() calls data(), rather than accessing the QHash "values" directly. If this hash was not \
initialized it, isExpandable() would always return the default value "false".
* createItemDataList() does not call retrieveData() for each item any more. However, if the sort role is \
such that sortRoleCompare() would try to access the QHash "values" when sorting, retrieveData() is called \
to ensure that sorting works correctly.
Things that could still be improved:
* Currently, the layouting in Compact View calls KFileItemModel::data() for every item. In the case that \
only the "Name" is shown, this could be prevented (similar to the approach that is used in Icons View, \
see https://git.reviewboard.kde.org/r/112253/
* When changing the roles (e.g., by adding or removing columns in Details View) or changing the sort \
role, retrieveData() is currently called for all items. Moreover, a changedItems signal is issued for all \
items, such that KFileItemModelRolesUpdater tries to load icons/previews for all items (like in Dolphin < \
4.11).
Diffs (updated)
-----
dolphin/src/kitemviews/kfileitemmodel.h 8410161
dolphin/src/kitemviews/kfileitemmodel.cpp 51ff142
Diff: http://git.reviewboard.kde.org/r/112725/diff/
Testing
-------
I opened a directory with 100,000 files in Icons View and Details View with a release build, and measured \
the memory usage with KSysGuard. Moreover, I measured the time required to load the directory using the \
following patch for DolphinView: http://paste.kde.org/pae0a64c1/
The time required for the initial loading is not very meaningful because it varies strongly. We can get a \
better idea of how much time is spent inside Dolphin for loading if all items are cached, so I re-loaded \
the folder 8 of times by pressing Alt+Up and then Alt+Left.
In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB (-17%), and the average time \
from 2898 ms to 2814 ms (-2.9%).
In Details View, the memory consumption is reduced from 225.1 MB to 166.6 MB (-26%), and the average time \
from 1539 ms to 1076 ms (-30%).
The savings are larger in Details View because retrieveData() stores more information about the items in \
the QHash "values" (like the "expanded parents count").
I could not see any significant differences in Compace View (as expected).
Thanks,
Frank Reininghaus
[Attachment #3 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="http://git.reviewboard.kde.org/r/112725/">http://git.reviewboard.kde.org/r/112725/</a>
</td>
</tr>
</table>
<br />
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: \
url('http://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black solid;"> <tr>
<td>
<div>Review request for Dolphin.</div>
<div>By Frank Reininghaus.</div>
<p style="color: grey;"><i>Updated Sept. 16, 2013, 11:02 p.m.</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Changes</h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Oops, it turns out that I forgot two places \
in the code which still assumed that the QHash is always filled with data. That caused a crash when \
grouping by "Name" in some situations. Fixing that is straightforward, but I guess I'll \
better test it a bit more and probably add a unit test to make sure that I don't mess anything up \
before pushing to master.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid \
#b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">This patch tries to prevent calling \
KFileItemModel::retrieveData() when possible in order to save memory and time when loading huge \
directories.
There are changes in the following members of KFileItemModel:
* data() and setData() are modified such that the QHash "values" for an item is initialized \
with retrieveData() if that has not happened already.
* indexForKeyboardSearch() uses the "text" from the KFileItem itself, rather than looking than \
calling data() (which can be expensive if called for many items because data() might call \
retrieveData()).
* isExpandable() calls data(), rather than accessing the QHash "values" directly. If this hash \
was not initialized it, isExpandable() would always return the default value "false".
* createItemDataList() does not call retrieveData() for each item any more. However, if the sort role is \
such that sortRoleCompare() would try to access the QHash "values" when sorting, retrieveData() \
is called to ensure that sorting works correctly.
Things that could still be improved:
* Currently, the layouting in Compact View calls KFileItemModel::data() for every item. In the case that \
only the "Name" is shown, this could be prevented (similar to the approach that is used in \
Icons View, see https://git.reviewboard.kde.org/r/112253/
* When changing the roles (e.g., by adding or removing columns in Details View) or changing the sort \
role, retrieveData() is currently called for all items. Moreover, a changedItems signal is issued for all \
items, such that KFileItemModelRolesUpdater tries to load icons/previews for all items (like in Dolphin \
< 4.11). </pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">I opened a directory with 100,000 files in \
Icons View and Details View with a release build, and measured the memory usage with KSysGuard. Moreover, \
I measured the time required to load the directory using the following patch for DolphinView: \
http://paste.kde.org/pae0a64c1/
The time required for the initial loading is not very meaningful because it varies strongly. We can get a \
better idea of how much time is spent inside Dolphin for loading if all items are cached, so I re-loaded \
the folder 8 of times by pressing Alt+Up and then Alt+Left.
In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB (-17%), and the average time \
from 2898 ms to 2814 ms (-2.9%).
In Details View, the memory consumption is reduced from 225.1 MB to 166.6 MB (-26%), and the average time \
from 1539 ms to 1076 ms (-30%).
The savings are larger in Details View because retrieveData() stores more information about the items in \
the QHash "values" (like the "expanded parents count").
I could not see any significant differences in Compace View (as expected).</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> (updated)</h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>dolphin/src/kitemviews/kfileitemmodel.h <span style="color: grey">(8410161)</span></li>
<li>dolphin/src/kitemviews/kfileitemmodel.cpp <span style="color: grey">(51ff142)</span></li>
</ul>
<p><a href="http://git.reviewboard.kde.org/r/112725/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic