[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: Review Request 112725: Save memory and time by lazy-loading the item data if possible
From: "Frank Reininghaus" <frank78ac () googlemail ! com>
Date: 2013-09-16 23:02:31
Message-ID: 20130916230231.25510.48216 () vidsolbach ! de
[Download RAW message or body]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/112725/
-----------------------------------------------------------
(Updated Sept. 16, 2013, 11:02 p.m.)
Review request for Dolphin.
Changes
-------
Oops, it turns out that I forgot two places in the code which still assumed that the \
QHash is always filled with data. That caused a crash when grouping by "Name" in some \
situations. Fixing that is straightforward, but I guess I'll better test it a bit \
more and probably add a unit test to make sure that I don't mess anything up before \
pushing to master.
Description
-------
This patch tries to prevent calling KFileItemModel::retrieveData() when possible in \
order to save memory and time when loading huge directories.
There are changes in the following members of KFileItemModel:
* data() and setData() are modified such that the QHash "values" for an item is \
initialized with retrieveData() if that has not happened already.
* indexForKeyboardSearch() uses the "text" from the KFileItem itself, rather than \
looking than calling data() (which can be expensive if called for many items because \
data() might call retrieveData()).
* isExpandable() calls data(), rather than accessing the QHash "values" directly. If \
this hash was not initialized it, isExpandable() would always return the default \
value "false".
* createItemDataList() does not call retrieveData() for each item any more. However, \
if the sort role is such that sortRoleCompare() would try to access the QHash \
"values" when sorting, retrieveData() is called to ensure that sorting works \
correctly.
Things that could still be improved:
* Currently, the layouting in Compact View calls KFileItemModel::data() for every \
item. In the case that only the "Name" is shown, this could be prevented (similar to \
the approach that is used in Icons View, see \
https://git.reviewboard.kde.org/r/112253/
* When changing the roles (e.g., by adding or removing columns in Details View) or \
changing the sort role, retrieveData() is currently called for all items. Moreover, a \
changedItems signal is issued for all items, such that KFileItemModelRolesUpdater \
tries to load icons/previews for all items (like in Dolphin < 4.11).
Diffs (updated)
-----
dolphin/src/kitemviews/kfileitemmodel.h 8410161
dolphin/src/kitemviews/kfileitemmodel.cpp 51ff142
Diff: http://git.reviewboard.kde.org/r/112725/diff/
Testing
-------
I opened a directory with 100,000 files in Icons View and Details View with a release \
build, and measured the memory usage with KSysGuard. Moreover, I measured the time \
required to load the directory using the following patch for DolphinView: \
http://paste.kde.org/pae0a64c1/
The time required for the initial loading is not very meaningful because it varies \
strongly. We can get a better idea of how much time is spent inside Dolphin for \
loading if all items are cached, so I re-loaded the folder 8 of times by pressing \
Alt+Up and then Alt+Left.
In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB (-17%), \
and the average time from 2898 ms to 2814 ms (-2.9%).
In Details View, the memory consumption is reduced from 225.1 MB to 166.6 MB (-26%), \
and the average time from 1539 ms to 1076 ms (-30%).
The savings are larger in Details View because retrieveData() stores more information \
about the items in the QHash "values" (like the "expanded parents count").
I could not see any significant differences in Compace View (as expected).
Thanks,
Frank Reininghaus
[Attachment #3 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="http://git.reviewboard.kde.org/r/112725/">http://git.reviewboard.kde.org/r/112725/</a>
</td>
</tr>
</table>
<br />
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('http://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Dolphin.</div>
<div>By Frank Reininghaus.</div>
<p style="color: grey;"><i>Updated Sept. 16, 2013, 11:02 p.m.</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Changes</h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Oops, it turns out that I forgot two places in the code which still \
assumed that the QHash is always filled with data. That caused a crash when grouping \
by "Name" in some situations. Fixing that is straightforward, but I guess \
I'll better test it a bit more and probably add a unit test to make sure that I \
don't mess anything up before pushing to master.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">This patch tries to prevent calling KFileItemModel::retrieveData() when \
possible in order to save memory and time when loading huge directories.
There are changes in the following members of KFileItemModel:
* data() and setData() are modified such that the QHash "values" for an \
item is initialized with retrieveData() if that has not happened already.
* indexForKeyboardSearch() uses the "text" from the KFileItem itself, \
rather than looking than calling data() (which can be expensive if called for many \
items because data() might call retrieveData()).
* isExpandable() calls data(), rather than accessing the QHash "values" \
directly. If this hash was not initialized it, isExpandable() would always return the \
default value "false".
* createItemDataList() does not call retrieveData() for each item any more. However, \
if the sort role is such that sortRoleCompare() would try to access the QHash \
"values" when sorting, retrieveData() is called to ensure that sorting \
works correctly.
Things that could still be improved:
* Currently, the layouting in Compact View calls KFileItemModel::data() for every \
item. In the case that only the "Name" is shown, this could be prevented \
(similar to the approach that is used in Icons View, see \
https://git.reviewboard.kde.org/r/112253/
* When changing the roles (e.g., by adding or removing columns in Details View) or \
changing the sort role, retrieveData() is currently called for all items. Moreover, a \
changedItems signal is issued for all items, such that KFileItemModelRolesUpdater \
tries to load icons/previews for all items (like in Dolphin < 4.11). </pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">I opened a directory with 100,000 files in Icons View and Details View \
with a release build, and measured the memory usage with KSysGuard. Moreover, I \
measured the time required to load the directory using the following patch for \
DolphinView: http://paste.kde.org/pae0a64c1/
The time required for the initial loading is not very meaningful because it varies \
strongly. We can get a better idea of how much time is spent inside Dolphin for \
loading if all items are cached, so I re-loaded the folder 8 of times by pressing \
Alt+Up and then Alt+Left.
In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB (-17%), \
and the average time from 2898 ms to 2814 ms (-2.9%).
In Details View, the memory consumption is reduced from 225.1 MB to 166.6 MB (-26%), \
and the average time from 1539 ms to 1076 ms (-30%).
The savings are larger in Details View because retrieveData() stores more information \
about the items in the QHash "values" (like the "expanded parents \
count").
I could not see any significant differences in Compace View (as expected).</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> \
(updated)</h1> <ul style="margin-left: 3em; padding-left: 0;">
<li>dolphin/src/kitemviews/kfileitemmodel.h <span style="color: \
grey">(8410161)</span></li>
<li>dolphin/src/kitemviews/kfileitemmodel.cpp <span style="color: \
grey">(51ff142)</span></li>
</ul>
<p><a href="http://git.reviewboard.kde.org/r/112725/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic