[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kfm-devel
Subject:    Re: Review Request 112725: Save memory and time by lazy-loading the item data if possible
From:       "Frank Reininghaus" <frank78ac () googlemail ! com>
Date:       2013-09-16 23:02:31
Message-ID: 20130916230231.25510.48216 () vidsolbach ! de
[Download RAW message or body]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/112725/
-----------------------------------------------------------

(Updated Sept. 16, 2013, 11:02 p.m.)


Review request for Dolphin.


Changes
-------

Oops, it turns out that I forgot two places in the code which still assumed \
that the QHash is always filled with data. That caused a crash when \
grouping by "Name" in some situations. Fixing that is straightforward, but \
I guess I'll better test it a bit more and probably add a unit test to make \
sure that I don't mess anything up before pushing to master.


Description
-------

This patch tries to prevent calling KFileItemModel::retrieveData() when \
possible in order to save memory and time when loading huge directories.


There are changes in the following members of KFileItemModel:

* data() and setData() are modified such that the QHash "values" for an \
item is initialized with retrieveData() if that has not happened already.

* indexForKeyboardSearch() uses the "text" from the KFileItem itself, \
rather than looking than calling data() (which can be expensive if called \
for many items because data() might call retrieveData()).

* isExpandable() calls data(), rather than accessing the QHash "values" \
directly. If this hash was not initialized it, isExpandable() would always \
return the default value "false".

* createItemDataList() does not call retrieveData() for each item any more. \
However, if the sort role is such that sortRoleCompare() would try to \
access the QHash "values" when sorting, retrieveData() is called to ensure \
that sorting works correctly.


Things that could still be improved:

* Currently, the layouting in Compact View calls KFileItemModel::data() for \
every item. In the case that only the "Name" is shown, this could be \
prevented (similar to the approach that is used in Icons View, see \
https://git.reviewboard.kde.org/r/112253/

* When changing the roles (e.g., by adding or removing columns in Details \
View) or changing the sort role, retrieveData() is currently called for all \
items. Moreover, a changedItems signal is issued for all items, such that \
KFileItemModelRolesUpdater tries to load icons/previews for all items (like \
in Dolphin < 4.11).


Diffs (updated)
-----

  dolphin/src/kitemviews/kfileitemmodel.h 8410161 
  dolphin/src/kitemviews/kfileitemmodel.cpp 51ff142 

Diff: http://git.reviewboard.kde.org/r/112725/diff/


Testing
-------

I opened a directory with 100,000 files in Icons View and Details View with \
a release build, and measured the memory usage with KSysGuard. Moreover, I \
measured the time required to load the directory using the following patch \
for DolphinView: http://paste.kde.org/pae0a64c1/

The time required for the initial loading is not very meaningful because it \
varies strongly. We can get a better idea of how much time is spent inside \
Dolphin for loading if all items are cached, so I re-loaded the folder 8 of \
times by pressing Alt+Up and then Alt+Left.

In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB \
(-17%), and the average time from 2898 ms to 2814 ms (-2.9%).

In Details View, the memory consumption is reduced from 225.1 MB to 166.6 \
MB (-26%), and the average time from 1539 ms to 1076 ms (-30%).

The savings are larger in Details View because retrieveData() stores more \
information about the items in the QHash "values" (like the "expanded \
parents count").

I could not see any significant differences in Compace View (as expected).


Thanks,

Frank Reininghaus


[Attachment #3 (text/html)]

<html>
 <body>
  <div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
   <table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px \
#c9c399 solid;">  <tr>
     <td>
      This is an automatically generated e-mail. To reply, visit:
      <a href="http://git.reviewboard.kde.org/r/112725/">http://git.reviewboard.kde.org/r/112725/</a>
  </td>
    </tr>
   </table>
   <br />




<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('http://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px \
black solid;">  <tr>
  <td>

<div>Review request for Dolphin.</div>
<div>By Frank Reininghaus.</div>


<p style="color: grey;"><i>Updated Sept. 16, 2013, 11:02 p.m.</i></p>



<h1 style="color: #575012; font-size: 10pt; margin-top: \
1.5em;">Changes</h1> <table width="100%" bgcolor="#ffffff" cellspacing="0" \
cellpadding="10" style="border: 1px solid #b8b5a0">  <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Oops, it turns out that I forgot two places in the code which \
still assumed that the QHash is always filled with data. That caused a \
crash when grouping by &quot;Name&quot; in some situations. Fixing that is \
straightforward, but I guess I&#39;ll better test it a bit more and \
probably add a unit test to make sure that I don&#39;t mess anything up \
before pushing to master.</pre>  </td>
 </tr>
</table>




<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description \
</h1>  <table width="100%" bgcolor="#ffffff" cellspacing="0" \
cellpadding="10" style="border: 1px solid #b8b5a0">  <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">This patch tries to prevent calling \
KFileItemModel::retrieveData() when possible in order to save memory and \
time when loading huge directories.


There are changes in the following members of KFileItemModel:

* data() and setData() are modified such that the QHash &quot;values&quot; \
for an item is initialized with retrieveData() if that has not happened \
already.

* indexForKeyboardSearch() uses the &quot;text&quot; from the KFileItem \
itself, rather than looking than calling data() (which can be expensive if \
called for many items because data() might call retrieveData()).

* isExpandable() calls data(), rather than accessing the QHash \
&quot;values&quot; directly. If this hash was not initialized it, \
isExpandable() would always return the default value &quot;false&quot;.

* createItemDataList() does not call retrieveData() for each item any more. \
However, if the sort role is such that sortRoleCompare() would try to \
access the QHash &quot;values&quot; when sorting, retrieveData() is called \
to ensure that sorting works correctly.


Things that could still be improved:

* Currently, the layouting in Compact View calls KFileItemModel::data() for \
every item. In the case that only the &quot;Name&quot; is shown, this could \
be prevented (similar to the approach that is used in Icons View, see \
https://git.reviewboard.kde.org/r/112253/

* When changing the roles (e.g., by adding or removing columns in Details \
View) or changing the sort role, retrieveData() is currently called for all \
items. Moreover, a changedItems signal is issued for all items, such that \
KFileItemModelRolesUpdater tries to load icons/previews for all items (like \
in Dolphin &lt; 4.11). </pre>
  </td>
 </tr>
</table>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing \
</h1> <table width="100%" bgcolor="#ffffff" cellspacing="0" \
cellpadding="10" style="border: 1px solid #b8b5a0">  <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">I opened a directory with 100,000 files in Icons View and \
Details View with a release build, and measured the memory usage with \
KSysGuard. Moreover, I measured the time required to load the directory \
using the following patch for DolphinView: http://paste.kde.org/pae0a64c1/

The time required for the initial loading is not very meaningful because it \
varies strongly. We can get a better idea of how much time is spent inside \
Dolphin for loading if all items are cached, so I re-loaded the folder 8 of \
times by pressing Alt+Up and then Alt+Left.

In Icons View, the memory consumption is reduced from 198.2 MB to 164.6 MB \
(-17%), and the average time from 2898 ms to 2814 ms (-2.9%).

In Details View, the memory consumption is reduced from 225.1 MB to 166.6 \
MB (-26%), and the average time from 1539 ms to 1076 ms (-30%).

The savings are larger in Details View because retrieveData() stores more \
information about the items in the QHash &quot;values&quot; (like the \
&quot;expanded parents count&quot;).

I could not see any significant differences in Compace View (as \
expected).</pre>  </td>
 </tr>
</table>




<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> \
(updated)</h1> <ul style="margin-left: 3em; padding-left: 0;">

 <li>dolphin/src/kitemviews/kfileitemmodel.h <span style="color: \
grey">(8410161)</span></li>

 <li>dolphin/src/kitemviews/kfileitemmodel.cpp <span style="color: \
grey">(51ff142)</span></li>

</ul>

<p><a href="http://git.reviewboard.kde.org/r/112725/diff/" \
style="margin-left: 3em;">View Diff</a></p>







  </td>
 </tr>
</table>




  </div>
 </body>
</html>



[prev in list] [next in list] [prev in thread] [next in thread] 

Configure | About | News | Add a list | Sponsored by KoreLogic