[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: Review Request 115432: Only initialize the hash m_items in KFileItemModel if it is needed
From: "Commit Hook" <null () kde ! org>
Date: 2014-02-07 9:25:45
Message-ID: 20140207092545.10823.92529 () probe ! kde ! org
[Download RAW message or body]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/115432/#review49165
-----------------------------------------------------------
This review has been submitted with commit e45fc620b4c0d629eadf1b13538020cf7e79a4e3 \
by Frank Reininghaus to branch master.
- Commit Hook
On Feb. 2, 2014, 7:03 p.m., Frank Reininghaus wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/115432/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2014, 7:03 p.m.)
>
>
> Review request for Dolphin.
>
>
> Bugs: 329494
> http://bugs.kde.org/show_bug.cgi?id=329494
>
>
> Repository: kde-baseapps
>
>
> Description
> -------
>
> KFileItemModel has a member
>
> QHash<KUrl, int> m_items
>
> which makes it possible to find the index of an item if the URL is known. m_items \
> is updated every time an item is added to/removed from the current directory, or if \
> the sort order changes. This is convenient in many situations, but sometimes, this \
> hash also causes problems:
>
> 1. In some corner cases, the model can get into a state where the indexes in \
> m_items are not correct. There were, e.g.,
> https://bugs.kde.org/show_bug.cgi?id=324371
> https://bugs.kde.org/show_bug.cgi?id=325359
>
> which I could reproduce by doing a search in Details View and expanding folders, \
> such that the same URL appeared twice in the view (but only once in m_items).
> But I think that there are other ways to make the relationship between m_items and \
> the model contents inconsistent. A crash that looks like it is most likely caused \
> by such problems is
> https://bugs.kde.org/show_bug.cgi?id=329494
>
>
> 2. Initializing m_items costs CPU cycles and memory, which is wasteful if m_items \
> is not used at all.
>
> Therefore, I propose to make the following changes:
>
> (a) Do not require that all URLs are in m_items any more. Instead, the new \
> requirement is: if there are N URLs in m_items, it is guaranteed that they are the \
> first N URLs in the model.
> (b) Replace most calls of m_items.value(url) by index(url) in KFileItemModel. Note \
> that some local variables 'index' had to be renamed for this to work.
> (c) In KFileItemModel::index(const KUrl&), check if the URL is in m_items already. \
> If that is not the case, add 1000 URLs to the hash, and check again. Repeat until \
> either the URL is found, or all URLs are in the hash (-1 is returned in the latter \
> case). Note that we always know where to continue adding URLs: if m_items.count() \
> is N, then "m_itemData.at(i)->item.url()" is the first URL that is not in m_items.
> BTW, my first version of index(const KUrl&) added URLs to the hash one by one, and \
> checked every time if the next URL is the right one by comparing the URLs with ==. \
> However, this was slow and required a lot more memory because comparing URLs with \
> the == operator triggers a parsing of the URL, which needs memory and CPU cycles. I \
> don't know if this is still the case in Qt 5, but in any case, adding new URLs in \
> larger batches and then checking if the hash contains the searched URL should have \
> a lower amortized cost in any case.
> (d) In insertItems() and removeItems(), clear the hash m_items. It will be \
> re-populated the next time index() is called. Note that Dolphin < 4.11 also cleared \
> the entire hash in these cases (it re-created the entire hash immediately though). \
> In contrast to that, 4.11 and 4.12 only update the URLs/indexes starting from the \
> first added/removed item. This can be problematic though - in bug 329494, we get a \
> crash when trying to remove one of the removed URLs from m_items.
>
> Diffs
> -----
>
> dolphin/src/kitemviews/kfileitemmodel.h 0224290
> dolphin/src/kitemviews/kfileitemmodel.cpp b3c56e1
>
> Diff: https://git.reviewboard.kde.org/r/115432/diff/
>
>
> Testing
> -------
>
> I could not find any regressions. Even if I could never reproduce the crash in bug \
> 329494, I'm confident that the crash is fixed because the line where it tries to \
> access the URL of a removed KFileItem and then crashes is removed (and there is no \
> other access to the KFileItem in that function).
> Memory usage when loading a directory with 100,000 files is reduced by about 5 MiB \
> in all view modes (from about 150 to 145) if previews are disabled. (If previews \
> are enabled, KFileItemModelRolesUpdater calls the index(KUrl&) method, which fills \
> m_items, and memory usage is as before).
> The time for loading a directory with 100,000 files is reduced by about 200 ms on \
> my system (to about 4.3 seconds in Icons View, 3.7 in Compact and 2.8 in Details).
>
> Thanks,
>
> Frank Reininghaus
>
>
[Attachment #3 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/115432/">https://git.reviewboard.kde.org/r/115432/</a>
</td>
</tr>
</table>
<br />
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">This review has been \
submitted with commit e45fc620b4c0d629eadf1b13538020cf7e79a4e3 by Frank Reininghaus \
to branch master.</pre> <br />
<p>- Commit Hook</p>
<br />
<p>On February 2nd, 2014, 7:03 p.m. UTC, Frank Reininghaus wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('https://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Dolphin.</div>
<div>By Frank Reininghaus.</div>
<p style="color: grey;"><i>Updated Feb. 2, 2014, 7:03 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>
<a href="http://bugs.kde.org/show_bug.cgi?id=329494">329494</a>
</div>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kde-baseapps
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">KFileItemModel has a member
QHash<KUrl, int> m_items
which makes it possible to find the index of an item if the URL is known. m_items is \
updated every time an item is added to/removed from the current directory, or if the \
sort order changes. This is convenient in many situations, but sometimes, this hash \
also causes problems:
1. In some corner cases, the model can get into a state where the indexes in m_items \
are not correct. There were, e.g.,
https://bugs.kde.org/show_bug.cgi?id=324371
https://bugs.kde.org/show_bug.cgi?id=325359
which I could reproduce by doing a search in Details View and expanding folders, such \
that the same URL appeared twice in the view (but only once in m_items).
But I think that there are other ways to make the relationship between m_items and \
the model contents inconsistent. A crash that looks like it is most likely caused by \
such problems is
https://bugs.kde.org/show_bug.cgi?id=329494
2. Initializing m_items costs CPU cycles and memory, which is wasteful if m_items is \
not used at all.
Therefore, I propose to make the following changes:
(a) Do not require that all URLs are in m_items any more. Instead, the new \
requirement is: if there are N URLs in m_items, it is guaranteed that they are the \
first N URLs in the model.
(b) Replace most calls of m_items.value(url) by index(url) in KFileItemModel. Note \
that some local variables 'index' had to be renamed for this to work.
(c) In KFileItemModel::index(const KUrl&), check if the URL is in m_items \
already. If that is not the case, add 1000 URLs to the hash, and check again. Repeat \
until either the URL is found, or all URLs are in the hash (-1 is returned in the \
latter case). Note that we always know where to continue adding URLs: if \
m_items.count() is N, then "m_itemData.at(i)->item.url()" is the first \
URL that is not in m_items.
BTW, my first version of index(const KUrl&) added URLs to the hash one by one, \
and checked every time if the next URL is the right one by comparing the URLs with \
==. However, this was slow and required a lot more memory because comparing URLs with \
the == operator triggers a parsing of the URL, which needs memory and CPU cycles. I \
don't know if this is still the case in Qt 5, but in any case, adding new URLs in \
larger batches and then checking if the hash contains the searched URL should have a \
lower amortized cost in any case.
(d) In insertItems() and removeItems(), clear the hash m_items. It will be \
re-populated the next time index() is called. Note that Dolphin < 4.11 also \
cleared the entire hash in these cases (it re-created the entire hash immediately \
though). In contrast to that, 4.11 and 4.12 only update the URLs/indexes starting \
from the first added/removed item. This can be problematic though - in bug 329494, we \
get a crash when trying to remove one of the removed URLs from m_items.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">I could not find any regressions. Even if I could never reproduce the \
crash in bug 329494, I'm confident that the crash is fixed because the line where \
it tries to access the URL of a removed KFileItem and then crashes is removed (and \
there is no other access to the KFileItem in that function).
Memory usage when loading a directory with 100,000 files is reduced by about 5 MiB in \
all view modes (from about 150 to 145) if previews are disabled. (If previews are \
enabled, KFileItemModelRolesUpdater calls the index(KUrl&) method, which fills \
m_items, and memory usage is as before).
The time for loading a directory with 100,000 files is reduced by about 200 ms on my \
system (to about 4.3 seconds in Icons View, 3.7 in Compact and 2.8 in Details).</pre> \
</td> </tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>dolphin/src/kitemviews/kfileitemmodel.h <span style="color: \
grey">(0224290)</span></li>
<li>dolphin/src/kitemviews/kfileitemmodel.cpp <span style="color: \
grey">(b3c56e1)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/115432/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic