[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: Review Request 115432: Only initialize the hash m_items in KFileItemModel if it is needed
From: "Commit Hook" <null () kde ! org>
Date: 2014-02-07 9:25:45
Message-ID: 20140207092545.10823.92529 () probe ! kde ! org
[Download RAW message or body]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/115432/#review49165
-----------------------------------------------------------
This review has been submitted with commit e45fc620b4c0d629eadf1b13538020cf7e79a4e3 by Frank Reininghaus \
to branch master.
- Commit Hook
On Feb. 2, 2014, 7:03 p.m., Frank Reininghaus wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/115432/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2014, 7:03 p.m.)
>
>
> Review request for Dolphin.
>
>
> Bugs: 329494
> http://bugs.kde.org/show_bug.cgi?id=329494
>
>
> Repository: kde-baseapps
>
>
> Description
> -------
>
> KFileItemModel has a member
>
> QHash<KUrl, int> m_items
>
> which makes it possible to find the index of an item if the URL is known. m_items is updated every time \
> an item is added to/removed from the current directory, or if the sort order changes. This is \
> convenient in many situations, but sometimes, this hash also causes problems:
>
> 1. In some corner cases, the model can get into a state where the indexes in m_items are not correct. \
> There were, e.g.,
> https://bugs.kde.org/show_bug.cgi?id=324371
> https://bugs.kde.org/show_bug.cgi?id=325359
>
> which I could reproduce by doing a search in Details View and expanding folders, such that the same URL \
> appeared twice in the view (but only once in m_items).
> But I think that there are other ways to make the relationship between m_items and the model contents \
> inconsistent. A crash that looks like it is most likely caused by such problems is
> https://bugs.kde.org/show_bug.cgi?id=329494
>
>
> 2. Initializing m_items costs CPU cycles and memory, which is wasteful if m_items is not used at all.
>
>
> Therefore, I propose to make the following changes:
>
> (a) Do not require that all URLs are in m_items any more. Instead, the new requirement is: if there are \
> N URLs in m_items, it is guaranteed that they are the first N URLs in the model.
> (b) Replace most calls of m_items.value(url) by index(url) in KFileItemModel. Note that some local \
> variables 'index' had to be renamed for this to work.
> (c) In KFileItemModel::index(const KUrl&), check if the URL is in m_items already. If that is not the \
> case, add 1000 URLs to the hash, and check again. Repeat until either the URL is found, or all URLs are \
> in the hash (-1 is returned in the latter case). Note that we always know where to continue adding \
> URLs: if m_items.count() is N, then "m_itemData.at(i)->item.url()" is the first URL that is not in \
> m_items.
> BTW, my first version of index(const KUrl&) added URLs to the hash one by one, and checked every time \
> if the next URL is the right one by comparing the URLs with ==. However, this was slow and required a \
> lot more memory because comparing URLs with the == operator triggers a parsing of the URL, which needs \
> memory and CPU cycles. I don't know if this is still the case in Qt 5, but in any case, adding new URLs \
> in larger batches and then checking if the hash contains the searched URL should have a lower amortized \
> cost in any case.
> (d) In insertItems() and removeItems(), clear the hash m_items. It will be re-populated the next time \
> index() is called. Note that Dolphin < 4.11 also cleared the entire hash in these cases (it re-created \
> the entire hash immediately though). In contrast to that, 4.11 and 4.12 only update the URLs/indexes \
> starting from the first added/removed item. This can be problematic though - in bug 329494, we get a \
> crash when trying to remove one of the removed URLs from m_items.
>
> Diffs
> -----
>
> dolphin/src/kitemviews/kfileitemmodel.h 0224290
> dolphin/src/kitemviews/kfileitemmodel.cpp b3c56e1
>
> Diff: https://git.reviewboard.kde.org/r/115432/diff/
>
>
> Testing
> -------
>
> I could not find any regressions. Even if I could never reproduce the crash in bug 329494, I'm \
> confident that the crash is fixed because the line where it tries to access the URL of a removed \
> KFileItem and then crashes is removed (and there is no other access to the KFileItem in that function). \
> Memory usage when loading a directory with 100,000 files is reduced by about 5 MiB in all view modes \
> (from about 150 to 145) if previews are disabled. (If previews are enabled, KFileItemModelRolesUpdater \
> calls the index(KUrl&) method, which fills m_items, and memory usage is as before).
> The time for loading a directory with 100,000 files is reduced by about 200 ms on my system (to about \
> 4.3 seconds in Icons View, 3.7 in Compact and 2.8 in Details).
>
> Thanks,
>
> Frank Reininghaus
>
>
[Attachment #3 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/115432/">https://git.reviewboard.kde.org/r/115432/</a>
</td>
</tr>
</table>
<br />
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: \
-o-pre-wrap; word-wrap: break-word;">This review has been submitted with commit \
e45fc620b4c0d629eadf1b13538020cf7e79a4e3 by Frank Reininghaus to branch master.</pre> <br />
<p>- Commit Hook</p>
<br />
<p>On February 2nd, 2014, 7:03 p.m. UTC, Frank Reininghaus wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: \
url('https://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black solid;"> <tr>
<td>
<div>Review request for Dolphin.</div>
<div>By Frank Reininghaus.</div>
<p style="color: grey;"><i>Updated Feb. 2, 2014, 7:03 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>
<a href="http://bugs.kde.org/show_bug.cgi?id=329494">329494</a>
</div>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kde-baseapps
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid \
#b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">KFileItemModel has a member
QHash<KUrl, int> m_items
which makes it possible to find the index of an item if the URL is known. m_items is updated every time \
an item is added to/removed from the current directory, or if the sort order changes. This is convenient \
in many situations, but sometimes, this hash also causes problems:
1. In some corner cases, the model can get into a state where the indexes in m_items are not correct. \
There were, e.g.,
https://bugs.kde.org/show_bug.cgi?id=324371
https://bugs.kde.org/show_bug.cgi?id=325359
which I could reproduce by doing a search in Details View and expanding folders, such that the same URL \
appeared twice in the view (but only once in m_items).
But I think that there are other ways to make the relationship between m_items and the model contents \
inconsistent. A crash that looks like it is most likely caused by such problems is
https://bugs.kde.org/show_bug.cgi?id=329494
2. Initializing m_items costs CPU cycles and memory, which is wasteful if m_items is not used at all.
Therefore, I propose to make the following changes:
(a) Do not require that all URLs are in m_items any more. Instead, the new requirement is: if there are N \
URLs in m_items, it is guaranteed that they are the first N URLs in the model.
(b) Replace most calls of m_items.value(url) by index(url) in KFileItemModel. Note that some local \
variables 'index' had to be renamed for this to work.
(c) In KFileItemModel::index(const KUrl&), check if the URL is in m_items already. If that is not the \
case, add 1000 URLs to the hash, and check again. Repeat until either the URL is found, or all URLs are \
in the hash (-1 is returned in the latter case). Note that we always know where to continue adding URLs: \
if m_items.count() is N, then "m_itemData.at(i)->item.url()" is the first URL that is not in \
m_items.
BTW, my first version of index(const KUrl&) added URLs to the hash one by one, and checked every time \
if the next URL is the right one by comparing the URLs with ==. However, this was slow and required a lot \
more memory because comparing URLs with the == operator triggers a parsing of the URL, which needs memory \
and CPU cycles. I don't know if this is still the case in Qt 5, but in any case, adding new URLs in \
larger batches and then checking if the hash contains the searched URL should have a lower amortized cost \
in any case.
(d) In insertItems() and removeItems(), clear the hash m_items. It will be re-populated the next time \
index() is called. Note that Dolphin < 4.11 also cleared the entire hash in these cases (it re-created \
the entire hash immediately though). In contrast to that, 4.11 and 4.12 only update the URLs/indexes \
starting from the first added/removed item. This can be problematic though - in bug 329494, we get a \
crash when trying to remove one of the removed URLs from m_items.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">I could not find any regressions. Even if I \
could never reproduce the crash in bug 329494, I'm confident that the crash is fixed because the line \
where it tries to access the URL of a removed KFileItem and then crashes is removed (and there is no \
other access to the KFileItem in that function).
Memory usage when loading a directory with 100,000 files is reduced by about 5 MiB in all view modes \
(from about 150 to 145) if previews are disabled. (If previews are enabled, KFileItemModelRolesUpdater \
calls the index(KUrl&) method, which fills m_items, and memory usage is as before).
The time for loading a directory with 100,000 files is reduced by about 200 ms on my system (to about 4.3 \
seconds in Icons View, 3.7 in Compact and 2.8 in Details).</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>dolphin/src/kitemviews/kfileitemmodel.h <span style="color: grey">(0224290)</span></li>
<li>dolphin/src/kitemviews/kfileitemmodel.cpp <span style="color: grey">(b3c56e1)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/115432/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic