[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: Review Request: Implemented multithreading in KFileItemModelSortAlgorithm
From: "Emmanuel Pescosta" <emmanuelpescosta099 () gmail ! com>
Date: 2012-10-25 11:13:38
Message-ID: 20121025111338.1942.33532 () vidsolbach ! de
[Download RAW message or body]
> On Oct. 25, 2012, 5:37 a.m., Frank Reininghaus wrote:
> > Thanks for working on this!
> > =
> > How many CPU cores do you have? I have only 2, so I can't really test t=
he ">2 cores" case. However, I have an idea what could be the reason for th=
e performance drop that you see when trying to use more than 2 threads: you=
actually try to create more threads than QtConcurrent lets you.
> > =
> > Example: Suppose you have 4 cores. Your parallelSort() function will:
> > =
> > * sort the first half using QtConcurrent::run(), which creates a thread=
1,
> > * sort the second half using QtConcurrent::run(), which creates a threa=
d 2.
> > =
> > Thread 1 will now split the first half into two quarters and:
> > =
> > * create a thread 3 for the first quarter,
> > * create a thread 4 for the second quarter.
> > =
> > At the same time, thread 2 tries to sort the third and fourth quarters =
concurrently, but Qt:Concurrent doesn't let it because the number of parall=
el threads is already equal to the number of cores -> it has to wait until =
the first half is finished. But note that threads 1 and 2 are actually idle=
in this scenario!
> > =
> > Suggested solution: in parallelSort(), only sort the first half using Q=
tConcurrent::run(), and sort the second one synchronously using a plain rec=
ursive call of parallelSort(). Could you try if that works?
> >
> How many CPU cores do you have?
I have 4 cores. (Every core runs with 100% while sorting)
> you actually try to create more threads than QtConcurrent lets you
Nope, I already thought about that ;) I implemented it without recursive fu=
nction calls.
> Suggested solution: in parallelSort(), only sort the first half using QtC=
oncurrent::run(), and sort the second one synchronously using a plain recur=
sive call of =
> parallelSort(). Could you try if that works?
Hmm ... 97 seconds for 500.000 files (With recursive parallelSort and with =
your suggestions)
I will upload my other patch, which uses as many threads as possible (accor=
ding to idealThreadCount())
- Emmanuel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/107025/#review20832
-----------------------------------------------------------
On Oct. 24, 2012, 5:20 p.m., Emmanuel Pescosta wrote:
> =
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/107025/
> -----------------------------------------------------------
> =
> (Updated Oct. 24, 2012, 5:20 p.m.)
> =
> =
> Review request for Dolphin and Frank Reininghaus.
> =
> =
> Description
> -------
> =
> Implemented multithreading in KFileItemModelSortAlgorithm.
> =
> If more than 100 items to sort and ideal thread count is greater than 1 -=
> sort them with parallelSort (2 Threads)
> =
> Use maximal 2 Threads, because more than 2 Threads are "slower" (more ove=
rhead than speed up). (I also have a patch which uses n Threads for sorting=
, if you want test it ;)
> =
> =
> Diffs
> -----
> =
> dolphin/src/kitemviews/private/kfileitemmodelsortalgorithm.h 3a596df =
> dolphin/src/kitemviews/private/kfileitemmodelsortalgorithm.cpp e0aac13 =
> =
> Diff: http://git.reviewboard.kde.org/r/107025/diff/
> =
> =
> Testing
> -------
> =
> About 2 seconds faster with sorting 500.000 files.
> About 5 seconds faster with sorting 1.000.000 files.
> =
> =
> Thanks,
> =
> Emmanuel Pescosta
> =
>
[Attachment #3 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="http://git.reviewboard.kde.org/r/107025/">http://git.reviewboard.kde.org/r/107025/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On October 25th, 2012, 5:37 a.m., <b>Frank \
Reininghaus</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px \
solid #d0d0d0; padding-left: 10px;"> <pre style="white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Thanks for working on this!
How many CPU cores do you have? I have only 2, so I can't really test the \
">2 cores" case. However, I have an idea what could be the reason for \
the performance drop that you see when trying to use more than 2 threads: you \
actually try to create more threads than QtConcurrent lets you.
Example: Suppose you have 4 cores. Your parallelSort() function will:
* sort the first half using QtConcurrent::run(), which creates a thread 1,
* sort the second half using QtConcurrent::run(), which creates a thread 2.
Thread 1 will now split the first half into two quarters and:
* create a thread 3 for the first quarter,
* create a thread 4 for the second quarter.
At the same time, thread 2 tries to sort the third and fourth quarters concurrently, \
but Qt:Concurrent doesn't let it because the number of parallel threads is \
already equal to the number of cores -> it has to wait until the first half is \
finished. But note that threads 1 and 2 are actually idle in this scenario!
Suggested solution: in parallelSort(), only sort the first half using \
QtConcurrent::run(), and sort the second one synchronously using a plain recursive \
call of parallelSort(). Could you try if that works? </pre>
</blockquote>
</blockquote>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">> How many CPU cores \
do you have? I have 4 cores. (Every core runs with 100% while sorting)
> you actually try to create more threads than QtConcurrent lets you
Nope, I already thought about that ;) I implemented it without recursive function \
calls.
> Suggested solution: in parallelSort(), only sort the first half using \
QtConcurrent::run(), and sort the second one synchronously using a plain recursive \
call of > parallelSort(). Could you try if that works?
Hmm ... 97 seconds for 500.000 files (With recursive parallelSort and with your \
suggestions)
I will upload my other patch, which uses as many threads as possible (according to \
idealThreadCount())</pre> <br />
<p>- Emmanuel</p>
<br />
<p>On October 24th, 2012, 5:20 p.m., Emmanuel Pescosta wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('http://git.reviewboard.kde.org/media/rb/images/review_request_box_top_bg.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Dolphin and Frank Reininghaus.</div>
<div>By Emmanuel Pescosta.</div>
<p style="color: grey;"><i>Updated Oct. 24, 2012, 5:20 p.m.</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">Implemented multithreading in KFileItemModelSortAlgorithm.
If more than 100 items to sort and ideal thread count is greater than 1 -> sort \
them with parallelSort (2 Threads)
Use maximal 2 Threads, because more than 2 Threads are "slower" (more \
overhead than speed up). (I also have a patch which uses n Threads for sorting, if \
you want test it ;)</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">About 2 seconds faster with sorting 500.000 files. About 5 seconds \
faster with sorting 1.000.000 files.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>dolphin/src/kitemviews/private/kfileitemmodelsortalgorithm.h <span style="color: \
grey">(3a596df)</span></li>
<li>dolphin/src/kitemviews/private/kfileitemmodelsortalgorithm.cpp <span \
style="color: grey">(e0aac13)</span></li>
</ul>
<p><a href="http://git.reviewboard.kde.org/r/107025/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic