[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-frameworks-devel
Subject: Re: Review Request 130084: Add a pair of flags forcing fsync during copy loop
From: Oswald Buddenhagen <ossi () kde ! org>
Date: 2017-05-05 8:25:13
Message-ID: 20170505082513.2829.71179 () mimi ! kde ! org
[Download RAW message or body]
--===============2750225537895842507==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/130084/#review103186
-----------------------------------------------------------
src/ioslaves/file/file_unix.cpp (line 112)
<https://git.reviewboard.kde.org/r/130084/#comment68663>
typo
src/ioslaves/file/file_unix.cpp (line 138)
<https://git.reviewboard.kde.org/r/130084/#comment68664>
why are you testing hotpluggable? what's the exact definition in this \
context? technically, any sata drive is hotpluggable ...
src/ioslaves/file/file_unix.cpp (line 324)
<https://git.reviewboard.kde.org/r/130084/#comment68665>
i'd use some parens here to make it more readable.
- Oswald Buddenhagen
On April 17, 2017, 1:27 p.m., KJ Tsanaktsidis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/130084/
> -----------------------------------------------------------
>
> (Updated April 17, 2017, 1:27 p.m.)
>
>
> Review request for KDE Frameworks, Oswald Buddenhagen and Thiago \
> Macieira.
>
> Repository: kio
>
>
> Description
> -------
>
> When copying a large-ish file (~1-2GB) from very fast storage to very \
> slow storage (e.g. an NVME SSD to a cheap USB flash drive) on a machine \
> with lots of RAM, Dolphin displays a progress bar which finishes in a \
> fraction of a second (i.e. as fast as it takes to read the source file \
> into the Linux page cache). Unmounting the drive then of course takes a \
> long time, with only an indeterminate spinner.
> This patch adds an option to force fsync during copy jobs, so that the \
> copy progress bar measures how long it will take to actually copy the \
> file to the destination.
> I've added two flags - Fsync and FsyncCrossFilesystem - to the JobEnum \
> flag. The former will cause all copy operations to fsync during the copy \
> loop, whilst the latter will only fsync copies that are across different \
> filesystems.
> If this patch gets OK'd, I have another patch which adds support for this \
> into the appropriate places in Dolphin. I would think that at least \
> FsyncCrossFilesystem should be the default, but Fsync always might be a \
> little heavy handed. At the least fsync'ing cross-filesystem copies \
> ensures that the unmount won't take forever.
>
> Diffs
> -----
>
> src/ioslaves/file/CMakeLists.txt b9132ced9d4a08b2cf9f9bbbaa3bd43f026c6469 \
> src/ioslaves/file/ConfigureChecks.cmake \
> 5a83d1b9fbe90c851c774e3b467468d93b5a2bd4 \
> src/ioslaves/file/config-kioslave-file.h.cmake \
> 372f79d01ad4597aae0b2ae62627648fe7680b64 src/ioslaves/file/file_unix.cpp \
> 3c1b9927e3dd2d0134f77caec6e6b24a0356d26f
> Diff: https://git.reviewboard.kde.org/r/130084/diff/
>
>
> Testing
> -------
>
> Tested the patch with KDE/Dolphin on Arch Linux, which is version 5.32.0. \
> The diff applies cleanly to master so I assume there shouldn't be any \
> issues there, but I've not actually checked that. As advertised, copying \
> a file to USB flash storage now displays an accurate progress bar.
> I experimented with how often fsync should be called on my hardware, and \
> I found calling it every ~1M copied caused no decrease in copy \
> performance whilst still providing accurate progress info. That is the \
> setting I've gone with in this patch. I'm open to suggestions on how this \
> could be tuned better though.
>
> Thanks,
>
> KJ Tsanaktsidis
>
>
--===============2750225537895842507==
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="12" style="border: \
1px #c9c399 solid; border-radius: 6px; -moz-border-radius: 6px; \
-webkit-border-radius: 6px;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/130084/">https://git.reviewboard.kde.org/r/130084/</a>
</td>
</tr>
</table>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid \
#C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid \
#C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/130084/diff/2/?file=494464#file494464line112" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/ioslaves/file/file_unix.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 2)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; \
"></pre></td> <th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; \
border-right: 1px solid #C0C0C0;" align="right"><font \
size="2">112</font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="c1">// I suppose it could have just been removed (althouth there \
isn't much</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">typo</p></pre> </div>
</div>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid \
#C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid \
#C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/130084/diff/2/?file=494464#file494464line138" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/ioslaves/file/file_unix.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 2)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; \
"></pre></td> <th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; \
border-right: 1px solid #C0C0C0;" align="right"><font \
size="2">138</font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="k">return</span> <span class="n">real_storagedrive</span><span \
class="o">-></span><span class="n">isRemovable</span><span \
class="p">()</span> <span class="o">||</span> <span \
class="n">real_storagedrive</span><span class="o">-></span><span \
class="n">isHotpluggable</span><span class="p">();</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">why are you testing \
hotpluggable? what's the exact definition in this context? technically, any \
sata drive is hotpluggable ...</p></pre> </div>
</div>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid \
#C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid \
#C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/130084/diff/2/?file=494464#file494464line324" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/ioslaves/file/file_unix.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 2)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; \
"></pre></td> <th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; \
border-right: 1px solid #C0C0C0;" align="right"><font \
size="2">324</font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="k">if</span> <span class="p">(</span><span \
class="n">should_fsync</span> <span class="o">&&</span> <span \
class="n">niters</span> <span class="o">%</span> <span \
class="mi">128</span> <span class="o">==</span> <span \
class="mi">0</span><span class="p">)</span> <span \
class="p">{</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">i'd use some parens here to \
make it more readable.</p></pre> </div>
</div>
<br />
<p>- Oswald Buddenhagen</p>
<br />
<p>On April 17th, 2017, 1:27 p.m. CEST, KJ Tsanaktsidis wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="12" \
style="border: 1px #888a85 solid; border-radius: 6px; -moz-border-radius: \
6px; -webkit-border-radius: 6px;"> <tr>
<td>
<div>Review request for KDE Frameworks, Oswald Buddenhagen and Thiago \
Macieira.</div> <div>By KJ Tsanaktsidis.</div>
<p style="color: grey;"><i>Updated April 17, 2017, 1:27 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kio
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description \
</h1> <table width="100%" bgcolor="#ffffff" cellspacing="0" \
cellpadding="10" style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">When copying a large-ish file \
(~1-2GB) from very fast storage to very slow storage (e.g. an NVME SSD to a \
cheap USB flash drive) on a machine with lots of RAM, Dolphin displays a \
progress bar which finishes in a fraction of a second (i.e. as fast as it \
takes to read the source file into the Linux page cache). Unmounting the \
drive then of course takes a long time, with only an indeterminate \
spinner.</p> <p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">This patch adds an option to \
force fsync during copy jobs, so that the copy progress bar measures how \
long it will take to actually copy the file to the destination.</p> <p \
style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">I've added two flags - Fsync and \
FsyncCrossFilesystem - to the JobEnum flag. The former will cause all copy \
operations to fsync during the copy loop, whilst the latter will only fsync \
copies that are across different filesystems.</p> <p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">If this patch gets OK'd, I have another patch which adds support \
for this into the appropriate places in Dolphin. I would think that at \
least FsyncCrossFilesystem should be the default, but Fsync always might be \
a little heavy handed. At the least fsync'ing cross-filesystem copies \
ensures that the unmount won't take forever.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing \
</h1> <table width="100%" bgcolor="#ffffff" cellspacing="0" \
cellpadding="10" style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">Tested the patch with \
KDE/Dolphin on Arch Linux, which is version 5.32.0. The diff applies \
cleanly to master so I assume there shouldn't be any issues there, but I've \
not actually checked that. As advertised, copying a file to USB flash \
storage now displays an accurate progress bar.</p> <p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">I experimented with how often fsync should be called on my \
hardware, and I found calling it every ~1M copied caused no decrease in \
copy performance whilst still providing accurate progress info. That is the \
setting I've gone with in this patch. I'm open to suggestions on how this \
could be tuned better though.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> \
</h1> <ul style="margin-left: 3em; padding-left: 0;">
<li>src/ioslaves/file/CMakeLists.txt <span style="color: \
grey">(b9132ced9d4a08b2cf9f9bbbaa3bd43f026c6469)</span></li>
<li>src/ioslaves/file/ConfigureChecks.cmake <span style="color: \
grey">(5a83d1b9fbe90c851c774e3b467468d93b5a2bd4)</span></li>
<li>src/ioslaves/file/config-kioslave-file.h.cmake <span style="color: \
grey">(372f79d01ad4597aae0b2ae62627648fe7680b64)</span></li>
<li>src/ioslaves/file/file_unix.cpp <span style="color: \
grey">(3c1b9927e3dd2d0134f77caec6e6b24a0356d26f)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/130084/diff/" \
style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
--===============2750225537895842507==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic