'Review Request 130084: Add a pair of flags forcing fsync during copy loop'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-frameworks-devel
Subject:    Review Request 130084: Add a pair of flags forcing fsync during copy loop
From:       KJ Tsanaktsidis <kjtsanaktsidis () gmail ! com>
Date:       2017-04-12 11:36:31
Message-ID: 20170412113631.2157.51628 () mimi ! kde ! org
[Download RAW message or body]

--===============5604433722974871500==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/130084/
-----------------------------------------------------------

Review request for KDE Frameworks.


Repository: kio


Description
-------

When copying a large-ish file (~1-2GB) from very fast storage to very slow storage \
(e.g. an NVME SSD to a cheap USB flash drive) on a machine with lots of RAM, Dolphin \
displays a progress bar which finishes in a fraction of a second (i.e. as fast as it \
takes to read the source file into the Linux page cache). Unmounting the drive then \
of course takes a long time, with only an indeterminate spinner.

This patch adds an option to force fsync during copy jobs, so that the copy progress \
bar measures how long it will take to actually copy the file to the destination.

I've added two flags - Fsync and FsyncCrossFilesystem - to the JobEnum flag. The \
former will cause all copy operations to fsync during the copy loop, whilst the \
latter will only fsync copies that are across different filesystems.

If this patch gets OK'd, I have another patch which adds support for this into the \
appropriate places in Dolphin. I would think that at least FsyncCrossFilesystem \
should be the default, but Fsync always might be a little heavy handed. At the least \
fsync'ing cross-filesystem copies ensures that the unmount won't take forever.


Diffs
-----


Diff: https://git.reviewboard.kde.org/r/130084/diff/


Testing
-------

Tested the patch with KDE/Dolphin on Arch Linux, which is version 5.32.0. The diff \
applies cleanly to master so I assume there shouldn't be any issues there, but I've \
not actually checked that. As advertised, copying a file to USB flash storage now \
displays an accurate progress bar.

I experimented with how often fsync should be called on my hardware, and I found \
calling it every ~1M copied caused no decrease in copy performance whilst still \
providing accurate progress info. That is the setting I've gone with in this patch. \
I'm open to suggestions on how this could be tuned better though.


Thanks,

KJ Tsanaktsidis


--===============5604433722974871500==
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit




<html>
 <body>
  <div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
   <table bgcolor="#f9f3c9" width="100%" cellpadding="12" style="border: 1px #c9c399 \
solid; border-radius: 6px; -moz-border-radius: 6px; -webkit-border-radius: 6px;">  \
<tr>  <td>
      This is an automatically generated e-mail. To reply, visit:
      <a href="https://git.reviewboard.kde.org/r/130084/">https://git.reviewboard.kde.org/r/130084/</a>
  </td>
    </tr>
   </table>
   <br />




<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="12" style="border: \
1px #888a85 solid; border-radius: 6px; -moz-border-radius: 6px; \
-webkit-border-radius: 6px;">  <tr>
  <td>

<div>Review request for KDE Frameworks.</div>
<div>By KJ Tsanaktsidis.</div>










<div style="margin-top: 1.5em;">
 <b style="color: #575012; font-size: 10pt;">Repository: </b>
kio
</div>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
 <table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0">  <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">When copying a large-ish file (~1-2GB) from very fast \
storage to very slow storage (e.g. an NVME SSD to a cheap USB flash drive) on a \
machine with lots of RAM, Dolphin displays a progress bar which finishes in a \
fraction of a second (i.e. as fast as it takes to read the source file into the Linux \
page cache). Unmounting the drive then of course takes a long time, with only an \
indeterminate spinner.</p> <p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">This patch adds an option to force \
fsync during copy jobs, so that the copy progress bar measures how long it will take \
to actually copy the file to the destination.</p> <p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">I've \
added two flags - Fsync and FsyncCrossFilesystem - to the JobEnum flag. The former \
will cause all copy operations to fsync during the copy loop, whilst the latter will \
only fsync copies that are across different filesystems.</p> <p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">If \
this patch gets OK'd, I have another patch which adds support for this into the \
appropriate places in Dolphin. I would think that at least FsyncCrossFilesystem \
should be the default, but Fsync always might be a little heavy handed. At the least \
fsync'ing cross-filesystem copies ensures that the unmount won't take \
forever.</p></pre>  </td>
 </tr>
</table>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0">  <tr>
  <td>
   <pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">Tested the patch with KDE/Dolphin on Arch Linux, which \
is version 5.32.0. The diff applies cleanly to master so I assume there shouldn't be \
any issues there, but I've not actually checked that. As advertised, copying a file \
to USB flash storage now displays an accurate progress bar.</p> <p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">I \
experimented with how often fsync should be called on my hardware, and I found \
calling it every ~1M copied caused no decrease in copy performance whilst still \
providing accurate progress info. That is the setting I've gone with in this patch. \
I'm open to suggestions on how this could be tuned better though.</p></pre>  </td>
 </tr>
</table>


<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">

</ul>

<p><a href="https://git.reviewboard.kde.org/r/130084/diff/" style="margin-left: \
3em;">View Diff</a></p>






  </td>
 </tr>
</table>



  </div>
 </body>
</html>


--===============5604433722974871500==--


[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic