[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-mac
Subject: Re: [KDE/Mac] Review Request 125123: [WIP] CMake plugin: prevent the QFileSystemWatcher from hitting
From: "Milian Wolff" <mail () milianw ! de>
Date: 2015-09-10 14:42:39
Message-ID: 20150910144239.24667.61416 () mimi ! kde ! org
[Download RAW message or body]
--===============1508090619274995610==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit
> On Sept. 10, 2015, 2 p.m., Milian Wolff wrote:
> > where does it crash? we get the warnings even on linux but it never crashes. are \
> > there parts of this patch you want us to include or not?
>
> René J.V. Bertin wrote:
> It crashes in the dynamic_cast from `f` to `folder`, which is why I create a fully \
> new folder instance when there was an error in adding the path. I don't know why it \
> crashes nor exactly on what, but for now it hasn't crashed with this approach. For \
> now I'm just putting this up for reference. I'm testing the patch on Linux too, and \
> haven't even yet been able to get warnings. I was expecting the kqueue backend to \
> be used on Linux too, but apparently another one is used there, which uses less \
> file descriptors (or none at all?).
Crashing in a dynamic cast means the pointer is invalid, and non-null. This should \
not happen, and I don't see how that is related to what you touch here. Is there an \
uninitialized ptr being returned? Is it dangling? And inotify does use file \
descriptors for every watcher as well, and there is (by default) a low limit (was it \
1024?) for that in the kernel. Baloo and other apps like KDevelop need more, which is \
why people and distros regularly increase that limit now to a couple of ten \
thousands. And again - even if the limit is reached (I've seen that many times) it \
never crashed. Find out why it crashed (see below). This patch here looks like it's \
working around an issue, instead of fixing it.
> On Sept. 10, 2015, 2 p.m., Milian Wolff wrote:
> > projectmanagers/cmake/cmakecommitchangesjob.cpp, line 233
> > <https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line233>
> >
> > in Qt 5 you can simply use the return value of this function. Why do you keep \
> > working on Qt 4 based KDevplatform on Mac - I really don't get it... You are \
> > making your life unneccessarily complicated.
>
> René J.V. Bertin wrote:
> It's really not hard to get once you remember that KF5 doesn't work on OS X. Parts \
> of it build in CI, but that's about it. I've more or less given up hope that this \
> situation will approve anytime soon.
Your valuable OS X time would be much better spent on making KF5 work, than fixing \
random things like this one on Qt 4 / KDE4 / KDevelop 4.
> On Sept. 10, 2015, 2 p.m., Milian Wolff wrote:
> > projectmanagers/cmake/cmakecommitchangesjob.cpp, line 227
> > <https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line227>
> >
> > wth? just use the assert above. if you hit that in a release build then you'll \
> > hit it in a debug build - and that should _not_ happen. thus assertion is fine no \
> > need to add this for release only
>
> René J.V. Bertin wrote:
> I didn't add these for nothing: the assert macros evaluate to noops in the release \
> builds. The application is likely to crash in that case, of course, but in this \
> case I find it better to abort it with an explicit message rather than crash in a \
> random later location.
Right, and you'll hit these issues in a debug build and fix them there. We do not \
litter our code base with "oh snap what if this happened?" in release mode for a \
reason - it's slowing things down.
- Milian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/125123/#review85118
-----------------------------------------------------------
On Sept. 9, 2015, 8:18 p.m., René J.V. Bertin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/125123/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2015, 8:18 p.m.)
>
>
> Review request for KDE Software on Mac OS X and KDevelop.
>
>
> Repository: kdevelop
>
>
> Description
> -------
>
> This is a confirmed bug in the 4.7 branch (and possibly in the KF5 version) where \
> the cmake project importer can add (many) more paths to QFileSystemWatcher objects \
> than the maximum supported number of open files \
> (https://bugs.kde.org/show_bug.cgi?id=341551). On OS X at least this leads almost \
> inevitably to a crash.
> This patch prevents this by keeping track of the number of added paths, and by not \
> adding more than a preconfigured maximum (determined empirically in this draft \
> implementation). The crash occurs in `folder = dynamic_cast<CMakeFolderItem*>(f);` \
> in `CMakeCommitChangesJob::makeChanges()`; as an additional protection that \
> function watches the `errno` variable for errors occurring during the `addPath()` \
> operation; if an error is signalled, `folder` is not obtained by dynamic casting \
> but by initialision a new `CMakeFolderItem`. `makeChanges()` also contains a few \
> ASSERT statements that should remain effective even when building with \
> `QT_NO_DEBUG`; the patch takes care of that as well.
> I'm presenting this mostly for reference, in hope it may be useful for Kdevelop/KF5 \
> too. In a real implementation the maximum number of patch watchers would of course \
> be configurable.
>
> Diffs
> -----
>
> projectmanagers/cmake/cmakecommitchangesjob.cpp 7ce24fb
> projectmanagers/cmake/cmakemanager.h 19fc0c1
> projectmanagers/cmake/cmakemanager.cpp 2caecdb
>
> Diff: https://git.reviewboard.kde.org/r/125123/diff/
>
>
> Testing
> -------
>
> On OS X 10.9.5 with kdelibs 4.14.11 and Qt 4.8.7 .
> m_MaxAllowedWatchedPaths is set to a value that is reasonably close to the limit \
> where errors start occurring, and large enough to allow loading for instance \
> kdepimlibs, kdepim and kdepim-runtime in a single session.
>
> Thanks,
>
> René J.V. Bertin
>
>
--===============1508090619274995610==
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="12" style="border: 1px #c9c399 \
solid; border-radius: 6px; -moz-border-radius: 6px; -webkit-border-radius: 6px;"> \
<tr> <td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/125123/">https://git.reviewboard.kde.org/r/125123/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On September 10th, 2015, 2 p.m. UTC, <b>Milian \
Wolff</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px solid \
#d0d0d0; padding-left: 10px;"> <pre style="white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">where does it crash? we get the warnings even on linux \
but it never crashes. are there parts of this patch you want us to include or \
not?</p></pre> </blockquote>
<p>On September 10th, 2015, 2:27 p.m. UTC, <b>René J.V. Bertin</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">It \
crashes in the dynamic_cast from <code style="text-rendering: inherit;color: \
#4444cc;padding: 0;white-space: normal;margin: 0;line-height: inherit;">f</code> to \
<code style="text-rendering: inherit;color: #4444cc;padding: 0;white-space: \
normal;margin: 0;line-height: inherit;">folder</code>, which is why I create a fully \
new folder instance when there was an error in adding the path. I don't know why it \
crashes nor exactly on what, but for now it hasn't crashed with this approach. For \
now I'm just putting this up for reference. I'm testing the patch on Linux too, and \
haven't even yet been able to get warnings. I was expecting the kqueue backend to be \
used on Linux too, but apparently another one is used there, which uses less file \
descriptors (or none at all?).</p></pre> </blockquote>
</blockquote>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">Crashing in a dynamic cast means the pointer is invalid, and non-null. This \
should not happen, and I don't see how that is related to what you touch here. Is \
there an uninitialized ptr being returned? Is it dangling? And inotify does use file \
descriptors for every watcher as well, and there is (by default) a low limit (was it \
1024?) for that in the kernel. Baloo and other apps like KDevelop need more, which is \
why people and distros regularly increase that limit now to a couple of ten \
thousands. And again - even if the limit is reached (I've seen that many times) it \
never crashed. Find out why it crashed (see below). This patch here looks like it's \
working around an issue, instead of fixing it.</p></pre> <br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On September 10th, 2015, 2 p.m. UTC, <b>Milian \
Wolff</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px solid \
#d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line227" \
style="color: black; font-weight: bold; text-decoration: \
underline;">projectmanagers/cmake/cmakecommitchangesjob.cpp</a> <span \
style="font-weight: normal;">
(Diff revision 2)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="4"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">void \
CMakeCommitChangesJob::start()</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td> <th \
bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid \
#C0C0C0;" align="right"><font size="2">227</font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">qFatal</span><span class="p">(</span> <span class="s">"In the wrong \
thread, %s:%d"</span><span class="p">,</span> <span \
class="n">__FILE__</span><span class="p">,</span> <span class="n">__LINE__</span> \
<span class="p">);</span></pre></td> </tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">wth? \
just use the assert above. if you hit that in a release build then you'll hit it in a \
debug build - and that should <em style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: normal;">not</em> happen. thus assertion is fine \
no need to add this for release only</p></pre> </blockquote>
<p>On September 10th, 2015, 2:27 p.m. UTC, <b>René J.V. Bertin</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">I \
didn't add these for nothing: the assert macros evaluate to noops in the release \
builds. The application is likely to crash in that case, of course, but in this case \
I find it better to abort it with an explicit message rather than crash in a random \
later location.</p></pre> </blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p \
style="padding: 0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">Right, and you'll hit these issues in a debug build and fix them there. We \
do not litter our code base with "oh snap what if this happened?" in release mode for \
a reason - it's slowing things down.</p></pre> <br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On September 10th, 2015, 2 p.m. UTC, <b>Milian \
Wolff</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: 2px solid \
#d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line233" \
style="color: black; font-weight: bold; text-decoration: \
underline;">projectmanagers/cmake/cmakecommitchangesjob.cpp</a> <span \
style="font-weight: normal;">
(Diff revision 2)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="4"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">void \
CMakeCommitChangesJob::start()</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#f0f0f0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2">224</font></th> <td bgcolor="#ffffff" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">m_manager</span><span class="o">-></span><span \
class="n">addWatcher</span><span class="p">(</span><span \
class="n">m_project</span><span class="p">,</span> <span class="n">m_path</span><span \
class="p">.</span><span class="n">toLocalFile</span><span \
class="p">());</span></pre></td> <th bgcolor="#f0f0f0" style="border-left: 1px solid \
#C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font \
size="2">233</font></th> <td bgcolor="#ffffff" width="50%"><pre style="font-size: \
8pt; line-height: 140%; margin: 0; "> <span class="n">m_manager</span><span \
class="o">-></span><span class="n">addWatcher</span><span class="p">(</span><span \
class="n">m_project</span><span class="p">,</span> <span class="n">m_path</span><span \
class="p">.</span><span class="n">toLocalFile</span><span \
class="p">());</span></pre></td> </tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">in Qt \
5 you can simply use the return value of this function. Why do you keep working on Qt \
4 based KDevplatform on Mac - I really don't get it... You are making your life \
unneccessarily complicated.</p></pre> </blockquote>
<p>On September 10th, 2015, 2:27 p.m. UTC, <b>René J.V. Bertin</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">It's \
really not hard to get once you remember that KF5 doesn't work on OS X. Parts of it \
build in CI, but that's about it. I've more or less given up hope that this situation \
will approve anytime soon.</p></pre> </blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; \
white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p \
style="padding: 0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">Your valuable OS X time would be much better spent on making KF5 work, than \
fixing random things like this one on Qt 4 / KDE4 / KDevelop 4.</p></pre> <br />
<p>- Milian</p>
<br />
<p>On September 9th, 2015, 8:18 p.m. UTC, René J.V. Bertin wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="12" style="border: \
1px #888a85 solid; border-radius: 6px; -moz-border-radius: 6px; \
-webkit-border-radius: 6px;"> <tr>
<td>
<div>Review request for KDE Software on Mac OS X and KDevelop.</div>
<div>By René J.V. Bertin.</div>
<p style="color: grey;"><i>Updated Sept. 9, 2015, 8:18 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kdevelop
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">This is a confirmed bug in the 4.7 branch (and \
possibly in the KF5 version) where the cmake project importer can add (many) more \
paths to QFileSystemWatcher objects than the maximum supported number of open files \
(https://bugs.kde.org/show_bug.cgi?id=341551). On OS X at least this leads almost \
inevitably to a crash.</p> <p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">This patch prevents this by keeping \
track of the number of added paths, and by not adding more than a preconfigured \
maximum (determined empirically in this draft implementation). The crash occurs in \
<code style="text-rendering: inherit;color: #4444cc;padding: 0;white-space: \
normal;margin: 0;line-height: inherit;">folder = \
dynamic_cast<CMakeFolderItem*>(f);</code> in <code style="text-rendering: \
inherit;color: #4444cc;padding: 0;white-space: normal;margin: 0;line-height: \
inherit;">CMakeCommitChangesJob::makeChanges()</code>; as an additional protection \
that function watches the <code style="text-rendering: inherit;color: \
#4444cc;padding: 0;white-space: normal;margin: 0;line-height: inherit;">errno</code> \
variable for errors occurring during the <code style="text-rendering: inherit;color: \
#4444cc;padding: 0;white-space: normal;margin: 0;line-height: \
inherit;">addPath()</code> operation; if an error is signalled, <code \
style="text-rendering: inherit;color: #4444cc;padding: 0;white-space: normal;margin: \
0;line-height: inherit;">folder</code> is not obtained by dynamic casting but by \
initialision a new <code style="text-rendering: inherit;color: #4444cc;padd ing: \
0;white-space: normal;margin: 0;line-height: inherit;">CMakeFolderItem</code>. <code \
style="text-rendering: inherit;color: #4444cc;padding: 0;white-space: normal;margin: \
0;line-height: inherit;">makeChanges()</code> also contains a few ASSERT statements \
that should remain effective even when building with <code style="text-rendering: \
inherit;color: #4444cc;padding: 0;white-space: normal;margin: 0;line-height: \
inherit;">QT_NO_DEBUG</code>; the patch takes care of that as well.</p> <p \
style="padding: 0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">I'm presenting this mostly for reference, in hope it may be useful for \
Kdevelop/KF5 too. In a real implementation the maximum number of patch watchers would \
of course be configurable.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">On OS X 10.9.5 with kdelibs 4.14.11 and Qt 4.8.7 . \
m_MaxAllowedWatchedPaths is set to a value that is reasonably close to the limit \
where errors start occurring, and large enough to allow loading for instance \
kdepimlibs, kdepim and kdepim-runtime in a single session.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>projectmanagers/cmake/cmakecommitchangesjob.cpp <span style="color: \
grey">(7ce24fb)</span></li>
<li>projectmanagers/cmake/cmakemanager.h <span style="color: \
grey">(19fc0c1)</span></li>
<li>projectmanagers/cmake/cmakemanager.cpp <span style="color: \
grey">(2caecdb)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/125123/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
--===============1508090619274995610==--
[Attachment #3 (text/plain)]
_______________________________________________
kde-mac@kde.org
List Information: https://mail.kde.org/mailman/listinfo/kde-mac
KDE/Mac Information: http://community.kde.org/Mac
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic