[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-frameworks-devel
Subject: Re: Review Request 125762: External extractor plugin support for KFileMetaData
From: "David Faure" <faure () kde ! org>
Date: 2015-12-06 10:43:59
Message-ID: 20151206104359.29696.23137 () mimi ! kde ! org
[Download RAW message or body]
--===============3136905585056649544==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/125762/#review89164
-----------------------------------------------------------
Ship it!
another round of low-level code review; but I'd say you can push afterwards, given \
the lack of objections on the overall mechanism.
src/extractors/externalextractor.cpp (line 85)
<https://git.reviewboard.kde.org/r/125762/#comment61010>
m_extractors(), would do the same, no need to copy from a default-constructed \
instance.
src/extractors/externalextractor.cpp (line 86)
<https://git.reviewboard.kde.org/r/125762/#comment61009>
this is used only here in this method, so it doesn't need to be a member variable
src/extractors/externalextractor.cpp (line 91)
<https://git.reviewboard.kde.org/r/125762/#comment61012>
const QString &extractor
src/extractors/externalextractor.cpp (line 94)
<https://git.reviewboard.kde.org/r/125762/#comment61011>
maybe with a timeout? + error handling?
- David Faure
On Oct. 24, 2015, 12:19 p.m., Boudhayan Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/125762/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2015, 12:19 p.m.)
>
>
> Review request for Baloo, KDE Frameworks, Pinak Ahuja, and Vishesh Handa.
>
>
> Repository: kfilemetadata
>
>
> Description
> -------
>
> This patch introduces support for external metadata extractors in KFileMetaData
>
> The external extractors themselves can be written in any language, provided that it \
> can be executed as a standalone executable (compiled or script with a hashbang), \
> with command line arguments, and can output data to stdout.
> The extractors are executed like so:
>
> * `extractor --mimetypes` - outputs a list of mimetypes supported by the extractor, \
> one per line.
> * `extractor filename` - outputs a json document with the metadata. The keys are \
> such that they can be directly used with PropertyInfo::fromName().
> At the KFileMetaData end, an additional internal plugin (ExternalExtractor) is \
> provided that forms a conduit between external extractors and the internal API. \
> This plugin looks for executables called kfilemetadata_extractor_<something> in \
> /usr/bin to find external extractors, and executes them with the --mimetypes arg to \
> find the list of mimetypes each extractor supports. ExternalExtractor then claims \
> to support all of these mimetypes, and then delegates to the extractor executable \
> when doing the actual extraction.
>
> Diffs
> -----
>
> README.md 19b1a26
> src/extractors/CMakeLists.txt 5dd223e
> src/extractors/externalextractor.h PRE-CREATION
> src/extractors/externalextractor.cpp PRE-CREATION
>
> Diff: https://git.reviewboard.kde.org/r/125762/diff/
>
>
> Testing
> -------
>
> Tested with the sample executable file extractor (as attched, written in python) \
> with the dump manual test in KFileMetaData. Works.
>
> File Attachments
> ----------------
>
> kfilemetadata_extractor_executable
> https://git.reviewboard.kde.org/media/uploaded/files/2015/10/23/146b657f-31d9-4117-a82f-ef966a6339d4__kfilemetadata_extractor_executable
>
>
> Thanks,
>
> Boudhayan Gupta
>
>
--===============3136905585056649544==
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="12" style="border: 1px #c9c399 \
solid; border-radius: 6px; -moz-border-radius: 6px; -webkit-border-radius: 6px;"> \
<tr> <td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/125762/">https://git.reviewboard.kde.org/r/125762/</a>
</td>
</tr>
</table>
<br />
<p>Ship it!</p>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">another round of low-level code review; but I'd say you can push \
afterwards, given the lack of objections on the overall mechanism.</p></pre> <br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125762/diff/4/?file=412432#file412432line85" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/extractors/externalextractor.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 4)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td> <th \
bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid \
#C0C0C0;" align="right"><font size="2">85</font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">m_extractors</span><span class="p">(</span><span \
class="n">QMultiHash</span><span class="o"><</span><span \
class="n">QString</span><span class="p">,</span> <span class="n">QString</span><span \
class="o">></span><span class="p">()),</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: \
inherit;">m_extractors(), would do the same, no need to copy from a \
default-constructed instance.</p></pre> </div>
</div>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125762/diff/4/?file=412432#file412432line86" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/extractors/externalextractor.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 4)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td> <th \
bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid \
#C0C0C0;" align="right"><font size="2">86</font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">m_textCodec</span><span class="p">(</span><span \
class="n">QTextCodec</span><span class="o">::</span><span \
class="n">codecForLocale</span><span class="p">())</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">this \
is used only here in this method, so it doesn't need to be a member \
variable</p></pre> </div>
</div>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125762/diff/4/?file=412432#file412432line91" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/extractors/externalextractor.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 4)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td> <th \
bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid \
#C0C0C0;" align="right"><font size="2">91</font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">Q_FOREACH</span><span class="p">(</span><span class="k">auto</span> <span \
class="n">extractor</span><span class="p">,</span> <span \
class="n">findExtractors</span><span class="p">())</span> <span \
class="p">{</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">const \
QString &extractor</p></pre> </div>
</div>
<br />
<div>
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; \
border-collapse: collapse; margin: 2px padding: 2px;"> <thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; \
font-size: 9pt; padding: 4px 8px; text-align: left;"> <a \
href="https://git.reviewboard.kde.org/r/125762/diff/4/?file=412432#file412432line94" \
style="color: black; font-weight: bold; text-decoration: \
underline;">src/extractors/externalextractor.cpp</a> <span style="font-weight: \
normal;">
(Diff revision 4)
</span>
</th>
</tr>
</thead>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" \
align="right"><font size="2"></font></th> <td bgcolor="#c5ffc4" width="50%"><pre \
style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td> <th \
bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid \
#C0C0C0;" align="right"><font size="2">94</font></th> <td bgcolor="#c5ffc4" \
width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span \
class="n">process</span><span class="p">.</span><span \
class="n">waitForFinished</span><span class="p">();</span></pre></td> </tr>
</tbody>
</table>
<div style="margin-left: 2em;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;"><p style="padding: \
0;text-rendering: inherit;margin: 0;line-height: inherit;white-space: inherit;">maybe \
with a timeout? + error handling?</p></pre> </div>
</div>
<br />
<p>- David Faure</p>
<br />
<p>On October 24th, 2015, 12:19 p.m. UTC, Boudhayan Gupta wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="12" style="border: \
1px #888a85 solid; border-radius: 6px; -moz-border-radius: 6px; \
-webkit-border-radius: 6px;"> <tr>
<td>
<div>Review request for Baloo, KDE Frameworks, Pinak Ahuja, and Vishesh Handa.</div>
<div>By Boudhayan Gupta.</div>
<p style="color: grey;"><i>Updated Oct. 24, 2015, 12:19 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kfilemetadata
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">This patch introduces support for external metadata \
extractors in KFileMetaData</p> <p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">The external extractors themselves can \
be written in any language, provided that it can be executed as a standalone \
executable (compiled or script with a hashbang), with command line arguments, and can \
output data to stdout.</p> <p style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: inherit;">The extractors are executed like \
so:</p> <ul style="padding: 0;text-rendering: inherit;margin: 0 0 0 1em;line-height: \
inherit;white-space: normal;"> <li style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: normal;"><code style="text-rendering: \
inherit;color: #4444cc;padding: 0;white-space: normal;margin: 0;line-height: \
inherit;">extractor --mimetypes</code> - outputs a list of mimetypes supported by the \
extractor, one per line.</li> <li style="padding: 0;text-rendering: inherit;margin: \
0;line-height: inherit;white-space: normal;"><code style="text-rendering: \
inherit;color: #4444cc;padding: 0;white-space: normal;margin: 0;line-height: \
inherit;">extractor filename</code> - outputs a json document with the metadata. The \
keys are such that they can be directly used with PropertyInfo::fromName().</li> \
</ul> <p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">At the KFileMetaData end, an additional internal \
plugin (ExternalExtractor) is provided that forms a conduit between external \
extractors and the internal API. This plugin looks for executables called \
kfilemetadata_extractor_<something> in /usr/bin to find external extractors, \
and executes them with the --mimetypes arg to find the list of mimetypes each \
extractor supports. ExternalExtractor then claims to support all of these mimetypes, \
and then delegates to the extractor executable when doing the actual \
extraction.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;"><p style="padding: 0;text-rendering: inherit;margin: 0;line-height: \
inherit;white-space: inherit;">Tested with the sample executable file extractor (as \
attched, written in python) with the dump manual test in KFileMetaData. \
Works.</p></pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>README.md <span style="color: grey">(19b1a26)</span></li>
<li>src/extractors/CMakeLists.txt <span style="color: grey">(5dd223e)</span></li>
<li>src/extractors/externalextractor.h <span style="color: \
grey">(PRE-CREATION)</span></li>
<li>src/extractors/externalextractor.cpp <span style="color: \
grey">(PRE-CREATION)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/125762/diff/" style="margin-left: \
3em;">View Diff</a></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">File Attachments \
</h1>
<li><a href="https://git.reviewboard.kde.org/media/uploaded/files/2015/10/23/146b657f \
-31d9-4117-a82f-ef966a6339d4__kfilemetadata_extractor_executable">kfilemetadata_extractor_executable</a></li>
</ul>
</td>
</tr>
</table>
</div>
</body>
</html>
--===============3136905585056649544==--
[Attachment #3 (text/plain)]
_______________________________________________
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic