[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: Review Request 114632: Improve pdf title extraction
From: "Luis Silva" <lacsilva () gmail ! com>
Date: 2014-01-06 17:46:41
Message-ID: 20140106174641.19832.23138 () probe ! kde ! org
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> On Dec. 26, 2013, 1:57 a.m., Christoph Feck wrote:
> > Hm, you broke the comment :)
>
> Luis Silva wrote:
> What do you mean? It all works fine here.
>
> Christoph Feck wrote:
> Yes, because the compiler does not read comments.
>
> Thomas Lübking wrote:
> Aside this, the approach seems too naive?
> DOIs have a defined structure, leading "doi: 10" (ignoring the case and making \
> colon and whitespace optional) and in general the "problematic" tokens will have a \
> massive digit overhead - so this could be used as additional test ( < 25 && \
> looksLikeIndex())
@Christoph: Just (finally) understood what you meant with "breaking the comment". I \
uploaded a new patch that (hopefully) fixes the issue in the correct way. @Thomas: \
The approach was meant to be naive. In this simple form, this patch takes care of all \
index-like cases as well as most other short garbage titles without further parsing. \
What would be the point of actually knowing if a very short title was actually a doi \
or an index?
- Luis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/114632/#review46156
-----------------------------------------------------------
On Dec. 23, 2013, 4:14 p.m., Luis Silva wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/114632/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2013, 4:14 p.m.)
>
>
> Review request for Baloo and Vishesh Handa.
>
>
> Repository: kfilemetadata
>
>
> Description
> -------
>
> A good portion of scientific papers in my collection had a doi or an index number \
> in the title. These are in general short string chains, shorter than the real \
> title. I improve extraction of titles from pdf's by setting a minimum size below \
> which parsing of the first page is forced. The cut-off size is arbitrarily set to \
> 25 characters (three "big words").
>
> Diffs
> -----
>
> src/extractors/popplerextractor.cpp b056581f51d10b632799586eed3cc15ac539fe80
>
> Diff: https://git.reviewboard.kde.org/r/114632/diff/
>
>
> Testing
> -------
>
> This improved the title extraction on my pdf collection of scientific papers by \
> quite a lot.
>
> Thanks,
>
> Luis Silva
>
>
[Attachment #5 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://git.reviewboard.kde.org/r/114632/">https://git.reviewboard.kde.org/r/114632/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <p style="margin-top: 0;">On December 26th, 2013, 1:57 a.m. UTC, \
<b>Christoph Feck</b> wrote:</p> <blockquote style="margin-left: 1em; border-left: \
2px solid #d0d0d0; padding-left: 10px;"> <pre style="white-space: pre-wrap; \
white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; \
word-wrap: break-word;">Hm, you broke the comment :)</pre> </blockquote>
<p>On January 6th, 2014, 3:24 p.m. UTC, <b>Luis Silva</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">What do you mean? It all \
works fine here. </pre> </blockquote>
<p>On January 6th, 2014, 3:50 p.m. UTC, <b>Christoph Feck</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Yes, because the \
compiler does not read comments.</pre> </blockquote>
<p>On January 6th, 2014, 4:11 p.m. UTC, <b>Thomas Lübking</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: \
10px;"> <pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Aside this, the approach \
seems too naive? DOIs have a defined structure, leading "doi: 10" (ignoring \
the case and making colon and whitespace optional) and in general the \
"problematic" tokens will have a massive digit overhead - so this could be \
used as additional test ( < 25 && looksLikeIndex())</pre> </blockquote>
</blockquote>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: \
-pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">@Christoph: Just \
(finally) understood what you meant with "breaking the comment". I uploaded \
a new patch that (hopefully) fixes the issue in the correct way. @Thomas: The \
approach was meant to be naive. In this simple form, this patch takes care of all \
index-like cases as well as most other short garbage titles without further parsing. \
What would be the point of actually knowing if a very short title was actually a doi \
or an index?</pre> <br />
<p>- Luis</p>
<br />
<p>On December 23rd, 2013, 4:14 p.m. UTC, Luis Silva wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('https://git.reviewboard.kde.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Baloo and Vishesh Handa.</div>
<div>By Luis Silva.</div>
<p style="color: grey;"><i>Updated Dec. 23, 2013, 4:14 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
kfilemetadata
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">A good portion of scientific papers in my collection had a doi or an \
index number in the title. These are in general short string chains, shorter than the \
real title. I improve extraction of titles from pdf's by setting a minimum size \
below which parsing of the first page is forced. The cut-off size is arbitrarily set \
to 25 characters (three "big words"). </pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">This improved the title extraction on my pdf collection of scientific \
papers by quite a lot.</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>src/extractors/popplerextractor.cpp <span style="color: \
grey">(b056581f51d10b632799586eed3cc15ac539fe80)</span></li>
</ul>
<p><a href="https://git.reviewboard.kde.org/r/114632/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic