[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-devel
Subject: Re: Akonadi & Billions Of NNTP Headers
From: Rohan Beckles <rohan.beckles () gmail ! com>
Date: 2012-10-12 10:59:56
Message-ID: 225015572.FgINH0nStt () vostok
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Gael --
Thank you for your response. Yes, I'm aware of Kwooty - in fact I use it. :) \
Unfortunately, Kwooty depends on NZB websites/search engines, and those services \
don't index everything which is why I'd like to develop a more traditional NNTP \
robot which indexes the headers on the local machine. I can then search them, and \
queue items for download.
However, you're right about sharing code. I'll be looking at Kwooty, KNode and \
KLibido for inspiration - this will be my first KDE project. :) However, the \
pressing issue is whether Akonadi can handle a full header pull from a big binary \
newsgroup. We're looking at around 5 billion headers from the biggest group. :)
Best,
--
Rohan Beckles
On Friday 12 Oct 2012 09:17:52 Gaël Beaudoin wrote:
Le 12/10/2012 01:31, Rohan Beckles a écrit :
Hello --
This is my first ever post to any KDE mailing list, so please excuse any n00bidity. \
:)
I'm in the process of planning an NNTP binary download robot for KDE 4.x, something \
along the lines of NewsLeecher or Newsman Pro (which are Windows applications). My \
plan is to use the existing RFC822 Mail resource and the NNTP agent (if it exists, \
otherwise write one), and feed the NNTP headers into Akonadi, so that the headers \
can be indexed and searched.
Question: more than a few binary newsgroups have header counts reaching past \
hundreds of millions into billions. Is Akonadi (or more precisely its MySQL \
backend) capable of dealing with that many records, especially on consumer hardware? \
I'm assuming that the average desktop has 4GB RAM and a single 500GB hard disk.
Many thanks,
Hi Rohan,
Do you by any chance know kwooty ? http://kwooty.sourceforge.net/
It sounds like it does what you would like to do. I use it and it's maintained, \
simple, fast and reliable. Maybe you can help the project or share some code with \
them?
Have a nice day,
Gaël
[Attachment #5 (unknown)]
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" \
"http://www.w3.org/TR/REC-html40/strict.dtd"> <html><head><meta name="qrichtext" \
content="1" /><style type="text/css"> p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'Droid Sans'; font-size:10pt; \
font-weight:400; font-style:normal;"> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">Hi Gael --</p> <p style="-qt-paragraph-type:empty; margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; "> </p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">Thank you for your response. Yes, I'm aware of Kwooty - in fact I \
use it. :) Unfortunately, Kwooty depends on NZB </p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">websites/search engines, and those services don't \
index everything which is why I'd like to develop a more traditional </p> <p style=" \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">NNTP robot which indexes the \
headers on the local machine. I can then search them, and queue items for \
download.</p> <p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p> \
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">However, you're right about \
sharing code. I'll be looking at Kwooty, KNode and KLibido for inspiration - this \
will </p> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">be my first \
KDE project. :) However, the pressing issue is whether Akonadi can handle a full \
header pull from a big </p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">binary newsgroup. We're looking at around 5 billion headers from \
the biggest group. :)</p> <p style="-qt-paragraph-type:empty; margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; "> </p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">Best,</p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">-- </p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;">Rohan Beckles</p> <p style="-qt-paragraph-type:empty; \
margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; \
-qt-block-indent:0; text-indent:0px; "> </p> <p style=" margin-top:0px; \
margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;">On Friday 12 Oct 2012 09:17:52 Gaël Beaudoin \
wrote:<br /></p> <p style=" margin-top:12px; margin-bottom:0px; margin-left:40px; \
margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Le \
12/10/2012 01:31, Rohan Beckles a écrit :<br /></p> <pre style=" margin-top:12px; \
margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Courier \
New,courier';">Hello --</span></pre> <pre style="-qt-paragraph-type:empty; \
margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0; font-family:'Courier \
New,courier';"><br /></pre> <pre style=" margin-top:0px; margin-bottom:0px; \
margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;"><span style=" font-family:'Courier New,courier';">This is my first \
ever post to any KDE mailing list, so please excuse any n00bidity. :)</span></pre> \
<pre style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; \
margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0; font-family:'Courier New,courier';"><br /></pre> <pre style=" \
margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" \
font-family:'Courier New,courier';">I'm in the process of planning an NNTP binary \
download robot for KDE 4.x, something along the lines of NewsLeecher or Newsman Pro \
</span></pre> <pre style=" margin-top:0px; margin-bottom:0px; margin-left:80px; \
margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span \
style=" font-family:'Courier New,courier';">(which are Windows applications). My \
plan is to use the existing RFC822 Mail resource and the NNTP agent (if it exists, \
otherwise write one), </span></pre> <pre style=" margin-top:0px; margin-bottom:0px; \
margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;"><span style=" font-family:'Courier New,courier';">and feed the \
NNTP headers into Akonadi, so that the headers can be indexed and searched. \
</span></pre> <pre style="-qt-paragraph-type:empty; margin-top:0px; \
margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; \
text-indent:0px; -qt-user-state:0; font-family:'Courier New,courier';"><br /></pre> \
<pre style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" \
font-family:'Courier New,courier';">Question: more than a few binary newsgroups have \
header counts reaching past hundreds of millions into billions. Is Akonadi (or more \
</span></pre> <pre style=" margin-top:0px; margin-bottom:0px; margin-left:80px; \
margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span \
style=" font-family:'Courier New,courier';">precisely its MySQL backend) capable of \
dealing with that many records, especially on consumer hardware? I'm assuming that \
the average </span></pre> <pre style=" margin-top:0px; margin-bottom:0px; \
margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;"><span style=" font-family:'Courier New,courier';">desktop has 4GB \
RAM and a single 500GB hard disk. </span></pre> <pre style="-qt-paragraph-type:empty; \
margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; \
-qt-block-indent:0; text-indent:0px; -qt-user-state:0; font-family:'Courier \
New,courier';"><br /></pre> <pre style=" margin-top:0px; margin-bottom:12px; \
margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;"><span style=" font-family:'Courier New,courier';">Many \
thanks,</span> </pre> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; \
margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Hi \
Rohan,<br /><br />Do you by any chance know kwooty ? <a \
href="http://kwooty.sourceforge.net/"><span style=" text-decoration: underline; \
color:#0057ae;">http://kwooty.sourceforge.net/</span></a><br />It sounds like it does \
what you would like to do. I use it and it's maintained, simple, fast and reliable. \
Maybe you can help the project or share some code with them?<br /><br />Have a nice \
day,<br />Gaël<br /></p> <p style=" margin-top:0px; margin-bottom:0px; \
margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; \
-qt-user-state:0;"><br /><br /></p></body></html>
>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic