[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-edu-devel
Subject: Re: Replacing file-system by database in KStars
From: Vijay Dhameliya <vijay.atwork13 () gmail ! com>
Date: 2014-01-30 0:36:57
Message-ID: CAH-KMQikO2bf55terhoHAUkefC4qXP4FeHYA50GdxLSW0XUNxQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi
Thank you David and John, I will implement database system for one big file
which is used everywhere and do analysis to get measure of significance.
Meanwhile I try to get Akash's opinion on the topic :)
Regards,
Vijay
On Wed, Jan 29, 2014 at 11:27 AM, John Layt <jlayt@kde.org> wrote:
> On Tuesday 28 Jan 2014 08:22:08 Vijay Dhameliya wrote:
> > Hi guys,
> >
> > Currently when KStars is launched, it reads data corresponding to
> different
> > Skyobject from respective file in loaddata() methods. And I have tracked
> > out all the classes where we are loading data by reading file.
> >
> > I researched bit on the topic and I found that loading data from database
> > is always much better option then doing same from file.
> >
> > If we replace file system with QSql following are the Pros:
> >
> > 1) We will not have to ship so many files with Kstars
> > 2) Loading from database is quicker than doing same from file
> > 3) Code for load methods will be reduced in size
> >
> > Cons:
> > 1) I will have to move all data from files into database by temporary
> > methods
> >
> > So I am planning to start coding to replace file system by database on my
> > local branch.
> >
> > Can you please give your views and suggestion regarding the same ? I am
> > sure that It will be very helpful to me. :)
>
> There's two areas you need to give serious thought to:
>
> 1) What database software? If a server like MySql, be warned that you will
> get a lot of people complaining about needing to run an entire database
> server
> just to use KStars (see the complaints we regularly have/had over in PIM
> and
> Amarok), or demanding you use their preferred server instead. If embedded
> like SQLite then will this give you the performance improvements you're
> looking for?
>
> 2) What are the sources of the data files and how often are they updated?
> If
> these are data files updated regularly by an external provider, and perhaps
> distributed through KGetHotNewStuff then you will still need code to load
> the
> new data and merge it into the old data in the database, so no real
> reduction
> in code required, but a whole increased level of complexity from the data
> sync. You also need to think of the initial download size of the database
> file and how that will affect distros, who can currently split the data
> files
> up in separate packages. And you need to check licensing, some of the
> files
> we use have to be downloaded by the users as we are not allowed to
> distribute
> them, so again you would still need import code for those.
>
> The option of having a single database that you just load what data you
> need
> for the view is good in theory, but the background management of that data
> often ends up not being worth the gains, and often the gains fail to work
> out.
> It could be worth a limited experiment though to investigate and benchmark
> the
> potential gains. If you do, pick the one big important file used
> everywhere,
> load it to one table, and benchmark any gains or losses. Don't assume that
> just because a textbook or website says it will be faster that it actually
> will be, you need to prove it.
>
> I'm not sure who the current maintainer is, but best check with them before
> doing anything too radical :-) Akarsh Simha would be a good person to get
> an
> opinion from, as would the Marble team who regularly deal with large data
> files.
>
> John.
>
> _______________________________________________
> kde-edu mailing list
> kde-edu@mail.kde.org
> https://mail.kde.org/mailman/listinfo/kde-edu
>
[Attachment #5 (text/html)]
<div dir="ltr">Hi<div><br></div><div>Thank you David and John, I will implement \
database system for one big file which is used everywhere and do analysis to get \
measure of significance. Meanwhile I try to get Akash's opinion on the topic :) \
</div> <div><br></div><div>Regards,</div><div>Vijay</div></div><div \
class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 29, 2014 at 11:27 \
AM, John Layt <span dir="ltr"><<a href="mailto:jlayt@kde.org" \
target="_blank">jlayt@kde.org</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tuesday 28 Jan 2014 \
08:22:08 Vijay Dhameliya wrote:<br> > Hi guys,<br>
><br>
> Currently when KStars is launched, it reads data corresponding to different<br>
> Skyobject from respective file in loaddata() methods. And I have tracked<br>
> out all the classes where we are loading data by reading file.<br>
><br>
> I researched bit on the topic and I found that loading data from database<br>
> is always much better option then doing same from file.<br>
><br>
> If we replace file system with QSql following are the Pros:<br>
><br>
> 1) We will not have to ship so many files with Kstars<br>
> 2) Loading from database is quicker than doing same from file<br>
> 3) Code for load methods will be reduced in size<br>
><br>
> Cons:<br>
> 1) I will have to move all data from files into database by temporary<br>
> methods<br>
><br>
> So I am planning to start coding to replace file system by database on my<br>
> local branch.<br>
><br>
> Can you please give your views and suggestion regarding the same ? I am<br>
> sure that It will be very helpful to me. :)<br>
<br>
</div></div>There's two areas you need to give serious thought to:<br>
<br>
1) What database software? If a server like MySql, be warned that you will<br>
get a lot of people complaining about needing to run an entire database server<br>
just to use KStars (see the complaints we regularly have/had over in PIM and<br>
Amarok), or demanding you use their preferred server instead. If embedded<br>
like SQLite then will this give you the performance improvements you're<br>
looking for?<br>
<br>
2) What are the sources of the data files and how often are they updated? If<br>
these are data files updated regularly by an external provider, and perhaps<br>
distributed through KGetHotNewStuff then you will still need code to load the<br>
new data and merge it into the old data in the database, so no real reduction<br>
in code required, but a whole increased level of complexity from the data<br>
sync. You also need to think of the initial download size of the database<br>
file and how that will affect distros, who can currently split the data files<br>
up in separate packages. And you need to check licensing, some of the files<br>
we use have to be downloaded by the users as we are not allowed to distribute<br>
them, so again you would still need import code for those.<br>
<br>
The option of having a single database that you just load what data you need<br>
for the view is good in theory, but the background management of that data<br>
often ends up not being worth the gains, and often the gains fail to work out.<br>
It could be worth a limited experiment though to investigate and benchmark the<br>
potential gains. If you do, pick the one big important file used everywhere,<br>
load it to one table, and benchmark any gains or losses. Don't assume that<br>
just because a textbook or website says it will be faster that it actually<br>
will be, you need to prove it.<br>
<br>
I'm not sure who the current maintainer is, but best check with them before<br>
doing anything too radical :-) Akarsh Simha would be a good person to get an<br>
opinion from, as would the Marble team who regularly deal with large data<br>
files.<br>
<span class="HOEnZb"><font color="#888888"><br>
John.<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
kde-edu mailing list<br>
<a href="mailto:kde-edu@mail.kde.org">kde-edu@mail.kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/kde-edu" \
target="_blank">https://mail.kde.org/mailman/listinfo/kde-edu</a><br> \
</div></div></blockquote></div><br></div>
_______________________________________________
kde-edu mailing list
kde-edu@mail.kde.org
https://mail.kde.org/mailman/listinfo/kde-edu
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic