From kde-edu-devel Thu Jan 30 00:36:57 2014 From: Vijay Dhameliya Date: Thu, 30 Jan 2014 00:36:57 +0000 To: kde-edu-devel Subject: Re: Replacing file-system by database in KStars Message-Id: X-MARC-Message: https://marc.info/?l=kde-edu-devel&m=139104223517336 MIME-Version: 1 Content-Type: multipart/mixed; boundary="--===============7135663922341076452==" --===============7135663922341076452== Content-Type: multipart/alternative; boundary=001a1133604445610904f12541d3 --001a1133604445610904f12541d3 Content-Type: text/plain; charset=ISO-8859-1 Hi Thank you David and John, I will implement database system for one big file which is used everywhere and do analysis to get measure of significance. Meanwhile I try to get Akash's opinion on the topic :) Regards, Vijay On Wed, Jan 29, 2014 at 11:27 AM, John Layt wrote: > On Tuesday 28 Jan 2014 08:22:08 Vijay Dhameliya wrote: > > Hi guys, > > > > Currently when KStars is launched, it reads data corresponding to > different > > Skyobject from respective file in loaddata() methods. And I have tracked > > out all the classes where we are loading data by reading file. > > > > I researched bit on the topic and I found that loading data from database > > is always much better option then doing same from file. > > > > If we replace file system with QSql following are the Pros: > > > > 1) We will not have to ship so many files with Kstars > > 2) Loading from database is quicker than doing same from file > > 3) Code for load methods will be reduced in size > > > > Cons: > > 1) I will have to move all data from files into database by temporary > > methods > > > > So I am planning to start coding to replace file system by database on my > > local branch. > > > > Can you please give your views and suggestion regarding the same ? I am > > sure that It will be very helpful to me. :) > > There's two areas you need to give serious thought to: > > 1) What database software? If a server like MySql, be warned that you will > get a lot of people complaining about needing to run an entire database > server > just to use KStars (see the complaints we regularly have/had over in PIM > and > Amarok), or demanding you use their preferred server instead. If embedded > like SQLite then will this give you the performance improvements you're > looking for? > > 2) What are the sources of the data files and how often are they updated? > If > these are data files updated regularly by an external provider, and perhaps > distributed through KGetHotNewStuff then you will still need code to load > the > new data and merge it into the old data in the database, so no real > reduction > in code required, but a whole increased level of complexity from the data > sync. You also need to think of the initial download size of the database > file and how that will affect distros, who can currently split the data > files > up in separate packages. And you need to check licensing, some of the > files > we use have to be downloaded by the users as we are not allowed to > distribute > them, so again you would still need import code for those. > > The option of having a single database that you just load what data you > need > for the view is good in theory, but the background management of that data > often ends up not being worth the gains, and often the gains fail to work > out. > It could be worth a limited experiment though to investigate and benchmark > the > potential gains. If you do, pick the one big important file used > everywhere, > load it to one table, and benchmark any gains or losses. Don't assume that > just because a textbook or website says it will be faster that it actually > will be, you need to prove it. > > I'm not sure who the current maintainer is, but best check with them before > doing anything too radical :-) Akarsh Simha would be a good person to get > an > opinion from, as would the Marble team who regularly deal with large data > files. > > John. > > _______________________________________________ > kde-edu mailing list > kde-edu@mail.kde.org > https://mail.kde.org/mailman/listinfo/kde-edu > --001a1133604445610904f12541d3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi

Thank you David and John, I will imp= lement database system for one big file which is used everywhere and do ana= lysis to get measure of significance. Meanwhile I try to get Akash's op= inion on the topic :)=A0

Regards,
Vijay


On Wed, Jan 29, 2014 at 11:27 AM= , John Layt <jlayt@kde.org> wrote:
On T= uesday 28 Jan 2014 08:22:08 Vijay Dhameliya wrote:
> Hi guys,
>
> Currently when KStars is launched, it reads data corresponding to diff= erent
> Skyobject from respective file in loaddata() methods. And I have track= ed
> out all the classes where we are loading data by reading file.
>
> I researched bit on the topic and I found that loading data from datab= ase
> is always much better option then doing same from file.
>
> If we replace file system with QSql following are the Pros:
>
> 1) We will not have to ship so many files with Kstars
> 2) Loading from database is quicker than doing same from file
> 3) Code for load methods will be reduced in size
>
> Cons:
> 1) I will have to move all data from files into database by temporary<= br> > methods
>
> So I am planning to start coding to replace file system by database on= my
> local branch.
>
> Can you please give your views and suggestion regarding the same ? I a= m
> sure that It will be very helpful to me. :)

There's two areas you need to give serious thought to:

1) What database software? =A0If a server like MySql, be warned that you wi= ll
get a lot of people complaining about needing to run an entire database ser= ver
just to use KStars (see the complaints we regularly have/had over in PIM an= d
Amarok), or demanding you use their preferred server instead. =A0If embedde= d
like SQLite then will this give you the performance improvements you're=
looking for?

2) What are the sources of the data files and how often are they updated? = =A0If
these are data files updated regularly by an external provider, and perhaps=
distributed through KGetHotNewStuff then you will still need code to load t= he
new data and merge it into the old data in the database, so no real reducti= on
in code required, but a whole increased level of complexity from the data sync. =A0You also need to think of the initial download size of the databas= e
file and how that will affect distros, who can currently split the data fil= es
up in separate packages. =A0And you need to check licensing, some of the fi= les
we use have to be downloaded by the users as we are not allowed to distribu= te
them, so again you would still need import code for those.

The option of having a single database that you just load what data you nee= d
for the view is good in theory, but the background management of that data<= br> often ends up not being worth the gains, and often the gains fail to work o= ut.
It could be worth a limited experiment though to investigate and benchmark = the
potential gains. =A0If you do, pick the one big important file used everywh= ere,
load it to one table, and benchmark any gains or losses. =A0Don't assum= e that
just because a textbook or website says it will be faster that it actually<= br> will be, you need to prove it.

I'm not sure who the current maintainer is, but best check with them be= fore
doing anything too radical :-) =A0Akarsh Simha would be a good person to ge= t an
opinion from, as would the Marble team who regularly deal with large data files.

John.

_______________________________________________
kde-edu mailing list
kde-edu@mail.kde.org
https://mail.kde.org/mailman/listinfo/kde-edu

--001a1133604445610904f12541d3-- --===============7135663922341076452== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ kde-edu mailing list kde-edu@mail.kde.org https://mail.kde.org/mailman/listinfo/kde-edu --===============7135663922341076452==--