[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] SQLite vs. Oracle (parallelized)
From:       Allan Edwards <wallanedwards () gmail ! com>
Date:       2009-02-24 6:25:12
Message-ID: 7dad920902232225l1bedf88apa4afbd2560a8b8d3 () mail ! gmail ! com
[Download RAW message or body]

I wanted to throw out a few more points about being parallel and
scalable in terms of data storage?

What is the reason we want to be this way?  Well, for tons of users
and to mitigate risk across lots of machines.  Yes, out of the box the
value that Oracle provides is of a designation that satisfies these
ideals for large scale systems.  YET, as memory, hard drive space,
processors with more cores, etc.  come online, how could you get
Sqlite to be SQLiteIKickOraclesButt?

Well, I although I am pressed for time in development of a number of
systems I don't have time to try this out but what Sqlite would need
is a broker like server that could utilize it's existing features.
The broker server could provide the same features as an Oracle by
splitting inserts across multiple sqlites that hit multiple disks on
multiple systems.  The reality is that Sqlite has taken care of the
nasty details of organizing data into a binary file cross platform,
but you could literally take the database and write code above the
engine itself that leveraged it's capabilities in such a way that it
worked like a large scale database.

Back years ago I was the head architect on a C based system in
semiconductor.  We had a crazy requirement to bust out hundreds to
thousands of grey scale images from a camera source in a real time
manufacturing solution.  We had to store this data on the drive
quickly.  Before I took charge of the project we failed to get the
necessary performance needed from an RDBMS on our first project time
delivery limits.  On the second round we bench marked all system
components and figured out how we needed to architect a system from
every hardware dependency.  We ended up writing a buffer management
library that basically was optimized to pre allocate memory of
specific sizes for specific image sizes.  This pre allocation allowed
for the most optimal approach in moving data into memory.  We then had
writer threads that had thread synced access into that buffer and
eventually the images would get written to an RDBMS.

The reason I shot you the example above is to give you food for
thought on how you might still utilize the great power of Sqlite if
you don't need the massive cost or overhead of an Oracle.  Sqlite is a
beautiful thing due to it's simplicity.  It is the most powerful
database management solution on the planet for it's size (this is
TOTALLY OBJECTIVE).  Installing Oracle on a computer, being force fed
crappy performing and unrobust java (the enterprise Oracle manager
crashes on me with HUGE stack traces) too butt with Oracle makes for
one nasty user experience.  Yet with sexy Sqlite you can "xcopy"
install that puppy and go to town.  So in your thought on your desired
solutions, IT MIGHT be good to think of what Sqlite offers and
consider writing a separate library that could fulfill your extrea
RDBMS requirements so you can take advantage of the sweetness this
little data storage gym offers.

I have personally written a socket based server on top of the database
and it works very well.  So I have actually scaled the database myself
as I preached in this email.  Don't be a WIMP and map shares to share
a database... write a socket based beauty like YOURS TRULY! hehe  For
"most" solutions it is wonderful.  After years and building millions
of lines of code keeping the business delivery requirements fulfilled
in the most simple manner seems to be the best approach for me.  If
you are the same, stay agnostic to all solutions available and run up
a strategy that will give you the best of all worlds.  And yes, at
times you have to write a little EXTRA code to get there! : - )

Allan

P.S.  If somebody does decide to build enterprise Sqlite, I would love
to throw in my 2 cents on how to write the stuff on the outside to add
in the big dog features.  I was working out tonight and while thinking
about this I believe you could actually maintain the wonderfulness of
the core engine, then scale the library from an outside piece of code.
 Then you can keep integrity on both sides of the fence and not make
sqlite into sqlitetoomuch.

On Mon, Feb 23, 2009 at 9:49 PM,  <python@bdurham.com> wrote:
> P Kishor,
>
>>> Most computers these days are multi-core. ..
>
>> One of things easy to overlook is that SQLite is not a PC-exclusive
> software. About 10 million copies of SQLite run on iPhone. Who knows
> how many run on other handhelds, embedded platforms, Vxworks, the
> like. SQLite used to fit on a floppy and probably still does.
>
> Good point!
>
> I lost my perspective on SQLite's intended audience.
>
> Regards,
> Malcolm
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
W Allan Edwards
214-289-2959
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[prev in list] [next in list] [prev in thread] [next in thread]