[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] SQLite vs. Oracle (parallelized)
From:       Alexey Pechnikov <pechnikov () mobigroup ! ru>
Date:       2009-02-28 8:59:59
Message-ID: 200902281200.00514.pechnikov () mobigroup ! ru
[Download RAW message or body]

Hello!

On Saturday 28 February 2009 00:31:37 python@bdurham.com wrote:
> Your approach of splitting large data sets sounds similar to what other
> SQLite users with large data sets seem to be doing. At a high level,
> this sounds like how one would partition data using Oracle? I'm going to
> start a new thread on this topic.

Oracle manage partitions automatically but SQLite databases are manages by 
your application. So you must write more code. But good result is that SQLite 
databases may be packed and removed to archive and your application can easy 
resolv this situation. And a lot of external programs (as example demon for 
network monitoring or tacacs/radius server) can store data in single databases 
without working with full dataset. And Oracle is using some buffers for modified 
data and can perform a lot of write/read transactions in parallels. SQLIte-
driven application must plan write operation and use some hashes/in-memory 
databases as buffers and perform saves sets of records to on-disk database 
because write operation with single record is not effective. So I have tcl data 
collectors which regular save chunks of data to on-disk SQLite database. Of 
cource users operations by web-interface performs immediately and it's work 
fine because in web-applications there are a lot of read operations and a few 
write operations.

> > With your hardware I think 100Gb dataset is not limit.
>
> Good news. I'm looking forward to verifying this over the next month or
> so.

Please mail to list or to me directly about you problems and results. I do 
some big systems based on SQLite now and the information about similar 
projects is important for me. 

I support debian repository with my own SQLite build with extra modules. For 
big projects is needed some additional functions (data compression, md5 
functions, extended csv import, etc.).

Debian etch: 
deb http://mobigroup.ru/debian/ etch main contrib non-free
deb-src http://mobigroup.ru/debian/ etch main contrib non-free

Debian lenny: 
deb http://mobigroup.ru/debian/ lenny main contrib non-free
deb-src http://mobigroup.ru/debian/ lenny main contrib non-free

Keyring may be installed as
sudo aptitude install debian-mobigroup-keyring

Best regards.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[prev in list] [next in list] [prev in thread] [next in thread]