[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] SQLite vs. Oracle (parallelized)
From:       python () bdurham ! com
Date:       2009-02-27 20:35:50
Message-ID: 1235766950.2223.1302787537 () webmail ! messagingengine ! com
[Download RAW message or body]

Allan,

Thanks for your reply. I'm new to SQLite, but have used a similar list
of databases (with an emphasis on server vs. client side databases)
professionally for years. My background is designing and building
enterprise BI, ETL, and data warehouse systems using databases like
Oracle, DB2, SQL Server and ETL tools like Informatica, Ab Initio, BO
Data Integrator, etc.

My goal is to be able to offer our customers cost effective, high
performance alternatives to traditional commercial ETL solutions. We
recently re-implemented a mid-sized Oracle/Informatica based ETL system
in Python using in memory data structures (dicts) and improved the
performance by a factor of 20x. We're now looking to expand this
approach to SQLite for environments where the volume of data is greater
than available memory.

> When you talk about performance comparisons your question really needs to be squared up to what the \
> database is used for. Are you on a server, workstation, or embedded? How large in the database? How are \
> the internal algos of the database engine lined up? What is the memory footprint? How did you configure \
> what is configurable in the database?

I'm interested in exploring whether or not SQLite can be used as an ETL
tool with large data sets (~80+ Gb). In this context, SQLite would be
run on 64-bit Intel servers with lots of RAM (16-64 Gb). The data would
be stored/processed on server local SCSI drives vs. located on a SAN.
File access would be via a single process per SQLite database. The
interface language would most likely be Python.

> The reality is I BET and I would love to know myself Sqlite is just as fast on inserts to the same hard \
> drive as oracle.

I would love to see such a benchmark as well.

> Are you some kind of political Oracle covering biggot?

LOL. No. I try to be as database agnostic as possible.

> Or, do you want another chance to post something that states you are trying to find the best data \
> storage solution to deliver sincere value to your client in terms of a database choice?

Subtle point here ... I'm not looking for a long term data storage
solution - I'm exploring the possibility of using SQLite as a way to
validate, transform, and pre-aggregate raw data that would in turn be
exported to raw text files and imported (in final form) to a customer's
data warehouse.

> Sometimes that choice is Sqlite, but in some cases it is Oracle instead.

Agreed.

Regards,
Malcolm
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[prev in list] [next in list] [prev in thread] [next in thread]