[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] Re: Regarding sqlite3_exec
From:       "Isaac Raway" <isaac.raway () gmail ! com>
Date:       2006-10-28 21:20:36
Message-ID: d671ff3c0610281420y2740ab99r4af428708b66d126 () mail ! gmail ! com
[Download RAW message or body]

You cannot even consider loading even a thousand records directly.

Get a set of ID numbers. Load each record as it's needed. This is very
basic stuff, and not even that hard to implement. I am just saying for
the record that this is not hard to do, hopefully no one else will be
scared away from the concept.

As a general rule of coding, do it right the first time. One thread
reading a list of integers while another is on the GUI is not complex.

Isaac

On 10/28/06, Da Martian <da.martian@gmail.com> wrote:
> Hi
>
> Thanks for the reposnse. The main reason is my record count could be from a
> few thousands to a million. But even at the lowly numbers of around 10000
> the interface can seem slugish if you read every record before displaying
> anything.
>
> As you mention, and has been disucssed above, doing stuff in the background
> is good way to go, but more complex. As a generla rule of coding I put as
> few unneccessary threads into a "phase 1" program as I can, because the
> complexity goes up hugly, threads can be complex to use, co-ordinate,
> interrupt etc... and chance of bugs goes up drmatically. So I tend to do
> that for a "Phase 2" - Bells and whistles phase and only when there isnt a
> simpler way.
>
> This thread has covered just about all approaches I can think of :-) thanks
> for the reponses.
>
> S
>
>
> On 10/27/06, Isaac Raway <isaac.raway@gmail.com> wrote:
> >
> > Why don't you design the table with a unique row ID, stored in an
> > integer field, then fetch a list of those ID numbers?
> >
> > For 5000 rows, assuming you store them in you application as 4 byte
> > longs, that's about 19 k of memory.
> >
> > Counting that result as you receive it isn't that difficult. If it
> > takes a long time (it probably won't) you can do it in another thread
> > and update the interface as appropriate.
> >
> > I'm not seeing a downside here.
> >
> > Isaac
> >
> > On 10/26/06, Da Martian <da.martian@gmail.com> wrote:
> > > No there isnt, but RDBM systems are a generalised data retrieval
> > mechanism.
> > > As such they suffer from that generality.
> > >
> > > Dont get me wrong, RDBM systems  are appropriate for 95% of all data
> > > requirements I have had to deal with and I would never dream of trying
> > to
> > > write one from scratch, nor can I imagine a world without them.
> > >
> > > However certain applications (Weather data, Gnome data, Large indices
> > (like
> > > google)) require using somethng designed specifically for that purpose.
> > If
> > > you customise data retrieval (and particluar your sorting/indcies/access
> > > path) you can leave rdbms in the dust in terms of performance. All I
> > have
> > > read about google, suggests they do exactly this. Although I must point
> > out,
> > > I dont actually know anything about google with any certainty. Just what
> > has
> > > "leaked" out over the years on the rumour mill. But designiing my own
> > > "google" like indices (on a smaller scale of coure) and some
> > specialisted
> > > weather stuff, it neccessary to throw away the rdbms and do it yourself.
> > For
> > > a goole query for instance, they know they will get a list of 1 or more
> > > words. They also know they will only ever search through the index of
> > words.
> > > They dont have other data types, records or tables. Why go through all
> > the
> > > hassles of compiling SQLs, and that generic overhead when your
> > application
> > > will only ever do one thing? You can just make an API like this
> > > "search(wordlist): Resultset. "
> > >
> > > You immediatly save yourself complexity and processing time. Then for
> > large
> > > indices you will know your data set, so instead of using a std BTree you
> > > would use a more appropraite DS possible with skip lists etc..
> > > .
> > > As for performing a database search twice, this whole thread has shown,
> > that
> > > sometimes the you have to :-)
> > >
> > > S
> > >
> > > On 10/25/06, John Stanton <johns@viacognis.com> wrote:
> > > >
> > > > There is no magic in data retrieval.  Google use the same physical
> > laws
> > > > as us ordinary mortals.
> > > >
> > > > I see no reason to ever perform a dataabase search twice.
> > > >
> > >
> > >
> >
> >
> > --
> > Isaac Raway
> > Entia non sunt multiplicanda praeter necessitatem.
> >
> > http://blueapples.org - blog
> > http://stonenotes.com - personal knowledge management
> >
> >
> > -----------------------------------------------------------------------------
> > To unsubscribe, send email to sqlite-users-unsubscribe@sqlite.org
> >
> > -----------------------------------------------------------------------------
> >
> >
>
>


-- 
Isaac Raway
Entia non sunt multiplicanda praeter necessitatem.

http://blueapples.org - blog
http://stonenotes.com - personal knowledge management

-----------------------------------------------------------------------------
To unsubscribe, send email to sqlite-users-unsubscribe@sqlite.org
-----------------------------------------------------------------------------

[prev in list] [next in list] [prev in thread] [next in thread]