[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lucene-dev
Subject:    Re: BitSet implementation and large index
From:       jian chen <chenjian1227 () gmail ! com>
Date:       2005-02-14 17:31:06
Message-ID: 7ca123910502140931311dd4b0 () mail ! gmail ! com
[Download RAW message or body]

Hi,

In database systems implementation, there is a type of index called
bit map indexing. The bitset implementation could borrow idea from the
database engine implementation.

You could squeeze all the 0's together and write how many of those
0's, that might be very memory saving.

There are various kinds of algorithms for doing this bitset
compression. A good book for reference is the "Database
impelmentations" from Ullman, and other two professors in Standford
university.

Cheers,

Jian


On Mon, 14 Feb 2005 09:29:26 -0600 (CST), tony@simpleobjects.com
<tony@simpleobjects.com> wrote:
> It seems that for a huge index, it might be a good idea to use a different
> implementation of the BitSet when doing filtering (assuming the
> non-filtered set is relatively small).  This would really help minimize
> the memory required for each filter operation.
> 
> Since the default implementation of BitSet allocates enough memory for
> each position in the set, it seems overkill for a set that has a small
> number of "on" values.
> 
> Any thoughts?
> 
> Tony Schwartz
> 
> by the way, I just started using Lucene about 2 weeks ago, and I am really
> loving it.  The sky's the limit for this framework.  Thanks very much to
> those of you involved in it's development.  Extremely powerful!
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org

[prev in list] [next in list] [prev in thread] [next in thread]