[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: MapFile inner workings
From:       Doug Cutting <cutting () apache ! org>
Date:       2007-06-20 17:05:09
Message-ID: 46795E45.3020804 () apache ! org
[Download RAW message or body]

Every 128th key is held in memory.  So if you've got 1M keys in a 
MapFile, then opening a MapFile.Reader would read 10k keys into memory. 
  Binary search is used on these in-memory keys, so that a maximum of 
127 entries must be scanned per random access.

Doug

Phantom wrote:
> Hi All
> 
> I know this is a tall ask. I am going through the source code. But could
> someone please tell me the intuition behind the design of the MapFile 
> class.
> If I were using the MapFile against the local file system are there any
> limitations to the number of items I can store. I mean can I have a MapFile
> on the local filesystem that has say 10GB of data. The reason I ask this is
> because I did read in the documentation that it behooves one to keep the 
> key
> small since the index is completely kept in memory. Could someone please
> enlighten me ?
> 
> Thanks
> Avinash
> 

[prev in list] [next in list] [prev in thread] [next in thread]