[prev in list] [next in list] [prev in thread] [next in thread] 

List:       python-list
Subject:    Re: Is Python Suitable for Large Find & Replace Operations?
From:       Steve Holden <steve () holdenweb ! com>
Date:       2005-06-14 17:01:58
Message-ID: d8n28p$942$1 () sea ! gmane ! org
[Download RAW message or body]

Brian wrote:
> Hi Rbt,
> 
> To give an example of processing a lot of data, I used Python to read 
> and process every word in a single text file that contained the entire 
> King James Bible version.  It processed it within about one second -- 
> split the words, etc.  Worked quite well.
> 
> Hope this helps,
> 
Hardly "big" by the OP's standards at only 4.5 MB (and roughly 790,000 
words, if anyone's interested).

The problem with processing terabytes is frequently the need to 
accomodate the data with techniques that allow the virtual memory to 
avoid becoming swamped. The last thing you need to do under such 
circumstances is create an in-memory copy of hte entire data set, since 
on most real computers this will inflict swapping behavior.

regards
  Steve
-- 
Steve Holden        +1 703 861 4237  +1 800 494 3119
Holden Web LLC             http://www.holdenweb.com/
Python Web Programming  http://pydish.holdenweb.com/

-- 
http://mail.python.org/mailman/listinfo/python-list
[prev in list] [next in list] [prev in thread] [next in thread]