[prev in list] [next in list] [prev in thread] [next in thread] 

List:       bash-bug
Subject:    bash sucks lots of RAM doing read & echo in loop
From:       wojtek () wojtek ! tensor ! gdynia ! pl
Date:       2008-02-04 19:05:17
Message-ID: 200802041905.m14J5HAi009208 () wojtek ! tensor ! gdynia ! pl
[Download RAW message or body]

Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: freebsd6.2
Compiler: cc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' \
-DCONF_OSTYPE='freebsd6.2' -DCONF_MACHTYPE='i386-portbld-freebsd6.2' \
-DCONF_VENDOR='portbld' -DLOCALEDIR='/usr/local/share/locale' -DPACKAGE='bash' \
-DSHELL  -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  -I/usr/local/include -O2 \
-fno-strict-aliasing -pipe -march=pentium3 uname output: FreeBSD \
wojtek.tensor.gdynia.pl 6.2-RELEASE-p10 FreeBSD 6.2-RELEASE-p10 #0: Tue Jan 15 \
16:35:37 CET 2008     root@FreeSBIE.LiveCD:/usr/src/sys/i386/compile/p234  i386 \
Machine Type: i386-portbld-freebsd6.2

Bash Version: 3.2
Patch Level: 17
Release Status: release

Description:
	bash keeps all output it echoed in memory.
	in my case it sucked over 3GB VM processing 2.5GB file.
	details in "Repeat-By:"

Repeat-By:
	such fragment of my script is executed

	while read ll;do  echo $RANDOM $RANDOM $ll; done <$tmp2 | $SORT | cut -f 3- -d " " \
>$tmp3

	where $tmp2 is 2.5GB input file, $tmp3 is output file and $SORT is "sort -S 512m"

	this routine has to randomize lines of input file $tmp2.

	and it works fine, but WHY bash gets over 3GB size, which grows as it processes \
$tmp2  input?
	
	it should not keep anything in memory but pipe to sort program (which takes 512MB as
	i wanted it+tempfiles). but it does both.

Fix:


[prev in list] [next in list] [prev in thread] [next in thread]