[prev in list] [next in list] [prev in thread] [next in thread]
List: ast-developers
Subject: Re: valgrind hits in ast-ksh.2012-06-26 when hitting memory limits... / was: Re: [ast-developers] st
From: Glenn Fowler <gsf () research ! att ! com>
Date: 2012-06-27 19:06:37
Message-ID: 201206271906.q5RJ6bmI002744 () terra ! research ! att ! com
[Download RAW message or body]
On Wed, 27 Jun 2012 14:28:29 +0200 Roland Mainz wrote:
> On Wed, Jun 27, 2012 at 1:25 PM, Cedric Blancher
> <cedric.blancher@googlemail.com> wrote:
> > On 27 June 2012 12:55, Cedric Blancher <cedric.blancher@googlemail.com> wrote:
> >> On 21 June 2012 08:18, Roland Mainz <roland.mainz@nrubsig.org> wrote:
> >>> Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo
> >>> which shows a problem in ast-ksh.2012-06-12 with memory usage for
> >>> str="$( < bigfile )" (and similar usage). The problem seems to be that
> >>> a larger input file size (for example 32MB) triggers excessive memory
> >>> usage, usually well beyond 512MB on Solaris 11/AMD64.
> >>>
> >>> Running the demo triggers the following error (this comes from the
> >>> limits defined via "ulimit" in the test (e.g. we assume that a file
> >>> size of 32MB will not cause more than 64MB of extra memory usage in
> >>> ksh93 (ksh93 itself is granted 16MB for itself&&running the script,
> >>> e.g. the ulimit limits are 80MB total in this case. However even that
> >>> is very very generous)):
> >>> -- snip --
> >>> $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh
> >>> -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile
> >>> bigfilecommandsubstitution001.sh: line 36: storage allocator out of
> >>> space on 8486912 byte request ( region 545751040 segments 133 busy
> >>> 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space]
> >>> -- snip --
> [snip]
> >>
> >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO
> >> showstopper because it prevents the use of larger texts in command
> >> substitutions:
> >>
> >> ksh bigfile.sh
> >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile
> >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte
> >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free
> >> 55:63535184:2850720 ) [Cannot allocate memory]
> >
> > I tried it with these values but it still fails:
> > integer map_file_size=16 # 32MB in $( <file )
> > integer memory_limit=512 # 64MB memory limit to handle the cmd substitution
> >
> > So ksh 2012-06026 requires more than 512MB to process a 16MB string in
> > a command substitution. This is really really bad.
> Grumpf... I agree... ;-((
> There is a 2nd issue... I am getting a couple of valgrind hits when a
> libast-based application (e.g. ksh93 running the test script of this
> bug) runs out of memory:
> -- snip --
> $ LC_ALL=C valgrind --read-var-info=yes --num-callers=50 ~/bin/ksh bigfile.sh
> ==22925== Memcheck, a memory error detector
> ==22925== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
> ==22925== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
> ==22925== Command: /home/test001/bin/ksh bigfile.sh
> [snip]
> ==22926== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
> -rw-r--r-- 1 test001 users 8306689 Jun 27 13:50 mytestfile
> ==22925== Conditional jump or move depends on uninitialised value(s)
> ==22925== at 0x552174: sfvprintf (sfvprintf.c:1017)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Conditional jump or move depends on uninitialised value(s)
> ==22925== at 0x552185: sfvprintf (sfvprintf.c:1019)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Conditional jump or move depends on uninitialised value(s)
> ==22925== at 0x55257F: sfvprintf (sfvprintf.c:1030)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Conditional jump or move depends on uninitialised value(s)
> ==22925== at 0x55258A: sfvprintf (sfvprintf.c:1030)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Conditional jump or move depends on uninitialised value(s)
> ==22925== at 0x552591: sfvprintf (sfvprintf.c:1030)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Use of uninitialised value of size 8
> ==22925== at 0x5525D2: sfvprintf (sfvprintf.c:1030)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> ==22925== Use of uninitialised value of size 8
> ==22925== at 0x5525E7: sfvprintf (sfvprintf.c:1030)
> ==22925== by 0x4FE0D8: errorv (error.c:532)
> ==22925== by 0x4FD280: error (error.c:350)
> ==22925== by 0x52046E: nomalloc (memfatal.c:62)
> ==22925== by 0x560DAA: _vmextend (vmprivate.c:85)
> ==22925== by 0x55D555: bestalloc (vmbest.c:675)
> ==22925== by 0x55E43E: bestresize (vmbest.c:935)
> ==22925== by 0x563329: _ast_realloc (malloc.c:549)
> ==22925== by 0x50D45A: stkgrow (stk.c:524)
> ==22925== by 0x50C916: stkexcept (stk.c:195)
> ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58)
> ==22925== by 0x5559C0: sfwr (sfwr.c:231)
> ==22925== by 0x555D83: sfwrite (sfwrite.c:108)
> ==22925== by 0x457244: mac_copy (macro.c:2471)
> ==22925== by 0x45683F: comsubst (macro.c:2244)
> ==22925== by 0x4521B1: varsub (macro.c:1168)
> ==22925== by 0x45079F: copyto (macro.c:633)
> ==22925== by 0x44EBD1: sh_mactrim (macro.c:183)
> ==22925== by 0x458AB5: nv_setlist (name.c:339)
> ==22925== by 0x47FF94: sh_exec (xec.c:1208)
> ==22925== by 0x418E2B: exfile (main.c:581)
> ==22925== by 0x417FFE: sh_main (main.c:353)
> ==22925== by 0x417128: main (pmain.c:45)
> ==22925==
> bigfile.sh: line 35: storage allocator out of space on 6782976 byte
> request ( region 338362368 segments 92 busy 166:7068096:6717344 free
> 98:331282688:6651808 ) [Cannot allocate memory]
> ==22925==
> -- snip --
when I try this, with various ulimit sizes, I don't get the sfprintf() hit
I either get "Valgrind's memory management: out of memory:" error
meaning valgrind itself asked for memory and it failed or I get
"/home/gsf/arch/linux.i386-64-test/bin/ksh: out of memory"
_______________________________________________
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic