'Re: [ast-developers] str="$( < bigfile )" with "bigfile" == 32MB triggers excessive memory usage...'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ast-developers
Subject:    Re: [ast-developers] str="$( < bigfile )" with "bigfile" == 32MB triggers excessive memory usage...
From:       Lionel Cons <lionelcons1972 () googlemail ! com>
Date:       2012-06-28 17:04:13
Message-ID: CAPJSo4UdMvpc3iTtWOW1h7Od3f65imjnsT=NvVyZTHzmpYHBvA () mail ! gmail ! com
[Download RAW message or body]

On 28 June 2012 17:33, Glenn Fowler <gsf@research.att.com> wrote:
>
> On Thu, 28 Jun 2012 13:43:52 +0200 Irek Szczesniak wrote:
>> On Thu, Jun 28, 2012 at 9:28 AM, Glenn Fowler <gsf@research.att.com> wrote:
>> >
>> > On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote:
>> >> On 21 June 2012 08:18, Roland Mainz <roland.mainz@nrubsig.org> wrote:
>> >> > Hi!
>> >> >
>> >> > ----
>> >> >
>> >> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo
>> >> > which shows a problem in ast-ksh.2012-06-12 with memory usage for
>> >> > str="$( < bigfile )" (and similar usage). The problem seems to be that
>> >> > a larger input file size (for example 32MB) triggers excessive memory
>> >> > usage, usually well beyond 512MB on Solaris 11/AMD64.
>> >> >
>> >> > Running the demo triggers the following error (this comes from the
>> >> > limits defined via "ulimit" in the test (e.g. we assume that a file
>> >> > size of 32MB will not cause more than 64MB of extra memory usage in
>> >> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script,
>> >> > e.g. the ulimit limits are 80MB total in this case. However even that
>> >> > is very very generous)):
>> >> > -- snip --
>> >> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh
>> >> > -rw-r--r--   1 test001  users    33226753 Jun 21 08:10 mytestfile
>> >> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of
>> >> > space on 8486912 byte request ( region 545751040 segments 133 busy
>> >> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space]
>> >> > -- snip --
>> >> >
>> >> >
>> >> > ----
>> >> >
>> >> > Bye,
>> >> > Roland
>> >> >
>> >> > --
>> >> >   __ .  . __
>> >> >  (o.\ \/ /.o) roland.mainz@nrubsig.org
>> >> >   \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
>> >> >   /O /==\ O\  TEL +49 641 3992797
>> >> >  (;O/ \/ \O;)
>> >> >
>> >> > _______________________________________________
>> >> > ast-developers mailing list
>> >> > ast-developers@research.att.com
>> >> > https://mailman.research.att.com/mailman/listinfo/ast-developers
>> >> >
>> >
>> >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO
>> >> showstopper because it prevents the use of larger texts in command
>> >> substitutions:
>> >
>> >> ksh bigfile.sh
>> >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile
>> >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte
>> >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free
>> >> 55:63535184:2850720 ) [Cannot allocate memory]
>> >
>> > this is a vmalloc fragmentation issue when mmap() is used to allocate region blocks
>> > coalescing adjacent mmap() blocks is not a portable op across mmap() implementations
>> > a side effect of this is fragmentation, sometimes very bad
>> >
>> > using mmap() instead of sbrk() by default started with the first thread-safe vmalloc
>> > implementation earlier this year; the ast beta 2012-06-28, to be posted later today,
>> > works around this by detecting if the current application requires thread-safety
>> > (via the ast aso(3) api); if it does require thread-safe then mmap() is used,
>> > otherwise it uses sbrk() (this is the case for all current ast commands), falling
>> > back to mmap() only if sbrk() fails
>
>> So how is this going to work for applications try to use libast but
>> are not ksh93? sbrk() is a poor choice if more than one allocator
>> tries to be in charge of it, which is alas the majority of today's
>> applications which depend on more than one toolkit (of which libast is
>> only one, but a major one). For example you can't use libast in
>> single-threaded Gnome or KDE applications when libast uses the sbrk()
>> allocator, they'll just crash and burn sooner or later.
>
> up to 2012 ast had been using sbrk()
> if mmap() implementations can't support coalescing efficiently (or at all)
> then we have to fall back to something that works
>
>> > the next iteration of thread-safe vmalloc is currently in test phase
>> > it will use sbrk() by default in a thread-safe way (only restriction - no api
>> > but vmalloc(3) may call sbrk()), and the workaround above will not be needed
>
>> That's IMO both useless and the wrong way to go. It might work for
>> ksh93 and the AST tools, but using sbrk() in any other context is not
>> going to work because GNU libc and other libc implementations assume
>> they are the sole users of sbrk(). Forget that very quickly or you'll
>> drown in bug reports for the rest of the decade.
>
> if -last is in the mix we expect that the ast vmalloc malloc family is the allocator

You can't assume that libast is the only allocator present in an
application. Almost every complex application, utility library (like
libast) or GUI toolkit (Gnome, KDE/QT) comes with its variation of an
allocator, for better or worse. Now if these are linked together, act
friendly and don't compete for unprotected resources all is good.
sbrk() however is a point of hard competition (...I'm the only one
who's using sbrk, all others are usurpers who don't belong here...)
and there's no way to protect against abuse (you can check for brk,
but it requires that all other users of sbrk do that as well. See
usurpers)

So the best way is stay away from sbrk. Its not worth the time wasted
on fixing the problems you'll get as there is no way you can fix it
the right way, where "right way" is a solution which is interoperable
with all other pretenders and usurpers. This is a hopeless battle.
Just stop thinking about that and try to get the fragmentation issues
solved.

> modulo the thread-safe growing pains this year this has served ast and its consmers well
>
> for gnu we use the gnu malloc hooks
> if the hooks are implemented correctly by gnu and used correctly by ast
> then this will not be a problem

Only if those hooks are used by all allocators, the platform has such
hooks and no one uses sbrk() directly. Please find me a platform,
toolkit or utility library outside AST which does that properly.

Lionel

_______________________________________________
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers
[prev in list] [next in list] [prev in thread] [next in thread] 