[prev in list] [next in list] [prev in thread] [next in thread] 

List:       glibc-help
Subject:    Amount of virtual Memory larger than heap peak
From:       singler () ira ! uka ! de (Johannes Singler)
Date:       2008-10-14 13:44:00
Message-ID: 48F4A216.4090901 () ira ! uka ! de
[Download RAW message or body]

Hi!

>> My program allocates and deallocates large chunks (Megabytes) of memory
>> many times (using new[], but this should be mapped to malloc).  For
>> example, using memusage.sh, I find out that the heap peak is about
>> 100MB.  However, the virtual memory size of my program has grown to
>> about twice this size, as shown by top.  The resident set size stays at
>> about 100MB, as expected.
> 
> You'll have to see what morecore does when freeing memory but I've
> been told that when malloc wants to create a new malloc arena that it
> moves sbrk.  Free can't move sbrk back down because another malloc
> call may be using the arena.

The chunks we are allocating are 1,5GB in size, so they are
allocated via mmap anyways.  I have double-checked that using strace.

>> I don't use mmap, but read and write large amounts of data from disk
>> using read(int fd, void *buf, size_t count) and the corresponding write.
> 
> You should probably use mmap to allocate and use mallopt to force mmap
> usage.  Therefore you can reclaim memory when you're done with it.

In the end, the problem was that some library set TOP_PAD to 128K.  So
the large chunk + 128K was allocated, and some minor allocations
prevented the block from being freed.  This lead to the memory
fragmentation.  The problem is solved by setting TOP_PAD to 0.

Thanks for your answer!

Johannes


[prev in list] [next in list] [prev in thread] [next in thread]