[prev in list] [next in list] [prev in thread] [next in thread] 

List:       turbine-jcs-user
Subject:    RE: Using group within region causing out of memory problem
From:       Amy Wang <amy.wang () intersperse ! com>
Date:       2003-05-30 2:36:03
[Download RAW message or body]


Thanks for your quick response. I am implementing a repository 
that needs to hold up hundred thousands of managed objects and 
their relations. It'd be efficient for those LRU objects to be cached. 

> I'm ignoring the problem with the groups for the moment.  Try 
> your test without using the groups at all.  
Without using the group, is there any way to retrieve the 
keySet or value iterator? I know it's bad practice but I need that for
object lookup often times. 

Since the group access provide more api, that's why I wrote a cache wrapper 
that implements java Map. All unit tests went fine except 
the memory issue. This becomes my biggest challenge if not 
using the group access. How to get the info similar to the api provided by
Map? Any suggestion?

> What was your test code like for JPROBE?  The indexed disk cache keeps
> the key and the location of the item in the file in memory.  This is
> kept in the disk element descriptor, which is very small.  Yes, if you
> will have an infinite number of items, it will take up an infinite
> amount of space.  This isn't a memory leak though.  
Yes. After I switch to region only cache, the performance is much better.

> Items are removed from the list of keys for the disk when they are
> removed.  This should not be a problem in most applications, 
> since there
> is no point in caching unique data and the size of the key is very
> small.  You should be able to have millions and millions of items
> without memory problems.  I can't understand how anyone would 
> ever have this many items in the cache.  
You are right, just checked the code, the element takes only the 
location info, average size is 24 bytes only.

> I guess if you were putting something like session related data in the
> cache and none of the sessions were cleaned up, then after months it
> could be a problem if you have as many users as, say, yahoo.
>   
> There is another, much slower b-tree auxiliary that doesn't keep many
> keys in memory.  
It would be a good idea to have it as a strategy and have the user 
determine whether memory or speed to be compromised.

> We may want to implement a disk cleanup system.  It could check to see
> if elements have expired, etc.  I've been thinking about how to have
> periodic defragmentation too.  Right now it just defrags on shutdown. 
> 
> What does your test code look like?  What are you planning on caching?
> Can you give some rough idea of what you are using JCS for?
My test is simply loading as many as object to stress test the jcs, but 
our application is more complex in that it deals with hundred thousands 
of active objects per server and multiple servers need to be supported. 
A cache with auxiliary support is definitely a great help. Also, JCS 
takes care of the synchronization issue which is a big plus. 

Thanks a lot for the advise. 

Amy

> Cheers,
> 
> Aaron
> 
> 
> 
> 
> > -----Original Message-----
> > From: Amy Wang [mailto:amy.wang@intersperse.com]
> > Sent: Thursday, May 29, 2003 7:08 PM
> > To: 'turbine-jcs-user@jakarta.apache.org'; 'turbine-jcs-
> > dev@jakarta.apache.org'
> > Subject: Using group within region causing out of memory problem
> > 
> > Hi, All,
> > 
> > I just wrote some tests against JCS performance.  Here is some
> interesting
> > thing I discovered.
> > If I put cached object to group using putInGroup(key, groupName,
> value),
> > it
> > will result in memory leak problem. Below is the memory report from
> > JProble:
> > Before it ran out of memory, the groupId, groupAttrName, 
> Integer(don't
> > know
> > what it is used for) and IndexedDiskElementDescriptor used 
> up most of
> the
> > heap.
> > 
> > Class
> > Memory usage
> > org.apache.jcs.engine.control.group.GroupId
> > 1,697,328 (21.0%)
> > org.apache.jcs.engine.control.group.GroupAttrName
> > 1,697,328 (21.0%)
> > java.lang.Integer
> > 1,115,024 (13.8%)
> > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskElementDescriptor
> > 1,624,560
> > (20.1%)
> > 
> > If I put the object directly to a region without using any 
> group, the
> > performance is getting better but still, 
> IndexedDiskElementDescriptor
> is
> > the
> > top one object consuming the heap. Why the
> IndexedDiskElementDescriptor
> > need
> > to keep the serialized form of the cached object. Shouldn't 
> the cached
> > object suppose to be written out to the auxiliary disk and 
> be removed
> from
> > the memory? Without using the grouping, the number of objects in JCS
> is 6
> > times more than using the grouping.
> > 
> > With this memory issue, it is really intimidating us from using JCS.
> Any
> > explanation or answer will be greatly appreciated.
> > 
> > Best regards,
> > 
> > Amy
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: 
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> turbine-jcs-user-help@jakarta.apache.org
> 

[prev in list] [next in list] [prev in thread] [next in thread]