[prev in list] [next in list] [prev in thread] [next in thread] 

List:       lucene-user
Subject:    Re: 2.1-dev memory leak?
From:       Michael McCandless <lucene () mikemccandless ! com>
Date:       2006-11-30 23:32:04
Message-ID: 456F69F4.3050906 () mikemccandless ! com
[Download RAW message or body]

Otis Gospodnetic wrote:

> Wow, that was fast - java-user support is just as fast as I heard! ;)

Well let's withhold judgment until we see if that tool really works
correctly :)

> I'll try your patch shortly. Like I said, the bug may be in my application.  Here \
> is a clue.  Memory usage increases with the number of open files (file descriptors) \
> on the system, and lsof gives: 
> COMMAND     PID    USER   FD      TYPE     DEVICE      SIZE       NODE NAME
> ...
> java      24476   xxx *562r      REG      253,0    657314    8470761 \
> /xxx/users/1/1-home/link/index/_20k.cfs java      24476   xxx *894r      REG      \
> 253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs java      24476  \
> xxx *078r      REG      253,0    657314    8470761 \
> /xxx/users/1/1-home/link/index/_20k.cfs java      24476   xxx *648r      REG      \
>                 253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs
> ...
> 
> If I'm reading this right, this tells me that this same file has been opened a \
> number of different times (note the FD column, all file descriptors are different). \
> This must correspond to multiple new IndexSearcher(...) calls, no?  Multiple new \
> IndexSearcher(...) calls on the same index are okay in my case - because the system \
> has tens of thousands of separate indices, it can't keep all IndexSearchers open at \
> all times, so I use the LRU algo to keep only recently used IndexSearchers open.  \
> The other ones I "let go" without an explicit close() call.  The assumption is that \
> the old IndexSearchers "expire", that they get garbage collected, as I'm no longer \
> holding references to them. Iff I understand this correctly, the fact that I see \
> these open file descriptors all pointing to the same index file tells me that the \
> old IndexSearchers are just hanging around and are not getting cleaned up. 
> I can also see the number of file descriptors increasing with time:
> 
> $ /usr/sbin/lsof | grep -c '/1-home/link/index/_20k.cfs'
> 14
> $ /usr/sbin/lsof | grep -c '/1-home/link/index/_20k.cfs'
> 23
> 
> This may still not point to my app having the bug, but it points to something not \
> releasing IndexSearcher/IndexReader, as that is not getting GCed as before.  I did \
> not change my logic for creating new IndexSearchers (inlined in my previous email). \
> On the other hand, this app has recently started getting a lot more search action, \
> so perhaps it's just that the GC is not cleaning things up fast enough.... I happen \
> to have an lsof output from the same system from July.  I see the same thing there \
> - a number of FDs open and pointing to the same .cfs index file.  Perhaps it's just \
> that the JVM GC was able to clean things up then, and now it can't, because the CPU \
> is maxed out.... really maxed out.

Otis, one more thing you could try is to call
SegmentInfos.setInfoStream(System.out).  This will cause lockless to
print details whenever it has to retry loading a segments_N file (just
in case something is going horribly wrong...).  This should only happen
when there is contention (writer committing just as reader is opening)
which depending on your application may or may not ever happen.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]