[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Fatal error from a cassandra node
From: Anurag Gujral <anurag.gujral () gmail ! com>
Date: 2011-03-31 0:49:53
Message-ID: AANLkTin-miJdFDkQu6voRZW_6f4aEviFtkiP=JA52xoq () mail ! gmail ! com
[Download RAW message or body]
I am using version 0.7.3 I am not sure what happened I tried connecting to
the node and when I could not connect I looked at error logs i saw the log
messages which I pasted below
On Wed, Mar 30, 2011 at 5:04 PM, Narendra Sharma
<narendra.sharma@gmail.com>wrote:
> OOM at startup with 16GB... seems like an issue. Which version are you
> using? Can you provide some details on "failed node"? What exactly happened?
> That might give some clue. Also, you might want to start with log level set
> to debug to findout what more on what exactly Cassandra is doing that is
> causing OOM.
>
> -Naren
>
>
> On Wed, Mar 30, 2011 at 4:45 PM, Anurag Gujral <anurag.gujral@gmail.com>wrote:
>
>>
>> I am using 16G of heap space how much more should i increase.
>> Please suggest
>>
>> Thanks
>> Anurag
>>
>> On Wed, Mar 30, 2011 at 11:43 AM, Narendra Sharma <
>> narendra.sharma@gmail.com> wrote:
>>
>>> http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size
>>>
>>>
>>>
>>> On Wed, Mar 30, 2011 at 11:41 AM, Peter Schuller <
>>> peter.schuller@infidyne.com> wrote:
>>>
>>>> > I have 6 node cassandra cluster all are setup with same
>>>> > configuration I am getting fatal exceptions in one of the nodes
>>>> > ERROR [Thread-604] 2011-03-29 20:19:13,218
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-604,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [Thread-607] 2011-03-29 19:47:29,272
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-607,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [Thread-605] 2011-03-29 19:38:09,081
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-605,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [MutationStage:2] 2011-03-29 19:37:16,659
>>>> > DebuggableThreadPoolExecutor.java (line 103) Error in
>>>> ThreadPoolExecutor
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [GossipStage:1] 2011-03-29 20:27:29,898
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[GossipStage:1,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> >
>>>> > All all the nodes have 32 G of ram.
>>>> >
>>>> > Everytime I try to restart the failed node I get the above errors.
>>>>
>>>> Unless something is outright wrong, it sounds like you need to
>>>> increase your JVM heap size in cassandra-env.sh. That you're getting
>>>> it on start-up sounds consistent with commit log reply filling the
>>>> heap in the form of memtables that are sized too big for your heap.
>>>>
>>>> There's a wiki page somewhere that describes the overall rule of thumb
>>>> for heap sizing, but I can't find it right now.
>>>>
>>>> --
>>>> / Peter Schuller
>>>>
>>>
>>>
>>>
>>> --
>>> Narendra Sharma
>>> Solution Architect
>>> *http://www.persistentsys.com*
>>> *http://narendrasharma.blogspot.com/*
>>>
>>>
>>>
>>
>
>
> --
> Narendra Sharma
> Solution Architect
> *http://www.persistentsys.com*
> *http://narendrasharma.blogspot.com/*
>
>
>
[Attachment #3 (text/html)]
I am using version 0.7.3 I am not sure what happened I tried connecting to the node \
and when I could not connect I looked at error logs i saw the log messages which I \
pasted below<br><br><div class="gmail_quote">On Wed, Mar 30, 2011 at 5:04 PM, \
Narendra Sharma <span dir="ltr"><<a \
href="mailto:narendra.sharma@gmail.com">narendra.sharma@gmail.com</a>></span> \
wrote:<br> <blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; \
border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">OOM at startup with \
16GB... seems like an issue. Which version are you using? Can you provide some \
details on "failed node"? What exactly happened? That might give some clue. \
Also, you might want to start with log level set to debug to findout what more on \
what exactly Cassandra is doing that is causing OOM.<div>
<br></div><div>-Naren<div><div></div><div class="h5"><br><div><br><div \
class="gmail_quote">On Wed, Mar 30, 2011 at 4:45 PM, Anurag Gujral <span \
dir="ltr"><<a href="mailto:anurag.gujral@gmail.com" \
target="_blank">anurag.gujral@gmail.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, \
204, 204); padding-left: 1ex;"> <br>I am using 16G of heap space how much more should \
i increase.<br>Please suggest<br><br>Thanks<br><font \
color="#888888">Anurag</font><div><div></div><div><br><div class="gmail_quote">On \
Wed, Mar 30, 2011 at 11:43 AM, Narendra Sharma <span dir="ltr"><<a \
href="mailto:narendra.sharma@gmail.com" \
target="_blank">narendra.sharma@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px \
solid rgb(204, 204, 204); padding-left: 1ex;"><a \
href="http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size" \
target="_blank">http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size</a><div>
<br></div><div><div><div></div><div><br><br><div class="gmail_quote">On Wed, Mar 30, \
2011 at 11:41 AM, Peter Schuller <span dir="ltr"><<a \
href="mailto:peter.schuller@infidyne.com" \
target="_blank">peter.schuller@infidyne.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px \
solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div>> I \
have 6 node cassandra cluster all are setup with same<br>
> configuration I am getting fatal exceptions in one of the nodes<br>
> ERROR [Thread-604] 2011-03-29 20:19:13,218 AbstractCassandraDaemon.java<br>
> (line 114) Fatal exception in thread Thread[Thread-604,5,main]<br>
> java.lang.OutOfMemoryError: Java heap space<br>
> ERROR [Thread-607] 2011-03-29 19:47:29,272 AbstractCassandraDaemon.java<br>
> (line 114) Fatal exception in thread Thread[Thread-607,5,main]<br>
> java.lang.OutOfMemoryError: Java heap space<br>
> ERROR [Thread-605] 2011-03-29 19:38:09,081 AbstractCassandraDaemon.java<br>
> (line 114) Fatal exception in thread Thread[Thread-605,5,main]<br>
> java.lang.OutOfMemoryError: Java heap space<br>
> ERROR [MutationStage:2] 2011-03-29 19:37:16,659<br>
> DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor<br>
> java.lang.OutOfMemoryError: Java heap space<br>
> ERROR [GossipStage:1] 2011-03-29 20:27:29,898 AbstractCassandraDaemon.java<br>
> (line 114) Fatal exception in thread Thread[GossipStage:1,5,main]<br>
> java.lang.OutOfMemoryError: Java heap space<br>
><br>
> All all the nodes have 32 G of ram.<br>
><br>
> Everytime I try to restart the failed node I get the above errors.<br>
<br>
</div></div>Unless something is outright wrong, it sounds like you need to<br>
increase your JVM heap size in cassandra-env.sh. That you're getting<br>
it on start-up sounds consistent with commit log reply filling the<br>
heap in the form of memtables that are sized too big for your heap.<br>
<br>
There's a wiki page somewhere that describes the overall rule of thumb<br>
for heap sizing, but I can't find it right now.<br>
<br>
--<br>
<font color="#888888">/ Peter Schuller<br>
</font></blockquote></div><br><br clear="all"><br></div></div>-- <br><font \
color="#999999">Narendra Sharma</font><div><font color="#999999">Solution \
Architect</font><br><div><i><a href="http://www.persistentsys.com" \
target="_blank">http://www.persistentsys.com</a></i></div>
<div><i><a href="http://narendrasharma.blogspot.com/" \
target="_blank">http://narendrasharma.blogspot.com/</a></i></div><div><br></div></div><br>
</div>
</blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><font \
color="#999999">Narendra Sharma</font><div><font color="#999999">Solution \
Architect</font><br><div><i><a href="http://www.persistentsys.com" \
target="_blank">http://www.persistentsys.com</a></i></div>
<div><i><a href="http://narendrasharma.blogspot.com/" \
target="_blank">http://narendrasharma.blogspot.com/</a></i></div><div><br></div></div><br>
</div></div></div></div>
</blockquote></div><br>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic