[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: RE: Upgrade from 2.1 to 3.11
From: "ZAIDI, ASAD A" <az192g () att ! com>
Date: 2018-08-28 16:13:52
Message-ID: 053247A8CBB6754B8345743B8F18D68D48924F47 () MOSTLS1MSGUSRFA ! ITServices ! sbc ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
You may want to check if coincidentally you're having expired cells in heap. GC log \
should be able to tell you OR look for tombstones in system.log file. See your \
compactions are under control and normal. This may not be related to upgrade at all!
From: Pradeep Chhetri [mailto:pradeep@stashaway.com]
Sent: Tuesday, August 28, 2018 3:32 AM
To: user@cassandra.apache.org
Subject: Re: Upgrade from 2.1 to 3.11
You may want to try upgrading to 3.11.3 instead which has some memory leaks fixes.
On Tue, Aug 28, 2018 at 9:59 AM, Mun Dega \
<mundega97@gmail.com<mailto:mundega97@gmail.com>> wrote: I am surprised that no one \
else ran into any issues with this version. GC can't catch up fast enough and there \
is constant Full GC taking place.
The result? unresponsive nodes makeing entire cluster unusable.
Any insight on this issue from anyone that is using this version would be \
appreciated.
Ma
On Fri, Aug 24, 2018, 04:30 Mohamadreza Rostami \
<mohamadrezarostami2@gmail.com<mailto:mohamadrezarostami2@gmail.com>> wrote: You have \
very large heap,it's take most of cpu time in GC stage.you should in maximum set \
heap on 12GB and enable row cache to your cluster become faster.
On Friday, 24 August 2018, Mun Dega <mundega97@gmail.com<mailto:mundega97@gmail.com>> \
wrote: 120G data
28G heap out of 48 on system
9 node cluster, RF3
On Thu, Aug 23, 2018, 17:19 Mohamadreza Rostami \
<mohamadrezarostami2@gmail.com<mailto:mohamadrezarostami2@gmail.com>> wrote: Hi,
How much data do you have? How much RAM do your servers have? How much do you have a \
heep? On Thu, Aug 23, 2018 at 10:14 PM Mun Dega \
<mundega97@gmail.com<mailto:mundega97@gmail.com>> wrote: Hello,
We recently upgraded from Cassandra 2.1 to 3.11.2 on one cluster. The process went \
OK including upgradesstable but we started to experience high latency for r/w, \
occasional OOM and long GC pause after.
For the same cluster with 2.1, we didn't have any issues like this. We also kept \
server specs, heap, all the same in post upgrade
Has anyone else had similar issues going to 3.11 and what are the major changes that \
could have such a major setback in the new version?
Ma Dega
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You \
may want to check if coincidentally you're having expired cells in heap. GC log \
should be able to tell you OR look for tombstones in system.log file. See your \
compactions are under control and normal. This may not be related to upgrade at \
all!<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Pradeep Chhetri \
[mailto:pradeep@stashaway.com] <br>
<b>Sent:</b> Tuesday, August 28, 2018 3:32 AM<br>
<b>To:</b> user@cassandra.apache.org<br>
<b>Subject:</b> Re: Upgrade from 2.1 to 3.11<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">You may want to try upgrading to 3.11.3 instead which has some \
memory leaks fixes.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 9:59 AM, Mun Dega <<a \
href="mailto:mundega97@gmail.com" target="_blank">mundega97@gmail.com</a>> \
wrote:<o:p></o:p></p> <blockquote style="border:none;border-left:solid #CCCCCC \
1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<p class="MsoNormal">I am surprised that no one else ran into any issues with this \
version. GC can't catch up fast enough and there is constant Full GC taking \
place.<o:p></o:p></p> <div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The result? unresponsive nodes makeing entire cluster \
unusable.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any insight on this issue from anyone that is using this version \
would be appreciated.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ma<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Aug 24, 2018, 04:30 Mohamadreza Rostami <<a \
href="mailto:mohamadrezarostami2@gmail.com" \
target="_blank">mohamadrezarostami2@gmail.com</a>> wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <p class="MsoNormal">You have very large \
heap,it's take most of cpu time in GC stage.you should in maximum set heap on \
12GB and enable row cache to your cluster become faster.<br> <br>
On Friday, 24 August 2018, Mun Dega <<a href="mailto:mundega97@gmail.com" \
target="_blank">mundega97@gmail.com</a>> wrote:<o:p></o:p></p> <blockquote \
style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<div>
<p class="MsoNormal">120G data<o:p></o:p></p>
<div>
<p class="MsoNormal">28G heap out of 48 on system<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">9 node cluster, RF3<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Aug 23, 2018, 17:19 Mohamadreza Rostami <<a \
href="mailto:mohamadrezarostami2@gmail.com" \
target="_blank">mohamadrezarostami2@gmail.com</a>> wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal">How much data do you have? How much RAM do your servers \
have? How much do you have a heep?<o:p></o:p></p> <div>
<div>
<p class="MsoNormal">On Thu, Aug 23, 2018 at 10:14 PM Mun Dega <<a \
href="mailto:mundega97@gmail.com" target="_blank">mundega97@gmail.com</a>> \
wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-right:0in"> <div>
<p class="MsoNormal">Hello,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We recently upgraded from Cassandra 2.1 to 3.11.2 on one \
cluster. The process went OK including upgradesstable but we started to \
experience high latency for r/w, occasional OOM and long GC pause \
after.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For the same cluster with 2.1, we didn't have any issues like \
this. <span style="font-family:"Arial",sans-serif">We also kept \
server specs, heap, all the same in post upgrade</span><o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Has anyone else had similar issues going to 3.11 and what are \
the major changes that could have such a major setback in the new \
version?<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ma Dega<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic