[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: MUTATION messages dropped
From: Aaron Morton <aaron () thelastpickle ! com>
Date: 2013-12-31 1:44:55
Message-ID: 6802AC96-DC4D-4E73-8966-48FDC8FEB4AB () thelastpickle ! com
[Download RAW message or body]
> I ended up changing memtable_flush_queue_size to be large enough to contain the \
> biggest flood I saw.
As part of the flush process the “Switch Lock” is taken to synchronise around the \
commit log. This is a reentrant Read Write lock, the flush path takes the write lock \
and write path takes the read part. When flushing a CF the write lock is taken, the \
commit log is updated, and memtable is added to the flush queue. If the queue is full \
then the write lock will be held blocking the write threads from taking the read \
lock.
There are a few reasons why the queue may be full, the simple one is the disk IO is \
not fast enough. Others are that the commit log segments are too small, there are \
lots of CF’s and/or lots of secondary indexes, or nodetoo flush is called frequently. \
Increasing the size of the queue is a good work around, and the correct approach if \
you have a lot of CF’s and/or secondary indexes.
Hope that helps.
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 21/12/2013, at 6:03 am, Ken Hancock <ken.hancock@schange.com> wrote:
> I ended up changing memtable_flush_queue_size to be large enough to contain the \
> biggest flood I saw.
> I monitored tpstats over time using a collection script and an analysis script that \
> I wrote to figure out what my largest peaks were. In my case, all my mutation \
> drops correlated with hitting the maximum memtable_flush_queue_size and then \
> mutations drops stopped as soon as the queue size dropped below the max.
> I threw the scripts up on github in case they're useful...
>
> https://github.com/hancockks/tpstats
>
>
>
>
> On Fri, Dec 20, 2013 at 1:08 AM, Alexander Shutyaev <shutyaev@gmail.com> wrote:
> Thanks for you answers.
>
> srmore,
>
> We are using v2.0.0. As for GC I guess it does not correlate in our case, because \
> we had cassandra running 9 days under production load and no dropped messages and I \
> guess that during this time there were a lot of GCs.
> Ken,
>
> I've checked the values you indicated. Here they are:
>
> node1 6498
> node2 6476
> node3 6642
>
> I guess this is not good :) What can we do to fix this problem?
>
>
> 2013/12/19 Ken Hancock <ken.hancock@schange.com>
> We had issues where the number of CF families that were being flushed would align \
> and then block writes for a very brief period. If that happened when a bunch of \
> writes came in, we'd see a spike in Mutation drops.
> Check nodetool tpstats for FlushWriter all time blocked.
>
>
> On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev <shutyaev@gmail.com> wrote:
> Hi all!
>
> We've had a problem with cassandra recently. We had 2 one-minute periods when we \
> got a lot of timeouts on the client side (the only timeouts during 9 days we are \
> using cassandra in production). In the logs we've found corresponding messages \
> saying something about MUTATION messages dropped.
> Now, the official faq [1] says that this is an indicator that the load is too high. \
> We've checked our monitoring and found out that 1-minute average cpu load had a \
> local peak at the time of the problem, but it was like 0.8 against 0.2 usual which \
> I guess is nothing for a 2 core virtual machine. We've also checked java threads - \
> there was no peak there and their count was reasonable ~240-250.
> Can anyone give us a hint - what should we monitor to see this "high load" and what \
> should we tune to make it acceptable?
> Thanks in advance,
> Alexander
>
> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>
>
>
> --
> Ken Hancock | System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
> Office: +1 (978) 889-3329 | ken.hancock@schange.com | hancockks | hancockks
>
>
> This e-mail and any attachments may contain information which is SeaChange \
> International confidential. The information enclosed is intended only for the \
> addressees herein and may not be copied or forwarded without permission from \
> SeaChange International.
>
>
>
> --
> Ken Hancock | System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
> Office: +1 (978) 889-3329 | ken.hancock@schange.com | hancockks | hancockks
>
>
> This e-mail and any attachments may contain information which is SeaChange \
> International confidential. The information enclosed is intended only for the \
> addressees herein and may not be copied or forwarded without permission from \
> SeaChange International.
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space;"><blockquote type="cite"><div \
dir="ltr">I ended up changing memtable_flush_queue_size to be large enough to contain \
the biggest flood I saw.<br></div></blockquote><div>As part of the flush process the \
“Switch Lock” is taken to synchronise around the commit log. This is a reentrant Read \
Write lock, the flush path takes the write lock and write path takes the read part. \
When flushing a CF the write lock is taken, the commit log is updated, and memtable \
is added to the flush queue. If the queue is full then the write lock will be held \
blocking the write threads from taking the read \
lock. </div><div><br></div><div>There are a few reasons why the queue may be \
full, the simple one is the disk IO is not fast enough. Others are that the commit \
log segments are too small, there are lots of CF’s and/or lots of secondary indexes, \
or nodetoo flush is called frequently. </div><div><br></div><div>Increasing the \
size of the queue is a good work around, and the correct approach if you have a lot \
of CF’s and/or secondary indexes. </div><div><br></div><div>Hope that \
helps.</div><div><br></div><br><div apple-content-edited="true"> <div style="color: \
rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; \
font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; \
text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: \
normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; \
-webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space; "><div>-----------------</div><div>Aaron \
Morton</div><div>New \
Zealand</div><div>@aaronmorton</div><div><br></div><div>Co-Founder & Principal \
Consultant</div><div>Apache Cassandra Consulting</div><div><a \
href="http://www.thelastpickle.com">http://www.thelastpickle.com</a></div></div> \
</div> <br><div><div>On 21/12/2013, at 6:03 am, Ken Hancock <<a \
href="mailto:ken.hancock@schange.com">ken.hancock@schange.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div><div>I \
ended up changing memtable_flush_queue_size to be large enough to contain the biggest \
flood I saw.<br><br></div>I monitored tpstats over time using a collection script and \
an analysis script that I wrote to figure out what my largest peaks were. In my \
case, all my mutation drops correlated with hitting the maximum \
memtable_flush_queue_size and then mutations drops stopped as soon as the queue size \
dropped below the max.<br>
<br></div>I threw the scripts up on github in case they're useful...<br><br><a \
href="https://github.com/hancockks/tpstats">https://github.com/hancockks/tpstats</a><br><div>
<br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On \
Fri, Dec 20, 2013 at 1:08 AM, Alexander Shutyaev <span dir="ltr"><<a \
href="mailto:shutyaev@gmail.com" target="_blank">shutyaev@gmail.com</a>></span> \
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Thanks for you \
answers.<div><br></div><div><b>srmore</b>,</div><div><br></div><div>We are using \
v2.0.0. As for GC I guess it does not correlate in our case, because we had cassandra \
running 9 days under production load and no dropped messages and I guess that during \
this time there were a lot of GCs.</div>
<div><br></div><div><b>Ken</b>,</div><div><br></div><div>I've checked the values you \
indicated. Here they are:</div><div><br></div><div>node1 \
6498</div><div>node2 6476</div> <div>node3 \
6642</div><div><br></div><div>I guess this is not good :) What can we do to fix this \
problem?</div></div><div class="gmail_extra"><br><br><div \
class="gmail_quote">2013/12/19 Ken Hancock <span dir="ltr"><<a \
href="mailto:ken.hancock@schange.com" \
target="_blank">ken.hancock@schange.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">We had issues where the number of CF families \
that were being flushed would align and then block writes for a very brief period. If \
that happened when a bunch of writes came in, we'd see a spike in Mutation drops.<br>
<br>Check nodetool tpstats for FlushWriter all time blocked.<br></div><div \
class="gmail_extra"><br><br><div class="gmail_quote"><div>On Thu, Dec 19, 2013 at \
7:12 AM, Alexander Shutyaev <span dir="ltr"><<a href="mailto:shutyaev@gmail.com" \
target="_blank">shutyaev@gmail.com</a>></span> wrote:<br>
</div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr">Hi all!<div><br></div><div>We've had a \
problem with cassandra recently. We had 2 one-minute periods when we got a lot of \
timeouts on the client side (the only timeouts during 9 days we are using cassandra \
in production). In the logs we've found corresponding messages saying something about \
MUTATION messages dropped.</div>
<div><br></div><div>Now, the official faq [1] says that this is an indicator that the \
load is too high. We've checked our monitoring and found out that 1-minute average \
cpu load had a local peak at the time of the problem, but it was like 0.8 against 0.2 \
usual which I guess is nothing for a 2 core virtual machine. We've also checked java \
threads - there was no peak there and their count was reasonable ~240-250.</div>
<div><br></div><div>Can anyone give us a hint - what should we monitor to see this \
"high load" and what should we tune to make it \
acceptable?</div><div><br></div><div>Thanks in advance,</div> \
<div>Alexander</div><div><br></div><div>[1] <a \
href="http://wiki.apache.org/cassandra/FAQ#dropped_messages" \
target="_blank">http://wiki.apache.org/cassandra/FAQ#dropped_messages</a></div></div> \
</blockquote></div></div><span><font color="#888888"><br><br clear="all"><span \
class="HOEnZb"><font color="#888888"><br>-- <br><table \
style="font-size:11px;font-family:Verdana,Arial,Helvetica,sans-serif" border="0" \
cellpadding="0" cellspacing="0" height="100" width="500">
<tbody><tr>
<td style="font-family:Arial,Helvetica,sans-serif;font-size:3mm;line-height:4mm;color:rgb(51,51,51)" \
bgcolor="#FFFFFF" valign="middle"><b><b><font color="#000066">Ken \
Hancock </font></b></b>| System Architect, Advanced Advertising <br>
<span style="color:rgb(0,0,102);font-weight:bold">SeaChange \
International </span><br>50 Nagog Park<br>Acton, Massachusetts 01720<br><a \
href="mailto:ken.hancock@schange.com" \
target="_blank">ken.hancock@schange.com</a> | <a \
href="http://www.schange.com/" target="_blank">www.schange.com</a> | NASDAQ:<a \
href="http://www.schange.com/en-US/Company/InvestorRelations.aspx" \
target="_blank">SEAC</a> <br>
Office: <a href="tel:%2B1%20%28978%29%20889-3329" value="+19788893329" \
target="_blank">+1 (978) 889-3329</a> | <img alt="Google Talk:"> <a \
href="mailto:ken.hancock@schange.com" \
target="_blank">ken.hancock@schange.com</a> | <img \
alt="Skype:">hancockks | <img alt="Yahoo IM:">hancockks</td>
<td valign="top"><a href="http://www.linkedin.com/in/kenhancock" target="_blank"><img \
alt="LinkedIn"></a><br></td></tr><tr><td valign="middle"><a \
href="http://www.schange.com/" target="_blank"><br>
<img alt="SeaChange International" border="0" height="62" hspace="0" vspace="0" \
width="348"><br>
</a></td></tr><tr><td \
style="font-family:Arial,Helvetica,sans-serif;font-size:3mm;line-height:4mm;color:rgb(51,51,51)" \
align="left" valign="top">This e-mail and any attachments may contain information \
which is SeaChange International confidential. The information enclosed is intended \
only for the addressees herein and may not be copied or forwarded without permission \
from SeaChange International.</td>
</tr></tbody></table>
</font></span></font></span></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br>-- <br><table style="font-family: Verdana, \
Arial, Helvetica, sans-serif; font-size: 11px;" border="0" cellpadding="0" \
cellspacing="0" height="100" width="500"><tbody><tr>
<td style="font-family:Arial,Helvetica,sans-serif;font-size:3mm;line-height:4mm;color:rgb(51,51,51)" \
bgcolor="#FFFFFF" valign="middle"><b><b><font color="#000066">Ken \
Hancock </font></b></b>| System Architect, Advanced Advertising <br>
<span style="color:rgb(0,0,102);font-weight:bold">SeaChange \
International </span><br>50 Nagog Park<br>Acton, Massachusetts 01720<br><a \
href="mailto:ken.hancock@schange.com" \
target="_blank">ken.hancock@schange.com</a> | <a \
href="http://www.schange.com/" target="_blank">www.schange.com</a> | NASDAQ:<a \
href="http://www.schange.com/en-US/Company/InvestorRelations.aspx" \
target="_blank">SEAC</a> <br>
Office: +1 (978) 889-3329 | <img \
src="https://s3.amazonaws.com/images.wisestamp.com/gtalk.png" alt="Google \
Talk:"> <a href="mailto:ken.hancock@schange.com" \
target="_blank">ken.hancock@schange.com</a> | <img \
src="https://s3.amazonaws.com/images.wisestamp.com/skype.png" \
alt="Skype:">hancockks | <img \
src="https://s3.amazonaws.com/images.wisestamp.com/yahoo.png" alt="Yahoo \
IM:">hancockks</td>
<td valign="top"><a href="http://www.linkedin.com/in/kenhancock" target="_blank"><img \
src="https://s3.amazonaws.com/images.wisestamp.com/linkedin.png" \
alt="LinkedIn"></a><br></td></tr><tr><td valign="middle"><a \
href="http://www.schange.com/" target="_blank"><br>
<img src="http://www.schange.com/Images/Emails/Signatures/SC_email_Sig_11_2010" \
alt="SeaChange International" \
longdesc="http://www.schange.com/Images/Emails/Signatures/SC_email_Sig_11_2010" \
border="0" height="62" hspace="0" vspace="0" width="348"><br>
</a></td></tr><tr><td \
style="font-family:Arial,Helvetica,sans-serif;font-size:3mm;line-height:4mm;color:rgb(51,51,51)" \
align="left" valign="top">This e-mail and any attachments may contain information \
which is SeaChange International confidential. The information enclosed is intended \
only for the addressees herein and may not be copied or forwarded without permission \
from SeaChange International.</td>
</tr></tbody></table>
</div>
</blockquote></div><br></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic