[prev in list] [next in list] [prev in thread] [next in thread] 

List:       cassandra-user
Subject:    Re: Hintedhandoff will never complete when a BIG rowmutation
From:       albert_e <dongzz82 () gmail ! com>
Date:       2010-06-28 16:31:05
Message-ID: AANLkTik8u3PVQrk8iK-K1KyAGk1QdZCXpWf_a6ckWK1T () mail ! gmail ! com
[Download RAW message or body]

In 0.6.2, HH sending MUTATION message using the same OutboundTcpConnection
with READ message. When HH transfering big mutation data, read operation
will be blocked and read storm may cause 100% disk I/O of the dest node.


2010/6/28 Jonathan Ellis <jbellis@gmail.com>

> Yes, you should increase your timeout if you are hinting big mutations
> (or big rows that were built from smaller mutations).
>
> 2010/6/28 Lu Ming <xluke@live.com>:
> > Hi:
> >      These days I found my Cassandra is strange, much slower than before.
> > And I Spent much time to figure it out and today I got the answer.
> >
> >     Some bad buy keeps on writing many data day and night, then made a
> very
> > big row mutation which size is about 140M.
> > In this period I restarted some Cassandra nodes, and when the nodes is
> alive
> > again, them got some hintedhandoff messages.
> > HintedHandOffManager.sendMessage() will send the rowmutations to these
> > nodes, but the rowmutation is too big to finish transferring in
> > 8 seconds (defined in DatabaseDescriptor.getRpcTimeout()), and
> sendMessage()
> > return false when got a TimeoutException.
> >
> > Every one hour HintedHandOffManager will check hintedhandoff ColumnFamily
> > then send out the big rowmutations to alive nodes,
> > It fails again because of the TimeoutException, so the task will never
> > finish and the big rowmutation is sending again and again.
> >
> >    In multi-datacenters,  a big rowmutation can not be transferred
> > in several seconds. so It is a potential risk when  a big
> > rowmutation occurs.
> >
> >
> >
> > Luke
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

[Attachment #3 (text/html)]

In 0.6.2, HH sending MUTATION message using the same OutboundTcpConnection 
with READ message. When HH transfering big 
mutation data, read operation will be blocked and read storm may cause 100% disk I/O \
of the dest node.  <br><br><br><div class="gmail_quote">2010/6/28 Jonathan Ellis \
<span dir="ltr">&lt;<a \
href="mailto:jbellis@gmail.com">jbellis@gmail.com</a>&gt;</span><br> <blockquote \
class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, \
204, 204); padding-left: 1ex;">Yes, you should increase your timeout if you are \
hinting big mutations<br> (or big rows that were built from smaller mutations).<br>
<br>
2010/6/28 Lu Ming &lt;<a href="mailto:xluke@live.com">xluke@live.com</a>&gt;:<br>
<div><div></div><div class="h5">&gt; Hi:<br>
&gt;      These days I found my Cassandra is strange, much slower than before.<br>
&gt; And I Spent much time to figure it out and today I got the answer.<br>
&gt;<br>
&gt;     Some bad buy keeps on writing many data day and night, then made a very<br>
&gt; big row mutation which size is about 140M.<br>
&gt; In this period I restarted some Cassandra nodes, and when the nodes is alive<br>
&gt; again, them got some hintedhandoff messages.<br>
&gt; HintedHandOffManager.sendMessage() will send the rowmutations to these<br>
&gt; nodes, but the rowmutation is too big to finish transferring in<br>
&gt; 8 seconds (defined in DatabaseDescriptor.getRpcTimeout()), and sendMessage()<br>
&gt; return false when got a TimeoutException.<br>
&gt;<br>
&gt; Every one hour HintedHandOffManager will check hintedhandoff ColumnFamily<br>
&gt; then send out the big rowmutations to alive nodes,<br>
&gt; It fails again because of the TimeoutException, so the task will never<br>
&gt; finish and the big rowmutation is sending again and again.<br>
&gt;<br>
&gt;    In multi-datacenters,  a big rowmutation can not be transferred<br>
&gt; in several seconds. so It is a potential risk when  a big<br>
&gt; rowmutation occurs.<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; Luke<br>
<br>
<br>
<br>
</div></div><font color="#888888">--<br>
Jonathan Ellis<br>
Project Chair, Apache Cassandra<br>
co-founder of Riptano, the source for professional Cassandra support<br>
<a href="http://riptano.com" target="_blank">http://riptano.com</a><br>
</font></blockquote></div><br>



[prev in list] [next in list] [prev in thread] [next in thread]