[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Unexpected high internode network activity
From: Gianluca Borello <gianluca () sysdig ! com>
Date: 2016-02-26 18:42:24
Message-ID: CAJjpQyR+ts8=psQR7hc2nbiYPSDrV+hOR3BwFe3X4-FHCVWFPw () mail ! gmail ! com
[Download RAW message or body]
Thank you for your reply.
- Repairs are not running on the cluster, in fact we've been "slacking"
when it comes to repair, mainly because we never manually delete our data
as it's always TTLed and we haven't had major failures or outages that
required repairing data (I know that's not a good reason anyway)
- We are not using server-to-server encryption
- internode_compression is set to all, and the application driver is lz4
- I just did a "nodetool flush && service cassandra restart" on one node of
the affected cluster and let it run for a few minutes, and these are the
statistics (all the nodes get the same ratio of network activity on port
9042 and port 7000, so pardon my raw estimates below in assuming that the
activity of a single node can reflect the activity of the whole cluster):
9042 traffic: 400 MB (split between 200 MB reads and 200 MB writes)
7000 traffic: 5 GB (counted twice by iftop, so 2.5 GB)
$ nodetool netstats -H
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 10167
Mismatch (Blocking): 210
Mismatch (Background): 151
Pool Name Active Pending Completed
Commands n/a 0 422986
Responses n/a 0 403144
If I do the same on a test cluster (with less activity and nodes but same
RF and configuration), I get, always for a single node:
9042 traffic: 250 MB (split between 100 MB reads and 150 MB writes)
7000 traffic: 1 GB (counted twice by iftop, so 500 MB)
$ nodetool netstats -H
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 6668
Mismatch (Blocking): 159
Mismatch (Background): 43
Pool Name Active Pending Completed
Commands n/a 0 125202
Responses n/a 0 141708
So, once again, in one cluster the internode activity is ~7 times the 9042
one, whereas in the test one is ~2, which is expected.
Thanks
On Fri, Feb 26, 2016 at 10:04 AM, Nate McCall <nate@thelastpickle.com>
wrote:
>
>> Unfortunately, these numbers still don't match at all.
>>
>> And yes, the cluster is in a single DC and since I am using the EC2
>> snitch, replicas are AZ aware.
>>
>>
> Are repairs running on the cluster?
>
> Other thoughts:
> - is internode_compression set to 'all' in cassandra.yaml (should be 'all'
> by default, but worth checking since you are using lz4 on the client)?
> - are you using server-to-server encryption ?
>
> You can compare the output of nodetool netstats on the test cluster with
> the AWS cluster as well to see if anything sticks out.
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
[Attachment #3 (text/html)]
<div dir="ltr">Thank you for your reply.<div><br></div><div>- Repairs are not running \
on the cluster, in fact we've been "slacking" when it comes to repair, \
mainly because we never manually delete our data as it's always TTLed and we \
haven't had major failures or outages that required repairing data (I know \
that's not a good reason anyway)</div><div><br></div><div>- We are not using \
server-to-server encryption</div><div><br></div><div>- internode_compression is set \
to all, and the application driver is lz4</div><div><br></div><div>- I just did a \
"nodetool flush && service cassandra restart" on one node of the \
affected cluster and let it run for a few minutes, and these are the statistics (all \
the nodes get the same ratio of network activity on port 9042 and port 7000, so \
pardon my raw estimates below in assuming that the activity of a single node can \
reflect the activity of the whole cluster):</div><div><br></div><div>9042 traffic: \
400 MB (split between 200 MB reads and 200 MB writes)</div><div>7000 traffic: 5 GB \
(counted twice by iftop, so 2.5 GB)</div><div><br></div><div>$ nodetool netstats \
-H</div><div>Mode: NORMAL<br></div><div><div>Not sending any streams.</div><div>Read \
Repair Statistics:</div><div>Attempted: 10167</div><div>Mismatch (Blocking): \
210</div><div>Mismatch (Background): 151</div><div>Pool Name \
Active Pending Completed</div><div>Commands \
n/a 0 422986</div><div>Responses \
n/a 0 403144</div></div><div><br></div><div>If I do the same \
on a test cluster (with less activity and nodes but same RF and configuration), I \
get, always for a single node:</div><div><br></div><div><div>9042 traffic: 250 MB \
(split between 100 MB reads and 150 MB writes)</div><div>7000 traffic: 1 GB (counted \
twice by iftop, so 500 MB)</div></div><div><br></div><div><div>$ nodetool netstats \
-H</div><div>Mode: NORMAL</div><div>Not sending any streams.</div><div>Read Repair \
Statistics:</div><div>Attempted: 6668</div><div>Mismatch (Blocking): \
159</div><div>Mismatch (Background): 43</div><div>Pool Name \
Active Pending Completed</div><div>Commands \
n/a 0 125202</div><div>Responses \
n/a 0 141708</div></div><div><br></div><div>So, once again, \
in one cluster the internode activity is ~7 times the 9042 one, whereas in the test \
one is ~2, which is expected.</div><div><br></div><div>Thanks</div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 26, 2016 at 10:04 AM, \
Nate McCall <span dir="ltr"><<a href="mailto:nate@thelastpickle.com" \
target="_blank">nate@thelastpickle.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div \
class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0px \
0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br></div><div>Unfortunately, \
these numbers still don't match at all. </div><div><br></div><div>And yes, the \
cluster is in a single DC and since I am using the EC2 snitch, replicas are AZ aware. \
</div><div><br></div></blockquote><div><br></div></span><div>Are repairs running on \
the cluster?</div><div class="gmail_quote"><br></div>Other thoughts:<br>- is \
internode_compression set to 'all' in cassandra.yaml (should be 'all' \
by default, but worth checking since you are using lz4 on the client)? </div><div \
class="gmail_quote">- are you using server-to-server encryption ?<br><br></div><div \
class="gmail_quote">You can compare the output of nodetool netstats on the test \
cluster with the AWS cluster as well to see if anything sticks out. </div><span \
class="HOEnZb"><font color="#888888"><div class="gmail_quote"> \
</div><div><br></div>-- <br><div><div dir="ltr">-----------------<br>Nate \
McCall<br>Austin, TX<br>@zznate<br><br>Co-Founder & Sr. Technical \
Consultant<br>Apache Cassandra Consulting<br><a href="http://www.thelastpickle.com" \
target="_blank">http://www.thelastpickle.com</a></div></div> \
</font></span></div></div> </blockquote></div><br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic