[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Read latency skyrockets after 95-98th percentile
From: Andras Szerdahelyi <andras.szerdahelyi () ignitionone ! com>
Date: 2012-07-31 14:09:18
Message-ID: FC6A8AE3-668E-4507-9395-6ACA9551A53E () dentsunetwork ! com
[Download RAW message or body]
Victor, the data is a single composite column <String, String, Integer> with a string \
value. The key is a UTF8 string I can reproduce this exact behavior with only 100 of \
these items on a 2GB heap.
Also, please see -XX:+PrintGCDetails output below, i'm not convinced it's GC. With a \
100 items no garbage collection is logged at all.
here's the GC output during the read
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 281480 bytes, 281480 total
> 17322K->308K(19136K), 0.0037375 secs] 28742K->11885K(83008K), 0.0038170 secs] \
> [Times: user=0.01 sys=0.00, real=0.01 secs]
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 261360 bytes, 261360 total
> 17332K->386K(19136K), 0.0034069 secs] 28909K->12098K(83008K), 0.0034849 secs] \
> [Times: user=0.00 sys=0.01, real=0.00 secs]
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 241336 bytes, 241336 total
> 17410K->383K(19136K), 0.0035252 secs] 29122K->12209K(83008K), 0.0036062 secs] \
> [Times: user=0.00 sys=0.00, real=0.00 secs]
thanks,
Andras
On 31 Jul 2012, at 16:04, Viktor Jevdokimov wrote:
What is a data load? Does it fits in RAM?
I bet it’s due to GC. Since this is a 1 node only, dynamic snitch is out of scope.
Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com<mailto:Viktor.Jevdokimov@adform.com>
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
What is Adform: watch this short video<http://vimeo.com/adform/display>
<signature-logo29.png><http://www.adform.com>
Disclaimer: The information contained in this message and attachments is intended \
solely for the attention and use of the named addressee and may be confidential. If \
you are not the intended recipient, you are reminded that the information remains the \
property of the sender. You must not use, disclose, distribute, copy, print or rely \
on this e-mail. If you have received this message in error, please contact the sender \
immediately and irrevocably delete this message and any copies.
From: Andras Szerdahelyi [mailto:andras.szerdahelyi@ignitionone.com]
Sent: Tuesday, July 31, 2012 15:53
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Read latency skyrockets after 95-98th percentile
hey list,
i've been trying to understand why we are seeing rather nasty read latency peaks in \
as much as 2% of our total read ( not sure what the underlying thrift call is, should \
be get_slice ) requests that we have been **unable to tie to garbage collection or \
blocking I/O.** This is what i mean by "nasty" :
Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.0242499999999999, 95th: \
2.1103499999999977, 98th: 5.2102600000000034, 99th: 13.24835000000001, 999th: \
41.89454000000041
This is measured around the Astyanax / Hector wrappers for running column slice \
queries and iterating over their results. FWIW they perform very similarly.
Now, the mean is awesome - 1.2ms for a read is really really fast, i think thrift \
overhead makes up for probably 1ms alone. What i can't live with is pretty much \
everything thats beyond the 98th percentile. Perhaps my expectations are misaligned \
to begin with and this is great, but the dataset under test that yielded this result \
( see below ) is extremely small - only a 1000 items per test made up of a few small \
dynamic composite columns and the read is being done sequentially, from a single \
threaded client. I just don't understand whats causing the peaks.
Here's the complete run-down on the setup:
The data is dynamic composite columns with string or integer values. The \
DynamicComposites themselves are made up of 3 Components each: \
<String,String,Integer> and <String,Integer,Integer>
The query is getting all columns ( in this particular test, that will be a single \
column ) of one of the composite types ( <String, String, Integer> or <String, \
Integer, Integer> ) for a particular string key. ( How this is achieved is more or \
less at https://gist.github.com/3207894 )
The production cluster is set up as follows:
30-40 read/sec clients from multiple hosts, NetworkTopologyStrategy, 2 replicas per \
DC, read consistency: ONE, 2DCs, 2 nodes per DC ) 8GB multi-core physical hosts
most settings on default in cassandra.yaml
In an attempt to isolate the problem, i've done the following ( this is where the \
results above are coming from but we see roughly the same numbers in prod )
- created a short unit test around the ( two ) classes that wrap the Astyanax and \
Hector libraries in our application, in which i'm only testing the \
read performance
- in setUp() i start Cassandra with
CassandraStoreKeeperTest.cassandra = new CassandraDaemon();
CassandraStoreKeeperTest.cassandra.init(null);
CassandraStoreKeeperTest.cassandra.start();
- same cassandra yaml as in prod, file at https://gist.github.com/3207894 ( mostly \
defaults )
- i pass the following flags to the JVM running my unit test: -server -Xmx2G and all \
the GC flags from cassandra-env.sh, plus the jamm java agent
- i warm up the embedded cassandra with 1k reads and 1k writes
- the test is 1000 single threaded, sequential reads via Hector and Astyanax ( no \
significant differences between the two ) via thrift to the embedded single-node \
Cassandra instance
- i read back the 1000 items saved in the warm-up
The results are
Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.0242499999999999, 95th: \
2.1103499999999977, 98th: 5.2102600000000034, 99th: 13.24835000000001, 999th: \
41.89454000000041
here's the GC output during the read
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 281480 bytes, 281480 total
> 17322K->308K(19136K), 0.0037375 secs] 28742K->11885K(83008K), 0.0038170 secs] \
> [Times: user=0.01 sys=0.00, real=0.01 secs]
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 261360 bytes, 261360 total
> 17332K->386K(19136K), 0.0034069 secs] 28909K->12098K(83008K), 0.0034849 secs] \
> [Times: user=0.00 sys=0.01, real=0.00 secs]
[GC [ParNew
Desired survivor size 1081344 bytes, new threshold 1 (max 1)
- age 1: 241336 bytes, 241336 total
> 17410K->383K(19136K), 0.0035252 secs] 29122K->12209K(83008K), 0.0036062 secs] \
> [Times: user=0.00 sys=0.00, real=0.00 secs]
Doesn't look like GC ( i could be wrong ) - what else could it be ?
all suggestions are highly appreciated..
Thanks!
Andras
ps: this is with Cassandra 1.1.0
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<base href="x-msg://142/">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; "> <div>Victor, the data is a single composite column <String, \
String, Integer> with a string value. The key is a UTF8 string</div> <div>I can \
reproduce this exact behavior with only 100 of these items on a 2GB heap.</div> \
<div><br> </div>
<div>Also, please see <span class="Apple-style-span" style="font-family: Monaco; \
font-size: 11px; ">-XX:+PrintGCDetails</span> output below, i'm not \
convinced it's GC. With a 100 items no garbage collection is logged at all.</div> \
<div><br> </div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1" style="page: WordSection1; ">
<div style="border-top-style: none; border-right-style: none; border-bottom-style: \
none; border-width: initial; border-color: initial; border-left-style: solid; \
border-left-color: blue; border-left-width: 1.5pt; padding-top: 0cm; padding-right: \
0cm; padding-bottom: 0cm; padding-left: 4pt; position: static; z-index: auto; "> \
<div> <div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> here's the GC \
output during the read<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 281480 bytes, 281480 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17322K->308K(19136K), 0.0037375 secs] 28742K->11885K(83008K), 0.0038170 secs] \
[Times: user=0.01 sys=0.00, real=0.01 secs] <o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 261360 bytes, 261360 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17332K->386K(19136K), 0.0034069 secs] 28909K->12098K(83008K), 0.0034849 secs] \
[Times: user=0.00 sys=0.01, real=0.00 secs] <o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 241336 bytes, 241336 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17410K->383K(19136K), 0.0035252 secs] 29122K->12209K(83008K), 0.0036062 secs] \
[Times: user=0.00 sys=0.00, real=0.00 secs] </div> </div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>thanks,</div>
<div>Andras</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On 31 Jul 2012, at 16:04, Viktor Jevdokimov wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1" style="page: WordSection1; ">
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); \
">What is a data load? Does it fits in RAM?<o:p></o:p></span></div> <div \
style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; \
font-size: 12pt; font-family: 'Times New Roman', serif; "> <span style="font-size: \
11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); \
"><o:p> </o:p></span></div> <div style="margin-top: 0cm; margin-right: 0cm; \
margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New \
Roman', serif; "> <span style="font-size: 11pt; font-family: Calibri, sans-serif; \
color: rgb(31, 73, 125); ">I bet it’s due to GC. Since this is a 1 node only, dynamic \
snitch is out of scope.<o:p></o:p></span></div> <div style="margin-top: 0cm; \
margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; \
font-family: 'Times New Roman', serif; "> <span style="font-size: 11pt; font-family: \
Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div> <div \
style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; \
font-size: 12pt; font-family: 'Times New Roman', serif; "> <span style="font-size: \
11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); \
"><o:p></o:p></span></div> <div style="margin-top: 0cm; margin-right: 0cm; \
margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New \
Roman', serif; "> <span style="font-size: 11pt; font-family: Calibri, sans-serif; \
color: rgb(31, 73, 125); "><o:p> </o:p></span></div> <br>
<br>
<div class="signature-container" style="padding-top: 0px; padding-right: 5px; \
padding-bottom: 0px; padding-left: 5px; font-family: Arial, sans-serif; font-size: \
11px; line-height: 12px; color: rgb(91, 91, 91); "> <table class="signature" \
style="border-bottom-color: rgb(214, 214, 214); border-bottom-width: 1px; \
border-bottom-style: solid; border-top-color: rgb(214, 214, 214); border-top-width: \
1px; border-top-style: solid; width: 1077px; border-collapse: collapse; "> <tbody>
<tr>
<td style="font-family: Arial, sans-serif; font-size: 11px; line-height: 12px; color: \
rgb(91, 91, 91); vertical-align: top; padding-top: 20px; padding-right: 0px; \
padding-bottom: 20px; padding-left: 0px; "> <div class="best-regards" style="color: \
rgb(173, 173, 173); margin-bottom: 15px; "> Best regards / Pagarbiai</div>
<div><strong>Viktor Jevdokimov</strong></div>
<div>Senior Developer</div>
<div></div>
<br>
<div>Email:<span class="Apple-converted-space"> </span><a \
href="mailto:Viktor.Jevdokimov@adform.com" style="color: blue; text-decoration: \
underline; ">Viktor.Jevdokimov@adform.com</a></div> <div>Phone: +370 5 212 3063, \
Fax +370 5 261 0453</div> <div>J. Jasinskio 16C, LT-01112 Vilnius, \
Lithuania</div> <div>Follow us on Twitter:<span \
class="Apple-converted-space"> </span><a title="@adforminsider" \
href="http://twitter.com/#!/adforminsider" style="color: blue; text-decoration: \
underline; ">@adforminsider</a></div> <div>What is Adform:<span \
class="Apple-converted-space"> </span><a title="What is Adform" \
href="http://vimeo.com/adform/display" style="color: blue; text-decoration: \
underline; ">watch this short video</a></div> </td>
<td class="right" style="font-family: Arial, sans-serif; font-size: 11px; \
line-height: 12px; color: rgb(91, 91, 91); text-align: right; vertical-align: top; \
padding-top: 20px; padding-right: 0px; padding-bottom: 20px; padding-left: 0px; "> \
<div><a href="http://www.adform.com" title="Adform News" style="color: blue; \
text-decoration: underline; "><span><signature-logo29.png></span></a></div> \
</td> </tr>
</tbody>
</table>
<div class="disclaimer" style="color: rgb(173, 173, 173); font-size: 9px; \
line-height: 11px; "> <br>
Disclaimer: The information contained in this message and attachments is intended \
solely for the attention and use of the named addressee and may be confidential. If \
you are not the intended recipient, you are reminded that the information remains the \
property of the sender. You must not use, disclose, distribute, copy, print or rely \
on this e-mail. If you have received this message in error, please contact the sender \
immediately and irrevocably delete this message and any copies.</div> </div>
<br>
<div style="border-top-style: none; border-right-style: none; border-bottom-style: \
none; border-width: initial; border-color: initial; border-left-style: solid; \
border-left-color: blue; border-left-width: 1.5pt; padding-top: 0cm; padding-right: \
0cm; padding-bottom: 0cm; padding-left: 4pt; position: static; z-index: auto; "> \
<div> <div style="border-right-style: none; border-bottom-style: none; \
border-left-style: none; border-width: initial; border-color: initial; \
border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; \
padding-top: 3pt; padding-right: 0cm; padding-bottom: 0cm; padding-left: 0cm; "> <div \
style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; \
font-size: 12pt; font-family: 'Times New Roman', serif; "> <b><span style="font-size: \
10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: \
10pt; font-family: Tahoma, sans-serif; "><span \
class="Apple-converted-space"> </span>Andras Szerdahelyi \
[mailto:andras.szerdahelyi@ignitionone.com]<span \
class="Apple-converted-space"> </span><br> <b>Sent:</b><span \
class="Apple-converted-space"> </span>Tuesday, July 31, 2012 15:53<br> \
<b>To:</b><span class="Apple-converted-space"> </span><a \
href="mailto:user@cassandra.apache.org" style="color: blue; text-decoration: \
underline; ">user@cassandra.apache.org</a><br> <b>Subject:</b><span \
class="Apple-converted-space"> </span>Read latency skyrockets after 95-98th \
percentile<o:p></o:p></span></div> </div>
</div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> <div style="margin-top: 0cm; margin-right: 0cm; margin-left: \
0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; \
"> hey list,<o:p></o:p></div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> i've been trying \
to understand why we are seeing rather nasty read latency peaks in as much as 2% of \
our total read ( not sure what the underlying thrift call is, should be \
get_slice ) requests that we have been **unable to tie to garbage collection or \
blocking I/O.** <o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> This is what i \
mean by "nasty" : <o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.0242499999999999, 95th: \
2.1103499999999977, 98th: 5.2102600000000034, 99th: 13.24835000000001, 999th: \
41.89454000000041<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> This is measured \
around the Astyanax / Hector wrappers for running column slice queries and iterating \
over their results. FWIW they perform very similarly.<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Now, the mean is \
awesome - 1.2ms for a read is really really fast, i think thrift overhead makes up \
for probably 1ms alone. What i can't live with is pretty much everything thats beyond \
the 98th percentile. Perhaps my expectations are misaligned to begin with and this \
is great, but the dataset under test that yielded this result ( see below ) is \
extremely small - only a 1000 items per test made up of a few small dynamic composite \
columns and the read is being done sequentially, from a single threaded client. \
I just don't understand whats causing the peaks.<o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Here's the \
complete run-down on the setup:<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> The data is \
dynamic composite columns with string or integer values. The DynamicComposites \
themselves are made up of 3 Components each: <String,String,Integer> and \
<String,Integer,Integer><o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> The query is \
getting all columns ( in this particular test, that will be a single column ) of \
one of the composite types ( <String, String, Integer> or <String, \
Integer, Integer> ) for a particular string key.<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> ( How this is \
achieved is more or less at <a href="https://gist.github.com/3207894" \
style="color: blue; text-decoration: underline; \
">https://gist.github.com/3207894</a><span \
class="Apple-converted-space"> </span>)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> The production \
cluster is set up as follows:<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> 30-40 read/sec \
clients from multiple hosts, NetworkTopologyStrategy, 2 replicas per DC, read \
consistency: ONE, 2DCs, 2 nodes per DC )<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> 8GB multi-core \
physical hosts<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> most settings on \
default in cassandra.yaml<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> In an attempt to \
isolate the problem, i've done the following ( this is where the results above are \
coming from but we see roughly the same numbers in prod )<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- created a short unit test around the ( two ) classes that wrap the Astyanax and \
Hector libraries in our application, in which i'm only testing the read \
performance<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- in setUp() i start Cassandra with <o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 8.5pt; font-family: Monaco, serif; ">CassandraStoreKeeperTest.<span \
style="color: rgb(0, 39, 204); ">cassandra</span><span \
class="Apple-converted-space"> </span>=<span \
class="Apple-converted-space"> </span><span style="color: rgb(147, 26, 104); \
">new</span><span class="Apple-converted-space"> </span>CassandraDaemon();<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 8.5pt; font-family: Monaco, serif; ">CassandraStoreKeeperTest.<span \
style="color: rgb(0, 39, 204); ">cassandra</span>.init(<span style="color: rgb(147, \
26, 104); ">null</span>);<o:p></o:p></span></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 8.5pt; font-family: Monaco, serif; ">CassandraStoreKeeperTest.<span \
style="color: rgb(0, 39, 204); ">cassandra</span>.start();<o:p></o:p></span></div> \
</div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 8.5pt; font-family: Monaco, serif; "><o:p> </o:p></span></div> \
</div> <div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
style="font-size: 13.5pt; font-family: Helvetica, sans-serif; ">- same cassandra yaml \
as in prod, file at <a href="https://gist.github.com/3207894" style="color: \
blue; text-decoration: underline; ">https://gist.github.com/3207894</a><span \
class="Apple-converted-space"> </span>( mostly defaults \
)<o:p></o:p></span></div> </div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <span \
class="apple-style-span"><span style="font-size: 13.5pt; font-family: Helvetica, \
sans-serif; ">- i pass the following flags to the JVM running my unit test: \
</span></span><span class="apple-style-span"><span style="font-size: 8.5pt; \
font-family: Monaco, serif; ">-server
-Xmx2G </span></span><span class="apple-style-span"><span style="font-size: \
13.5pt; font-family: Helvetica, sans-serif; ">and all the GC flags from \
cassandra-env.sh, plus the jamm java agent</span></span><span style="font-size: \
8.5pt; font-family: Monaco, serif; "><o:p></o:p></span></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- i warm up the embedded cassandra with 1k reads and 1k writes<o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- the test is 1000 single threaded, sequential reads via Hector and Astyanax ( no \
significant differences between the two ) via thrift to the embedded single-node \
Cassandra instance<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- i read back the 1000 items saved in the warm-up<o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> The results \
are<o:p></o:p></div> </div>
<div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.0242499999999999, 95th: \
2.1103499999999977, 98th: 5.2102600000000034, 99th: 13.24835000000001, 999th: \
41.89454000000041<o:p></o:p></div> </div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> here's the GC \
output during the read<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 281480 bytes, 281480 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17322K->308K(19136K), 0.0037375 secs] 28742K->11885K(83008K), 0.0038170 secs] \
[Times: user=0.01 sys=0.00, real=0.01 secs] <o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 261360 bytes, 261360 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17332K->386K(19136K), 0.0034069 secs] 28909K->12098K(83008K), 0.0034849 secs] \
[Times: user=0.00 sys=0.01, real=0.00 secs] <o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> [GC \
[ParNew<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Desired survivor \
size 1081344 bytes, new threshold 1 (max 1)<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
- age 1: 241336 bytes, 241336 \
total<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> : \
17410K->383K(19136K), 0.0035252 secs] 29122K->12209K(83008K), 0.0036062 secs] \
[Times: user=0.00 sys=0.00, real=0.00 secs] <o:p></o:p></div> </div>
</div>
</div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> Doesn't look \
like GC ( i could be wrong ) - what else could it be ?<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> all suggestions \
are highly appreciated..<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
Thanks!<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
Andras<o:p></o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
ps: this is with Cassandra 1.1.0<o:p></o:p></div>
</div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
<div>
<div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: \
0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> \
<o:p> </o:p></div> </div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic