[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: RE: Upgrade from C* 3 to C* 4 per datacenter
From: "Michalis Kotsiouros (EXT) via user" <user () cassandra ! apache ! org>
Date: 2023-10-27 10:04:42
Message-ID: DU2PR07MB8361868F95E519257AAFD024C6DCA () DU2PR07MB8361 ! eurprd07 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hello everyone,
@Sebastian thanks a lot for the indication about cloning the cluster to do the tests. \
It is really good information. @Aaron thanks a lot for the heads up.
Thanks a lot for the valuable contribution of the community in this case!!!
BR
MK
From: Aaron Ploetz <aaronploetz@gmail.com>
Sent: October 26, 2023 21:08
To: user@cassandra.apache.org
Subject: Re: Upgrade from C* 3 to C* 4 per datacenter
Just a heads-up, but there have been issues (at least one) reported when upgrading a \
multi-DC cluster from 3.x to 4.x when the cluster uses node-to-node SSL/TLS \
encryption. This is largely attributed to the fact that the secure port in 4.x \
changes to 9142, whereas in 3.x it continues to run on 9042 (same as non-SSL/TLS).
On Thu, Oct 26, 2023 at 2:03 PM Sebastian Marsching \
<sebastian@marsching.com<mailto:sebastian@marsching.com>> wrote: Hi,
as we are currently facing the same challenge (upgrading an existing cluster from C* \
3 to C* 4), I wanted to share our strategy with you. It largely is what Scott already \
suggested, but I have some extra details, so I thought it might still be useful.
We duplicated our cluster using the strategy described at \
http://adamhutson.com/cloning-cassandra-clusters-the-fast-way/<https://protect2.fireey \
e.com/v1/url?k=31323334-501d5122-313273af-454445555731-115c472cae66ec83&q=1&e=a393db99 \
-c4f3-414a-abfb-815cb3868d2d&u=http%3A%2F%2Fadamhutson.com%2Fcloning-cassandra-clusters-the-fast-way%2F>. \
Of course it is possible to figure out all the steps on your own, but I feel like \
this detailed guide saved me at least a few hours, if not days. Instead of restoring \
from a backup, we chose to create a snapshot on the live nodes and copy the data from \
there, but this does not really change the overall process.
We only run a single data-center cluster, but I think that this process easily \
translates to a multi data-center setup. In this case, you can choose to only clone a \
single data center or you can clone a few or all of them, if you deem this to be \
necessary for your tests. The only "limitation" is that for each data center that you \
clone, you need exactly the same number of nodes in your test cluster that you have \
in the respective data center of your production cluster.
Once the cluster is cloned, you can test whatever you like (e.g. upgrade to C* 4, \
test operations in a mixed-version cluster, etc.).
Our experience with the upgrade from C* 3.11 to C* 4.1 on the test cluster was quite \
smooth. The only problem that we saw was that when later adding a second data center \
to the test cluster, we got a lot of CorruptSSTableExceptions on one of the nodes in \
the existing data center. We first attributed this to the upgrade, but later we found \
out that this also happens when running on C* 3.11.
We now believe that the hardware of one of the nodes that we used for the test \
cluster has a defect, because the exceptions were limited to this exact node, even \
after moving data around. It just took us a while to figure this out, because the \
hardware for the test cluster was brand new, so "broken hardware" wasn't our first \
guess. We are still in the process of definitely proving that this specific piece of \
hardware is broken, but we are now sufficiently confident in the stability of C* 4, \
that we are soon going to move forward with upgrading the production cluster.
-Sebastian
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello everyone,<o:p></o:p></p>
<p class="MsoNormal">@Sebastian thanks a lot for the indication about cloning the \
cluster to do the tests. It is really good information.<o:p></o:p></p> <p \
class="MsoNormal">@Aaron thanks a lot for the heads up. <o:p></o:p></p> <p \
class="MsoNormal">Thanks a lot for the valuable contribution of the community in this \
case!!!<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">BR<o:p></o:p></p>
<p class="MsoNormal">MK<o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Aaron Ploetz <aaronploetz@gmail.com> <br>
<b>Sent:</b> October 26, 2023 21:08<br>
<b>To:</b> user@cassandra.apache.org<br>
<b>Subject:</b> Re: Upgrade from C* 3 to C* 4 per datacenter<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Just a heads-up, but there have been issues (at least one) \
reported when upgrading a multi-DC cluster from 3.x to 4.x when the cluster uses \
node-to-node SSL/TLS encryption. This is largely attributed to the fact that the \
secure port in 4.x changes to 9142, whereas in 3.x it continues to run on 9042 (same \
as non-SSL/TLS).<o:p></o:p></p> </div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Oct 26, 2023 at 2:03 PM Sebastian Marsching <<a \
href="mailto:sebastian@marsching.com">sebastian@marsching.com</a>> \
wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">as we are currently facing the same challenge (upgrading an \
existing cluster from C* 3 to C* 4), I wanted to share our strategy with you. It \
largely is what Scott already suggested, but I have some extra details, so I thought \
it might still be useful.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We duplicated our cluster using the strategy described \
at <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-4544 \
45555731-115c472cae66ec83&q=1&e=a393db99-c4f3-414a-abfb-815cb3868d2d&u=http%3A%2F%2Fadamhutson.com%2Fcloning-cassandra-clusters-the-fast-way%2F" \
target="_blank">http://adamhutson.com/cloning-cassandra-clusters-the-fast-way/</a>. \
Of course it is possible to figure out all the steps on your own, but I feel like \
this detailed guide saved me at least a few hours, if not days. Instead of restoring \
from a backup, we chose to create a snapshot on the live nodes and copy the data from \
there, but this does not really change the overall process.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We only run a single data-center cluster, but I think that this \
process easily translates to a multi data-center setup. In this case, you can choose \
to only clone a single data center or you can clone a few or all of them, if you deem \
this to be necessary for your tests. The only "limitation" is that for each data \
center that you clone, you need exactly the same number of nodes in your test cluster \
that you have in the respective data center of your production \
cluster.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Once the cluster is cloned, you can test whatever you like (e.g. \
upgrade to C* 4, test operations in a mixed-version cluster, etc.).<o:p></o:p></p> \
</div> <div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Our experience with the upgrade from C* 3.11 to C* 4.1 on the \
test cluster was quite smooth. The only problem that we saw was that when later \
adding a second data center to the test cluster, we got a lot of \
CorruptSSTableExceptions on one of the nodes in the existing data center. We first \
attributed this to the upgrade, but later we found out that this also happens when \
running on C* 3.11.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We now believe that the hardware of one of the nodes that we \
used for the test cluster has a defect, because the exceptions were limited to this \
exact node, even after moving data around. It just took us a while to figure this \
out, because the hardware for the test cluster was brand new, so "broken hardware" \
wasn't our first guess. We are still in the process of definitely proving that this \
specific piece of hardware is broken, but we are now sufficiently confident in the \
stability of C* 4, that we are soon going to move forward with upgrading the \
production cluster.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Sebastian<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic