[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Cassandra 4.0.6 token mismatch issue in production environment
From: Jaydeep Chovatia <chovatia.jaydeep () gmail ! com>
Date: 2023-10-23 23:27:23
Message-ID: CABzeAR70jWoe=SaWk82505yKqpMHLkrDtsoHBB-zYjEpvfHM8A () mail ! gmail ! com
[Download RAW message or body]
Sounds good. Thanks a lot for all your help!
Jaydeep
On Mon, Oct 23, 2023 at 3:30 PM Jeff Jirsa <jjirsa@gmail.com> wrote:
> Not aware of any that survive node restart, though in the past, there were
> races around starting an expansion while one node was partitioned/down (and
> missing the initial gossip / UP). A heap dump could have told us a bit more
> conclusively, but it's hard to guess for now.
>
>
>
> On Mon, Oct 23, 2023 at 3:22 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
> > The issue was persisting on a few nodes despite no changes to the
> > topology. Even node restarting did not help. Only after we evacuated those
> > nodes, the issue got resolved.
> >
> > Do you think of a possible situation under which this could happen?
> >
> > Jaydeep
> >
> > On Sat, Oct 21, 2023 at 10:25 AM Jaydeep Chovatia <
> > chovatia.jaydeep@gmail.com> wrote:
> >
> > > Thanks, Jeff!
> > > I will keep this thread updated on our findings.
> > >
> > > Jaydeep
> > >
> > > On Sat, Oct 21, 2023 at 9:37 AM Jeff Jirsa <jjirsa@gmail.com> wrote:
> > >
> > > > That code path was added to protect against invalid gossip states
> > > >
> > > > For this logger to be issued, the coordinator receiving the query must
> > > > identify a set of replicas holding the data to serve the read, and one of
> > > > the selected replicas must disagree that it's a replica based on its view
> > > > of the token ring
> > > >
> > > > This probably means that at least one node in your cluster has an
> > > > invalid view of the ring - if you issue a "nodetool ring" from every host
> > > > and compare them, you'll probably notice one or more is wrong
> > > >
> > > > It's also possible this happens for a few seconds during adding /
> > > > moving / removing hosts
> > > >
> > > > If you weren't changing the topology of the cluster, it's likely the
> > > > case that bouncing the cluster fixes it
> > > >
> > > > (Im unsure of the defaults and not able to look it up, but cassandra
> > > > can log or log and drop the read - you probably want to drop the read log,
> > > > which is the right solution so it doesn't accidentally return a missing /
> > > > empty result set as a valid query result, instead it'll force it to read
> > > > from other replicas or time out)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Oct 20, 2023, at 10:57 PM, Jaydeep Chovatia <
> > > > chovatia.jaydeep@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I am using Cassandra 4.0.6 in production, and receiving the following error. \
> > > > This indicates that Cassandra nodes have mismatch in token-owership.
> > > > Has anyone seen this issue before?
> > > >
> > > > Received a read request from /XX.XX.XXX.XXX:YYYYY for a range that is not \
> > > > owned by the current replica Read(keyspace.table columns=*/[c1] rowFilter= \
> > > > limits=LIMIT 100 key=7BE78B90-AD66-406B-AA05-6A062F72F542:0 \
> > > > filter=slice(slices=ALL, reversed=false), nowInSec=1697751757).
> > > > Jaydeep
> > > >
> > > >
[Attachment #3 (text/html)]
<div dir="ltr">Sounds good. Thanks a lot for all your \
help!<div><br></div><div>Jaydeep</div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Mon, Oct 23, 2023 at 3:30 PM Jeff Jirsa <<a \
href="mailto:jjirsa@gmail.com">jjirsa@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr">Not aware of any that survive node \
restart, though in the past, there were races around starting an expansion while one \
node was partitioned/down (and missing the initial gossip / UP). A heap dump could \
have told us a bit more conclusively, but it's hard to guess for now. \
<div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Oct 23, 2023 at 3:22 PM Jaydeep Chovatia <<a \
href="mailto:chovatia.jaydeep@gmail.com" \
target="_blank">chovatia.jaydeep@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="auto">The issue was persisting on a few \
nodes despite no changes to the topology. Even node restarting did not help. Only \
after we evacuated those nodes, the issue got resolved.</div><div \
dir="auto"><br></div><div dir="auto">Do you think of a possible situation under which \
this could happen?</div><div dir="auto"><br></div><div \
dir="auto">Jaydeep</div><div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Sat, Oct 21, 2023 at 10:25 AM Jaydeep Chovatia <<a \
href="mailto:chovatia.jaydeep@gmail.com" \
target="_blank">chovatia.jaydeep@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="auto">Thanks, Jeff!</div><div \
dir="auto">I will keep this thread updated on our findings.</div><div \
dir="auto"><br></div><div dir="auto">Jaydeep</div><div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 21, 2023 at 9:37 AM \
Jeff Jirsa <<a href="mailto:jjirsa@gmail.com" \
target="_blank">jjirsa@gmail.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="auto">That code path was added to \
protect against invalid gossip states<div><br></div><div><div dir="ltr">For this \
logger to be issued, the coordinator receiving the query must identify a set of \
replicas holding the data to serve the read, and one of the selected replicas must \
disagree that it's a replica based on its view of the token ring</div><div \
dir="ltr"><br></div><div dir="ltr">This probably means that at least one node in your \
cluster has an invalid view of the ring - if you issue a "nodetool ring" from every \
host and compare them, you'll probably notice one or more is wrong </div><div \
dir="ltr"><br></div><div dir="ltr">It's also possible this happens for a few seconds \
during adding / moving / removing hosts </div><div dir="ltr"><br></div><div \
dir="ltr">If you weren't changing the topology of the cluster, it's likely the case \
that bouncing the cluster fixes it </div><div dir="ltr"><br></div><div dir="ltr">(Im \
unsure of the defaults and not able to look it up, but cassandra can log or log and \
drop the read - you probably want to drop the read log, which is the right solution \
so it doesn't accidentally return a missing / empty result set as a valid query \
result, instead it'll force it to read from other replicas or time out)</div><div \
dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div \
dir="ltr"><br></div><div dir="ltr"><br><blockquote type="cite">On Oct 20, 2023, at \
10:57 PM, Jaydeep Chovatia <<a href="mailto:chovatia.jaydeep@gmail.com" \
target="_blank">chovatia.jaydeep@gmail.com</a>> \
wrote:<br><br></blockquote></div><blockquote type="cite"><div \
dir="ltr"></div></blockquote></div></div><div dir="auto"><div><blockquote \
type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><pre \
style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px; \
font-variant-ligatures:none;line-height:1.50001;word-break:normal;border-radius:4px;overflow-y:hidden;font-family:monospace;color:rgb(29,28,29)"><font \
face="arial, sans-serif" \
style="font-family:arial,sans-serif;color:rgb(29,28,29)">Hi,</font></pre><pre \
style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px; \
font-variant-ligatures:none;line-height:1.50001;word-break:normal;border-radius:4px;overflow-y:hidden;font-family:monospace;color:rgb(29,28,29)"><font \
face="arial, sans-serif" style="font-family:arial,sans-serif;color:rgb(29,28,29)">I \
am using Cassandra 4.0.6 in production, and receiving the following error. This \
indicates that Cassandra nodes have mismatch in token-owership.</font></pre><pre \
style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px; \
font-variant-ligatures:none;line-height:1.50001;word-break:normal;border-radius:4px;overflow-y:hidden;font-family:monospace;color:rgb(29,28,29)"><font \
face="arial, sans-serif" style="font-family:arial,sans-serif;color:rgb(29,28,29)">Has \
anyone seen this issue before?</font></pre><pre \
style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-variant-li \
gatures:none;line-height:1.50001;word-break:normal;border-radius:4px;overflow-y:hidden;font-family:monospace;color:rgb(29,28,29)"><font \
face="arial, sans-serif" size="1" \
style="font-family:arial,sans-serif;color:rgb(29,28,29)">Received a read request from \
/XX.XX.XXX.XXX:YYYYY for a range that is not owned by the current replica \
Read(keyspace.table columns=*/[c1] rowFilter= limits=LIMIT 100 \
key=7BE78B90-AD66-406B-AA05-6A062F72F542:0 filter=slice(slices=ALL, reversed=false), \
nowInSec=1697751757).</font></pre><pre \
style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px; \
font-variant-ligatures:none;line-height:1.50001;word-break:normal;border-radius:4px;overflow-y:hidden;font-family:monospace;color:rgb(29,28,29)"><font \
face="arial, sans-serif" \
style="font-family:arial,sans-serif;color:rgb(29,28,29)">Jaydeep</font></pre></div></div></div>
</div></blockquote></div></div></blockquote></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic