[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Compactions are stuck in 4.0.5 version
From: vaibhav khedkar <vkhedkar7 () gmail ! com>
Date: 2023-01-15 1:33:10
Message-ID: CALgidco2WAyJGtoU2YSr1VSZGaPB3brNaERyqHgdcj4gDqmKNw () mail ! gmail ! com
[Download RAW message or body]
Thank you so much Scott.
Increasing the value from 64 to 128 fixed the issue for us.
We will certainly look at our data model and understand why the partitions
are growing to such a large value.
Thanks
vaibhav
On Fri, Jan 13, 2023 at 9:45 PM C. Scott Andreas <scott@paradoxica.net>
wrote:
> Hi Vaibhav, thanks for reaching out.
>
> Based on my understanding of this exception, this may be due to the index
> for this partition exceeding 2GiB (which is *extremely* large for a
> partition index component).
>
> Reducing the size of the column index below 2GiB may resolve this issue.
> You may be able to do so by increasing the size of
> "column_index_size_in_kb" in cassandra.yaml (which defines the granularity
> of this index). This may enable the compaction to complete - e.g., by
> swapping the default of "64" with "128".
>
> Docs for this param are in cassandra.yaml:
> # Granularity of the collation index of rows within a partition.
> # Increase if your rows are large, or if you have a very large
> # number of rows per partition. The competing goals are these:
> #
> # - a smaller granularity means more index entries are generated
> # and looking up rows withing the partition by collation column
> # is faster
> # - but, Cassandra will keep the collation index in memory for hot
> # rows (as part of the key cache), so a larger granularity means
> # you can cache more hot rows
>
> However, the root of this issue is likely a partition whose size has
> become extraordinary. Would recommend determining what has resulted in an
> individual partition growing so large that its index has exceeded 2GiB and
> determining if it can be removed or if the data model for the table can be
> adjusted to avoid such a large number of rows being stored within one
> partition.
>
> – Scott
>
> On Jan 13, 2023, at 9:24 PM, vaibhav khedkar <vkhedkar7@gmail.com> wrote:
>
>
> Hello All,
>
> We are facing an issue where few of the nodes are not able to complete
> compactions.
> We tried restarting, scrubbing and even rebuilding an entire node but
> nothing seems to work so far.
>
> It's a 10 Region installation with close to 150 nodes.
>
> Datatax support
> <https://support.datastax.com/s/article/ERROR-Failure-serializing-partition-key>
> suggested rebuilding the node but that did not help. Any help is
> appreciated.
>
> Following is the logtrace.
>
> ERROR [CompactionExecutor:50] 2023-01-14 05:12:20,795
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:50,1,main]
> org.apache.cassandra.db.rows.PartitionSerializationException: Failed to
> serialize partition key '<key>' on table '<table>' in keyspace
> '<keyspace>'.
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:240)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:84)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.BufferOverflowException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:136)
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:154)
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:161)
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:121)
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:121)
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:93)
> at
> org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:61)
> at
> org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:38)
> at
> org.apache.cassandra.db.marshal.ValueAccessor.writeWithVIntLength(ValueAccessor.java:164)
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:451)
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:397)
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:132)
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:339)
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:110)
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:91)
> at org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:223)
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:271)
> at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:118)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:216)
>
> ... 14 common frames omitted
>
> Thanks
> vaibhav
>
>
>
>
>
--
Best Regards,
Vaibhav Khedkar
Google Inc.
Mobile : (806)- 252 - 2912
Email: vkhedkar7@gmail.com
[Attachment #3 (text/html)]
<div dir="ltr">Thank you so much Scott. <div><br></div><div>Increasing the value \
from 64 to 128 fixed the issue for us. </div><div><br></div><div>We will certainly \
look at our data model and understand why the partitions are growing to such a large \
value. </div><div><br></div><div>Thanks</div><div>vaibhav </div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 13, 2023 at 9:45 PM \
C. Scott Andreas <<a \
href="mailto:scott@paradoxica.net">scott@paradoxica.net</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>Hi \
Vaibhav, thanks for reaching out.<br></div><div><br></div><div>Based on my \
understanding of this exception, this may be due to the index for this partition \
exceeding 2GiB (which is *extremely* large for a partition index \
component).<br></div><div><br></div><div>Reducing the size of the column index below \
2GiB may resolve this issue. You may be able to do so by increasing the size of \
"column_index_size_in_kb" in cassandra.yaml (which defines the granularity \
of this index). This may enable the compaction to complete - e.g., by swapping the \
default of "64" with "128".<br></div><div><br></div><div>Docs for \
this param are in cassandra.yaml:<br></div><div># Granularity of the collation index \
of rows within a partition.<br></div></div><div># Increase if your rows are large, or \
if you have a very large<br></div><div># number of rows per partition. The \
competing goals are these:<br></div><div>#<br></div><div># - a smaller granularity \
means more index entries are generated<br></div><div># and looking up rows \
withing the partition by collation column<br></div><div># is \
faster<br></div><div># - but, Cassandra will keep the collation index in memory for \
hot<br></div><div># rows (as part of the key cache), so a larger granularity \
means<br></div><div># you can cache more hot \
rows<br></div><div><br></div><div>However, the root of this issue is likely a \
partition whose size has become extraordinary. Would recommend determining what has \
resulted in an individual partition growing so large that its index has exceeded 2GiB \
and determining if it can be removed or if the data model for the table can be \
adjusted to avoid such a large number of rows being stored within one \
partition.<br></div><div><br></div><div>– \
Scott</div><div><div><br></div><blockquote type="cite"><div>On Jan 13, 2023, at 9:24 \
PM, vaibhav khedkar <<a href="mailto:vkhedkar7@gmail.com" \
target="_blank">vkhedkar7@gmail.com</a>> \
wrote:<br></div><div><br></div><div><br></div><div><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello All, \
<br></div><div><br></div><div>We are facing an issue where few of the nodes are not \
able to complete compactions. <br></div><div><div>We tried restarting, scrubbing and \
even rebuilding an entire node but nothing seems to work so far. \
<br></div><div><br></div><div>It's a 10 Region installation with close to 150 \
nodes. <br></div></div><div><br></div><div><div><a rel="noopener noreferrer" \
href="https://support.datastax.com/s/article/ERROR-Failure-serializing-partition-key" \
target="_blank">Datatax support</a> suggested rebuilding the node but that did not \
help. Any help is appreciated. <br></div><div><br></div><div>Following is the \
logtrace. <br></div></div><div><br></div><div><div>ERROR [CompactionExecutor:50] \
2023-01-14 05:12:20,795 CassandraDaemon.java:581 - Exception in thread \
Thread[CompactionExecutor:50,1,main]<br></div><div>org.apache.cassandra.db.rows.PartitionSerializationException: \
Failed to serialize partition key '<key>' on table \
'<table>' in keyspace '<keyspace>'.<br></div><div> at \
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:240)<br></div><div> \
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)<br></div><div> \
at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:84)<br></div><div> \
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)<br></div><div> \
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)<br></div><div> \
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)<br></div><div> \
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)<br></div><div> \
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)<br></div><div> \
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)<br></div><div> \
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br></div><div> \
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br></div><div> at \
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)<br></div><div> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)<br></div><div> \
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)<br></div><div> \
at java.lang.Thread.run(Thread.java:748)<br></div><div>Caused by: \
java.nio.BufferOverflowException: null<br></div><div> at \
org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:136)<br></div><div> \
at org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:154)<br></div><div> \
at org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:161)<br></div><div> \
at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:121)<br></div><div> \
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:121)<br></div><div> \
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:93)<br></div><div> \
at org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:61)<br></div><div> \
at org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:38)<br></div><div> \
at org.apache.cassandra.db.marshal.ValueAccessor.writeWithVIntLength(ValueAccessor.java:164)<br></div><div> \
at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:451)<br></div><div> \
at org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:397)<br></div><div> \
at org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:132)<br></div><div> \
at org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:339)<br></div><div> \
at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:110)<br></div><div> \
at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:91)<br></div><div> \
at org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:223)<br></div><div> \
at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:271)<br></div><div> at \
org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:118)<br></div><div> \
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:216)<br></div><div> \
... 14 common frames \
omitted<br></div></div><div><br></div><div>Thanks<br></div><div>vaibhav<br></div></div \
></div></div></div></div></div></blockquote><div><br></div><div><br></div></div><div><br></div></div></blockquote></div><br \
> clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div \
> dir="ltr"><div dir="ltr"><div dir="ltr"><div><span style="color:rgb(0,0,0)">Best \
> Regards,<br></span></div><div><span style="color:rgb(0,0,0)">Vaibhav \
> Khedkar</span></div><div><span style="color:rgb(0,0,0)">Google Inc. \
> </span></div><div><span style="color:rgb(0,0,0)">Mobile : (806)- 252 - \
> 2912</span><br style="color:rgb(0,0,0)"><br style="color:rgb(0,0,0)"><span \
> style="color:rgb(0,0,0)">Email: </span><a style="color:rgb(0,0,0)" \
> href="mailto:vkhedkar7@gmail.com" \
> target="_blank">vkhedkar7@gmail.com</a><br></div></div></div></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic