[prev in list] [next in list] [prev in thread] [next in thread] 

List:       cassandra-dev
Subject:    Race condition causing corrupted sstable stats files
From:       Hannu_Kröger <hkroger () gmail ! com>
Date:       2017-08-22 14:20:08
Message-ID: CAN-FP2or8KkR4VRAqKxdAW-_Rn6emJDL1Y9OYmeZ-xhPZFRU=g () mail ! gmail ! com
[Download RAW message or body]


Hello everyone,

We ran into some nasty bug while upgrading to Cassandra 3.11:
https://issues.apache.org/jira/browse/CASSANDRA-13752

Basically there is possible race condition in tombstone histogram
serialization. If someone modifies tombstone histogram after size is
serialized but before entries are serialized then you will end up in the
situation where stats files tells that you have X entries in the histogram
but after that you actually have X+N (N=1 seen in wild) entries written in
the file. Cassandra cannot deserialize that when it next time reads the
file (after a restart at least).

I have a fix for it:
https://github.com/hkroger/cassandra/tree/cassandra-3.11-13752

Writing a test for it is a bit hard. Also testing manually that it actually
fixes it is hard so…

How can we push this forward? It's sort of critical for us and I would like
some more experienced C* dev to give fatherly or motherly advice :)

BR,
Hannu Kröger


[prev in list] [next in list] [prev in thread] [next in thread]