[prev in list] [next in list] [prev in thread] [next in thread] 

List:       cassandra-commits
Subject:    [jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs
From:       "Jeff Jirsa (JIRA)" <jira () apache ! org>
Date:       2017-09-29 23:51:03
Message-ID: JIRA.13034973.1484429293000.242762.1506729063679 () Atlassian ! JIRA
[Download RAW message or body]


    [ https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.pl \
ugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186704#comment-16186704 \
] 

Jeff Jirsa commented on CASSANDRA-13123:
----------------------------------------

Hi folks,

Pretty sure this commit breaks {{CommitLogSegmentManagerTest}} - have seen a pretty \
sharp rise in failures, and reverting this commit seems to solve them.

{code}
    [junit] INFO  23:34:52 Initializing CommitLogTest.Standard1
    [junit] INFO  23:34:52 Initializing CommitLogTest.Standard2
    [junit] ------------- ---------------- ---------------
    [junit] Testcase: \
testShutdownWithPendingTasks(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):	FAILED
  [junit] null
    [junit] junit.framework.AssertionFailedError
    [junit] 	at org.apache.cassandra.db.Keyspace.open(Keyspace.java:105)
    [junit] 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testShutdownWithPendingTasks(CommitLogSegmentManagerTest.java:147)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$10.evaluate(BMUnitRunner.java:371)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
  [junit]
    [junit]
    [junit] Testcase: \
testCompressedCommitLogBackpressure(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):	FAILED
  [junit] expected:<3> but was:<1>
    [junit] junit.framework.AssertionFailedError: expected:<3> but was:<1>
    [junit] 	at org.apache.cassandra.Util.spinAssertEquals(Util.java:535)
    [junit] 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testCompressedCommitLogBackpressure(CommitLogSegmentManagerTest.java:112)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$9.evaluate(BMUnitRunner.java:342)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
  [junit] 	at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
  [junit]
    [junit]
    [junit] Test org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest FAILED
   [delete] Deleting directory \
/Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/commitlog:0  [delete] \
Deleting directory /Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/data:0  \
[delete] Deleting directory \
/Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/saved_caches:0 [junitreport] \
Processing /Users/jjirsa/Desktop/Dev/cassandra/build/test/TESTS-TestSuites.xml to \
/var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913 [junitreport] Loading \
stylesheet jar:file:/usr/local/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
 [junitreport] Transform time: 256ms
[junitreport] Deleting: \
/var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913 {code}


> Draining a node might fail to delete all inactive commitlogs
> ------------------------------------------------------------
> 
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: Jan Urbański
> Assignee: Jan Urbański
> Fix For: 3.0.15, 3.11.1, 4.0
> 
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 13123-trunk.txt
> 
> 
> After issuing a drain command, it's possible that not all of the inactive \
> commitlogs are removed. The drain command shuts down the CommitLog instance, which \
> in turn shuts down the CommitLogSegmentManager. This has the effect of discarding \
> any pending management tasks it might have, like the removal of inactive \
> commitlogs. This in turn leads to an excessive amount of commitlogs being left \
> behind after a drain and a lengthy recovery after a restart. With a fleet of dozens \
> of nodes, each of them leaving several GB of commitlogs after a drain and taking up \
> to two minutes to recover them on restart, the additional time required to restart \
> the entire fleet becomes noticeable. This problem is not present in 3.x or trunk \
> because of the CLSM rewrite done in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]