[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: java.io.IOException: Failed during snapshot creation
From: Jens Rantil <jens.rantil () tink ! se>
Date: 2015-05-26 8:48:14
Message-ID: CAL+ArfWbo8iDF_pHuAwDFHfuTiHG2WwK8H0DcVoKV7JEcsenCQ () mail ! gmail ! com
[Download RAW message or body]
Sachin,
Are also the subdirectories in /var/lib/cassandra (or elsewhere) also the
correct owner/group? Is the Cassandra user allowed to create snapshot
directory and write to it?
Cheers,
Jens
On Tue, May 26, 2015 at 8:13 AM, Sachin PK <sachinpray4me@gmail.com> wrote:
> my Cassandra folder's owner,group are cassandra
>
> On Tue, May 26, 2015 at 2:20 AM, Brice Argenson <bargenson@gmail.com>
> wrote:
>
> > I also have the same kind of exceptions once. It was caused by a bad
> > configuration of the Cassandra data folder permissions (process running as
> > user Cassandra with root as owner of the folder). It was then preventing
> > any snapshot creations during a repair.
> >
> > On Mon, May 25, 2015 at 3:43 PM Yuki Morishita <mor.yuki@gmail.com>
> > wrote:
> >
> > > Hi, it can be timimg out waiting for snapshot to complete.
> > > See https://issues.apache.org/jira/browse/CASSANDRA-8696 for workaround.
> > >
> > > On Mon, May 25, 2015 at 10:44 AM, Mark Reddy <mark.l.reddy@gmail.com>
> > > wrote:
> > > > Can you check your logs for any other other error message around the
> > > time of
> > > > the repair? Something to look for would be "Error occurred during
> > > snapshot
> > > > phase".
> > > >
> > > >
> > > > Regards,
> > > > Mark
> > > >
> > > > On 25 May 2015 at 14:56, Sachin PK <sachinpray4me@gmail.com> wrote:
> > > > >
> > > > > Hey I'm new to Cassandra ,I have 4 node cluster with each node 16GB
> > > VPS,
> > > > > initially I had one seed node, I added one of the existing nodes as
> > > seed
> > > > > nodes , restarted nodes one by one after that one of my node went
> > > down . I
> > > > > ran nodetool repair on it when I checked the log i could find some
> > > errors
> > > > >
> > > > >
> > > > > ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,905
> > > > > RepairSession.java:303 - [repair
> > > #968bce30-02e1-11e5-9310-45191f4c93ae]
> > > > > session completed with the following error
> > > > > java.io.IOException: Failed during snapshot creation.
> > > > > at
> > > > >
> > > org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> > >
> > > > > ~[apache-cassandra-2.1.5.jar:2.1.5]
> > > > > at
> > > org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)
> > > > > ~[apache-cassandra-2.1.5.jar:2.1.5]
> > > > > at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> > > > > ~[guava-16.0.jar:na]
> > > > > at
> > > > >
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > > [na:1.7.0_80]
> > > > > at
> > > > >
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > > [na:1.7.0_80]
> > > > > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> > > > > ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,907
> > > > > CassandraDaemon.java:223 - Exception in thread
> > > > > Thread[AntiEntropySessions:3,5,RMI Runtime]
> > > > > java.lang.RuntimeException: java.io.IOException: Failed during
> > > snapshot
> > > > > creation.
> > > > > at com.google.common.base.Throwables.propagate(Throwables.java:160)
> > > > > ~[guava-16.0.jar:na]
> > > > > at
> > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> > > > > ~[apache-cassandra-2.1.5.jar:2.1.5]
> > > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > > ~[na:1.7.0_80]
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > ~[na:1.7.0_80]
> > > > > at
> > > > >
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > > ~[na:1.7.0_80]
> > > > > at
> > > > >
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > > [na:1.7.0_80]
> > > > > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> > > > > Caused by: java.io.IOException: Failed during snapshot creation.
> > > > > at
> > > > >
> > > org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> > >
> > > > > ~[apache-cassandra-2.1.5.jar:2.1.5]
> > > > > at
> > > org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)
> > > > > ~[apache-cassandra-2.1.5.jar:2.1.5]
> > > > > at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> > > > > ~[guava-16.0.jar:na]
> > > > > ... 3 common frames omitted
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Yuki Morishita
> > > t:yukim (http://twitter.com/yukim)
> > >
> >
>
>
> --
> Thanks &Regards,
>
> Sachin PK
>
> www.sachinpk.tk
>
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>
[Attachment #3 (text/html)]
<div dir="ltr">Sachin,<div><br></div><div>Are also the subdirectories in \
/var/lib/cassandra (or elsewhere) also the correct owner/group? Is the Cassandra user \
allowed to create snapshot directory and write to \
it?</div><div><br></div><div>Cheers,</div><div>Jens</div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Tue, May 26, 2015 at 8:13 AM, \
Sachin PK <span dir="ltr"><<a href="mailto:sachinpray4me@gmail.com" \
target="_blank">sachinpray4me@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">my Cassandra folder's owner,group are \
cassandra</div><div class="gmail_extra"><div><div class="h5"><br><div \
class="gmail_quote">On Tue, May 26, 2015 at 2:20 AM, Brice Argenson <span \
dir="ltr"><<a href="mailto:bargenson@gmail.com" \
target="_blank">bargenson@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">I also have the same kind of exceptions once. \
It was caused by a bad configuration of the Cassandra data folder permissions \
(process running as user Cassandra with root as owner of the folder). <span \
style="line-height:1.5;font-size:13.1999998092651px">It was then preventing any \
snapshot creations during a repair.</span></div><div><div><br><div \
class="gmail_quote">On Mon, May 25, 2015 at 3:43 PM Yuki Morishita <<a \
href="mailto:mor.yuki@gmail.com" target="_blank">mor.yuki@gmail.com</a>> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex">Hi, it can be timimg out waiting for snapshot to \
complete.<br> See <a href="https://issues.apache.org/jira/browse/CASSANDRA-8696" \
target="_blank">https://issues.apache.org/jira/browse/CASSANDRA-8696</a> for \
workaround.<br> <br>
On Mon, May 25, 2015 at 10:44 AM, Mark Reddy <<a \
href="mailto:mark.l.reddy@gmail.com" target="_blank">mark.l.reddy@gmail.com</a>> \
wrote:<br> > Can you check your logs for any other other error message around the \
time of<br> > the repair? Something to look for would be "Error occurred \
during snapshot<br> > phase".<br>
><br>
><br>
> Regards,<br>
> Mark<br>
><br>
> On 25 May 2015 at 14:56, Sachin PK <<a href="mailto:sachinpray4me@gmail.com" \
target="_blank">sachinpray4me@gmail.com</a>> wrote:<br> >><br>
>> Hey I'm new to Cassandra ,I have 4 node cluster with each node 16GB \
VPS,<br> >> initially I had one seed node, I added one of the existing nodes as \
seed<br> >> nodes , restarted nodes one by one after that one of my node went \
down . I<br> >> ran nodetool repair on it when I checked the log i could find \
some errors<br> >><br>
>><br>
>> ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,905<br>
>> RepairSession.java:303 - [repair #968bce30-02e1-11e5-9310-45191f4c93ae]<br>
>> session completed with the following error<br>
>> java.io.IOException: Failed during snapshot creation.<br>
>> at<br>
>> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)<br>
>> ~[apache-cassandra-2.1.5.jar:2.1.5]<br>
>> at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)<br>
>> ~[apache-cassandra-2.1.5.jar:2.1.5]<br>
>> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)<br>
>> ~[guava-16.0.jar:na]<br>
>> at<br>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
>> [na:1.7.0_80]<br>
>> at<br>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
>> [na:1.7.0_80]<br>
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]<br>
>> ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,907<br>
>> CassandraDaemon.java:223 - Exception in thread<br>
>> Thread[AntiEntropySessions:3,5,RMI Runtime]<br>
>> java.lang.RuntimeException: java.io.IOException: Failed during snapshot<br>
>> creation.<br>
>> at com.google.common.base.Throwables.propagate(Throwables.java:160)<br>
>> ~[guava-16.0.jar:na]<br>
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)<br>
>> ~[apache-cassandra-2.1.5.jar:2.1.5]<br>
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br>
>> ~[na:1.7.0_80]<br>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) \
~[na:1.7.0_80]<br> >> at<br>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
>> ~[na:1.7.0_80]<br>
>> at<br>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
>> [na:1.7.0_80]<br>
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]<br>
>> Caused by: java.io.IOException: Failed during snapshot creation.<br>
>> at<br>
>> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)<br>
>> ~[apache-cassandra-2.1.5.jar:2.1.5]<br>
>> at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)<br>
>> ~[apache-cassandra-2.1.5.jar:2.1.5]<br>
>> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)<br>
>> ~[guava-16.0.jar:na]<br>
>> ... 3 common frames omitted<br>
>><br>
>><br>
><br>
<br>
<br>
<br>
--<br>
Yuki Morishita<br>
t:yukim (<a href="http://twitter.com/yukim" \
target="_blank">http://twitter.com/yukim</a>)<br> </blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span \
class="HOEnZb"><font color="#888888">-- <br><div><font face="'comic sans ms', \
sans-serif" size="4">Thanks &Regards,<br><br> Sachin PK</font><br><span \
style="background-color:rgb(255,255,255)"><font \
color="#ff0000"><br></font></span><div><span \
style="background-color:rgb(255,255,255)"><font color="#ff0000" size="4" \
face="'comic sans ms', sans-serif"><a href="http://www.sachinpk.tk" \
target="_blank">www.sachinpk.tk</a></font></span></div><div><div><br></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
class="gmail_signature"><div dir="ltr"><div>Jens Rantil</div><div>Backend \
engineer</div><div>Tink AB</div><div><br></div><div>Email: <a \
href="mailto:jens.rantil@tink.se" style="color:rgb(17,85,204)" \
target="_blank">jens.rantil@tink.se</a></div><div>Phone: +46 708 84 18 \
32</div><div>Web: <a href="http://www.tink.se/" style="color:rgb(17,85,204)" \
target="_blank">www.tink.se</a></div><div><br></div><div><a \
href="https://www.facebook.com/#!/tink.se" \
style="color:rgb(17,85,204);font-family:arial;font-size:small" \
target="_blank">Facebook</a><span style="font-family:arial;font-size:small"> \
</span><a href="http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&a \
mp;trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary" \
style="color:rgb(17,85,204);font-family:arial;font-size:small" \
target="_blank">Linkedin</a><span style="font-family:arial;font-size:small"> \
</span><a href="https://twitter.com/tink" \
style="color:rgb(17,85,204);font-family:arial;font-size:small" \
target="_blank">Twitter</a></div></div></div> </div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic