[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Cassandra's log is full of mesages reset by peers even without traffic
From: Jean Carlo <jean.jeancarl48 () gmail ! com>
Date: 2016-01-29 10:17:16
Message-ID: CAHW7YSTNWWww3y1o1Jz2d85Tjsj=ZK9xoH1WRZVw0MUrcbUxZA () mail ! gmail ! com
[Download RAW message or body]
Hi anuj,
Thx for your replay, acctually I paste part of the result of the command
gre done over one log and I can see only the ip of the machine local
grep "Unexpected exception during request"
/var/opt/hosting/log/cassandra/system.log
INFO [SharedPool-Worker-1] 2016-01-29 10:40:47,744 Message.java:532 -
Unexpected exception during request; channel = [id: 0x6ebe93cb, /
172.16.162.4:9042]
INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 -
Unexpected exception during request; channel = [id: 0xff15eb8c, /
172.16.162.4:9042]
INFO [SharedPool-Worker-1] 2016-01-29 10:54:33,721 Message.java:532 -
Unexpected exception during request; channel = [id: 0xc42cc7ff, /
172.16.162.2:11436 :> /172.16.162.5:9042]
INFO [SharedPool-Worker-2] 2016-01-29 10:45:47,761 Message.java:532 -
Unexpected exception during request; channel = [id: 0x603349e4, /
172.16.162.4:9042]
INFO [SharedPool-Worker-3] 2016-01-29 10:48:17,766 Message.java:532 -
Unexpected exception during request; channel = [id: 0x5bed4eae, /
172.16.162.4:9042]
INFO [SharedPool-Worker-2] 2016-01-29 10:48:27,767 Message.java:532 -
Unexpected exception during request; channel = [id: 0x6136756b, /
172.16.162.4:9042]
INFO [SharedPool-Worker-4] 2016-01-29 10:48:27,767 Message.java:532 -
Unexpected exception during request; channel = [id: 0x17c83eb8, /
172.16.162.4:9042]
INFO [SharedPool-Worker-4] 2016-01-29 10:52:07,778 Message.java:532 -
Unexpected exception during request; channel = [id: 0x1a78b589, /
172.16.162.4:9042]
INFO [SharedPool-Worker-2] 2016-01-29 10:52:17,779 Message.java:532 -
Unexpected exception during request; channel = [id: 0x017117b3, /
172.16.162.4:9042]
INFO [SharedPool-Worker-2] 2016-01-29 11:01:37,813 Message.java:532 -
Unexpected exception during request; channel = [id: 0x3efbd844, /
172.16.162.4:9042]
Then I don't know if the firewall has something to do on that case, becasue
it is a local connection over native protocol
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Fri, Jan 29, 2016 at 11:02 AM, Anuj Wadehra <anujw_2003@yahoo.co.in>
wrote:
> Hi Jean,
>
> Please make sure that your Firewall is not dropping TCP connections which
> are in use. Tcp keep alive on all nodes must be less than the firewall
> setting. Please refer to
>
> https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html \
> for details on TCP settings.
>
>
> Thanks
> Anuj
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>
> On Fri, 29 Jan, 2016 at 3:21 pm, Jean Carlo
> <jean.jeancarl48@gmail.com> wrote:
> Hello guys,
>
> I have a cluster cassandra 2.1.12 with 6 nodes. All the logs of my nodes
> are having this messages marked as INFO
>
> INFO [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 -
> Unexpected exception during request; channel = [id: 0xff15eb8c, /
> 172.16.162.4:9042]
> java.io.IOException: Error while read(...): Connection reset by peer
> at io.netty.channel.epoll.Native.readAddress(Native Method)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
> This happens either the cluster is stressed or not. Btw it is not
> production. The ip marked there (172.16.162.4) belongs to a node of the
> cluster, this is not the only node that appears, acctually we are having
> all the node's ip having that reset by peer problem.
>
> Our cluster is having more reads than writes. like 50 reads per second.
>
> Any one got the same problem?
>
>
> Best regards
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
>
[Attachment #3 (text/html)]
<div dir="ltr"><div>Hi anuj,<br><br></div>Thx for your replay, acctually I paste part \
of the result of the command gre done over one log and I can see only the ip of the \
machine local<br><br><br>grep "Unexpected exception during request" \
/var/opt/hosting/log/cassandra/system.log<br><br>INFO [SharedPool-Worker-1] \
2016-01-29 10:40:47,744 Message.java:532 - Unexpected exception during request; \
channel = [id: 0x6ebe93cb, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - Unexpected exception \
during request; channel = [id: 0xff15eb8c, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-1] 2016-01-29 10:54:33,721 Message.java:532 - Unexpected exception \
during request; channel = [id: 0xc42cc7ff, /<a \
href="http://172.16.162.2:11436">172.16.162.2:11436</a> :> /<a \
href="http://172.16.162.5:9042">172.16.162.5:9042</a>]<br>INFO \
[SharedPool-Worker-2] 2016-01-29 10:45:47,761 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x603349e4, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-3] 2016-01-29 10:48:17,766 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x5bed4eae, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-2] 2016-01-29 10:48:27,767 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x6136756b, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-4] 2016-01-29 10:48:27,767 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x17c83eb8, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-4] 2016-01-29 10:52:07,778 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x1a78b589, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-2] 2016-01-29 10:52:17,779 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x017117b3, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br>INFO \
[SharedPool-Worker-2] 2016-01-29 11:01:37,813 Message.java:532 - Unexpected exception \
during request; channel = [id: 0x3efbd844, /<a \
href="http://172.16.162.4:9042">172.16.162.4:9042</a>]<br><br><div \
class="gmail_extra">Then I don't know if the firewall has something to do on that \
case, becasue it is a local connection over native protocol<br clear="all"></div><div \
class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><div \
style="border-collapse:collapse;font-family:arial,helvetica,sans-serif;font-size:13px"><span \
style="font-family:arial,helvetica,sans-serif;font-size:13px;border-collapse:collapse"><span \
style="border-collapse:separate;font-family:arial;font-size:small"><br></span></span></div><div \
style="border-collapse:collapse"><font face="arial, helvetica, sans-serif" \
size="2">Best regards<br></font></div><div style="border-collapse:collapse"><span \
style="border-collapse:collapse"><span style="border-collapse:separate"><font \
face="arial, helvetica, sans-serif" size="2"><br></font></span></span></div><div \
style="border-collapse:collapse"><span style="border-collapse:collapse"><span \
style="border-collapse:separate"><font face="arial, helvetica, sans-serif" \
size="2">Jean Carlo</font></span></span></div><font face="arial, helvetica, \
sans-serif" size="2"><br>"The best way to predict the future is to invent \
it" Alan Kay</font><br></div></div></div></div> <br><div class="gmail_quote">On \
Fri, Jan 29, 2016 at 11:02 AM, Anuj Wadehra <span dir="ltr"><<a \
href="mailto:anujw_2003@yahoo.co.in" \
target="_blank">anujw_2003@yahoo.co.in</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Hi Jean,<div><br></div><div>Please make sure that your \
Firewall is not dropping TCP connections which are in use. Tcp keep alive on all \
nodes must be less than the firewall setting. Please refer to </div><div><a \
href="https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html" \
target="_blank">https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html</a> \
for details on TCP settings.</div><div><br></div><div><br></div><div>Thanks</div><div>Anuj<br><br><div><a \
href="https://overview.mail.yahoo.com/mobile/?.src=Android" target="_blank">Sent from \
Yahoo Mail on Android</a></div> <br> <blockquote style="margin:0 0 20px 0"> <div>On \
Fri, 29 Jan, 2016 at 3:21 pm, Jean Carlo</div><div><<a \
href="mailto:jean.jeancarl48@gmail.com" \
target="_blank">jean.jeancarl48@gmail.com</a>> wrote:</div> <div><div class="h5"> \
<div style="padding:10px 0 0 20px;margin:10px 0 0 0;border-left:1px solid #6d00f6"> \
<div dir="ltr"><div>Hello guys, <br><br></div>I have a cluster cassandra 2.1.12 with \
6 nodes. All the logs of my nodes are having this messages marked as INFO<br><br><div \
style="text-align:left"></div><div style="text-align:left">INFO \
[SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 - Unexpected exception \
during request; channel = [id: 0xff15eb8c, /<a rel="nofollow" \
href="http://172.16.162.4:9042" \
target="_blank">172.16.162.4:9042</a>]<br>java.io.IOException: Error while read(...): \
Connection reset by peer<br> at \
io.netty.channel.epoll.Native.readAddress(Native Method) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) \
~[netty-all-4.0.23.Final.jar:4.0.23.Final]<br> at \
java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]<br><br></div><div \
style="text-align:left">This happens either the cluster is stressed or not. Btw it is \
not production. The ip marked there (172.16.162.4) belongs to a node of the cluster, \
this is not the only node that appears, acctually we are having all the node's ip \
having that reset by peer problem.<br><br></div><div style="text-align:left">Our \
cluster is having more reads than writes. like 50 reads per second. \
<br><br></div><div style="text-align:left">Any one got the same \
problem?<br></div><div style="text-align:left"><br \
clear="all"></div><div><div><div><div><div><div dir="ltr"><div><div \
style="border-collapse:collapse;font-family:arial,helvetica,sans-serif;font-size:13px"><span \
style="font-family:arial,helvetica,sans-serif;font-size:13px;border-collapse:collapse"><span \
style="border-collapse:separate;font-family:arial;font-size:small"></span></span></div></div></div></div></div></div></div></div><div><div><div><div><div><div \
dir="ltr"><div><div style="border-collapse:collapse;font-family:arial,helvetica,sans-serif;font-size:13px"><span \
style="font-family:arial,helvetica,sans-serif;font-size:13px;border-collapse:collapse"><span \
style="border-collapse:separate;font-family:arial;font-size:small"><br></span></span></div><div \
style="border-collapse:collapse"><font face="arial, helvetica, sans-serif" \
size="2">Best regards<br></font></div><div style="border-collapse:collapse"><span \
style="border-collapse:collapse"><span style="border-collapse:separate"><font \
face="arial, helvetica, sans-serif" size="2"><br></font></span></span></div><div \
style="border-collapse:collapse"><span style="border-collapse:collapse"><span \
style="border-collapse:separate"><font face="arial, helvetica, sans-serif" \
size="2">Jean Carlo</font></span></span></div><font face="arial, helvetica, \
sans-serif" size="2"><br>"The best way to predict the future is to invent \
it" Alan Kay</font><br></div></div></div></div> </div></div></div></div> </div> \
</div></div></blockquote></div></blockquote></div><br></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic