[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-user
Subject: Re: Broken pipe with Thrift
From: Aaron Morton <aaron () thelastpickle ! com>
Date: 2013-12-31 2:06:46
Message-ID: AE985E80-FAB3-432B-B0BE-B0D3B24EC2C7 () thelastpickle ! com
[Download RAW message or body]
> One question, which is confusing , it's a server side issue or client side?
Check the server log for errors to make sure it’s not a server side issue.
Also check if there could be something in network that is killing long lived \
connections. Check the thrift lib the client is using is the same as the one in the \
cassandra lib on the server.
Can you do some simple tests using cqlsh from the client machine? That would \
eliminate the client driver.
Hope that helps.
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 25/12/2013, at 4:35 am, Steven A Robenalt <srobenal@stanford.edu> wrote:
> In our case, the issue was on the server side, but since you're on the 1.2.x \
> branch, it's not likely to be the same issue. Hopefully, somone else who is using \
> the 1.2.x branch will have more insight than I do.
>
> On Mon, Dec 23, 2013 at 11:52 PM, Vivek Mishra <mishra.vivs@gmail.com> wrote:
> Hi Steven,
> One question, which is confusing , it's a server side issue or client side?
>
> -Vivek
>
>
>
>
> On Tue, Dec 24, 2013 at 12:30 PM, Vivek Mishra <mishra.vivs@gmail.com> wrote:
> Hi Steven,
> Thanks for your reply. We are using version 1.2.9.
>
> -Vivek
>
>
> On Tue, Dec 24, 2013 at 12:27 PM, Steven A Robenalt <srobenal@stanford.edu> wrote:
> Hi Vivek,
>
> Which release are you using? We had an issue with 2.0.2 that was solved by a fix in \
> 2.0.3.
>
> On Mon, Dec 23, 2013 at 10:47 PM, Vivek Mishra <mishra.vivs@gmail.com> wrote:
> Also to add. It works absolutely fine on single node.
>
> -Vivek
>
>
> On Tue, Dec 24, 2013 at 12:15 PM, Vivek Mishra <mishra.vivs@gmail.com> wrote:
> Hi,
> I have a 6 node, 2DC cluster setup. I have configured consistency level to QUORUM. \
> But very often i am getting "Broken pipe" \
> com.impetus.client.cassandra.CassandraClientBase (CassandraClientBase.java:1926) - \
> Error while executing native CQL query Caused by: .
> org.apache.thrift.transport.TTransportExceptionjava.net.SocketException: Broken \
> pipe at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransportjava:147)
> at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
> at
> org.apache.cassandra.thrift.Cassandra$Client.send_execute_cql3_query(Cassandra.java:1556)
> at
> org.apache.cassandra.thrift.Cassandra$Client.execute_cql3_query(Cassandra.java:1546)
>
>
> I am simply reading few records from a column family(not huge amount of data)
>
> Connection pooling and socket time out is properly configured. I have even modified \
> read_request_timeout_in_ms
> request_timeout_in_ms
> write_request_timeout_in_ms in cassandra.yaml to higher value.
>
>
> any idea? Is it an issue at server side or with client API?
>
> -Vivek
>
>
>
>
> --
> Steve Robenalt
> Software Architect
> HighWire | Stanford University
> 425 Broadway St, Redwood City, CA 94063
>
> srobenal@stanford.edu
> http://highwire.stanford.edu
>
>
>
>
>
>
>
>
>
>
> --
> Steve Robenalt
> Software Architect
> HighWire | Stanford University
> 425 Broadway St, Redwood City, CA 94063
>
> srobenal@stanford.edu
> http://highwire.stanford.edu
>
>
>
>
>
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space;"><blockquote type="cite"><div \
class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" \
style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, \
204, 204); border-left-style: solid; padding-left: 1ex; position: static; z-index: \
auto;"><div dir="ltr"><div>One question, which is confusing , it's a server side \
issue or client side?</div><span class="HOEnZb"><font \
color="#888888"></font></span></div></blockquote></div></div></blockquote><font \
color="#888888"></font><div>Check the server log for errors to make sure it’s not a \
server side issue. </div><div>Also check if there could be something in network \
that is killing long lived connections. </div><div>Check the thrift lib the \
client is using is the same as the one in the cassandra lib on the \
server. </div><div><br></div><div>Can you do some simple tests using cqlsh from \
the client machine? That would eliminate the client \
driver. </div><div><br></div><div>Hope that \
helps.</div><div><br></div><div><br></div><div apple-content-edited="true"> <div \
style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; \
font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: \
normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; \
white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; \
-webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space; "><div>-----------------</div><div>Aaron \
Morton</div><div>New \
Zealand</div><div>@aaronmorton</div><div><br></div><div>Co-Founder & Principal \
Consultant</div><div>Apache Cassandra Consulting</div><div><a \
href="http://www.thelastpickle.com">http://www.thelastpickle.com</a></div></div> \
</div> <br><div><div>On 25/12/2013, at 4:35 am, Steven A Robenalt <<a \
href="mailto:srobenal@stanford.edu">srobenal@stanford.edu</a>> wrote:</div><br \
class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">In our case, \
the issue was on the server side, but since you're on the 1.2.x branch, it's not \
likely to be the same issue. Hopefully, somone else who is using the 1.2.x branch \
will have more insight than I do.</div> <div class="gmail_extra"><br><br><div \
class="gmail_quote">On Mon, Dec 23, 2013 at 11:52 PM, Vivek Mishra <span \
dir="ltr"><<a href="mailto:mishra.vivs@gmail.com" \
target="_blank">mishra.vivs@gmail.com</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; \
border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; \
position: static; z-index: auto;"><div dir="ltr">Hi Steven,<div>One question, which \
is confusing , it's a server side issue or client side?</div><span \
class="HOEnZb"><font color="#888888"><div> \
<br></div><div>-Vivek</div><div><br></div><div><br></div></font></span></div><div \
class="HOEnZb"><div class="h5"><div class="gmail_extra"> <br><br><div \
class="gmail_quote">On Tue, Dec 24, 2013 at 12:30 PM, Vivek Mishra <span \
dir="ltr"><<a href="mailto:mishra.vivs@gmail.com" \
target="_blank">mishra.vivs@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div dir="ltr">Hi Steven,<div>Thanks for your reply. We are using version \
1.2.9.</div><span><font \
color="#888888"><div><br></div><div>-Vivek</div></font></span></div><div><div \
class="gmail_extra"> <br><br><div class="gmail_quote">On Tue, Dec 24, 2013 at 12:27 \
PM, Steven A Robenalt <span dir="ltr"><<a href="mailto:srobenal@stanford.edu" \
target="_blank">srobenal@stanford.edu</a>></span> wrote:<br> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi Vivek,<div><br></div><div>Which release are \
you using? We had an issue with 2.0.2 that was solved by a fix in 2.0.3.</div>
</div><div class="gmail_extra"><div><br><br><div class="gmail_quote">On Mon, Dec 23, \
2013 at 10:47 PM, Vivek Mishra <span dir="ltr"><<a \
href="mailto:mishra.vivs@gmail.com" \
target="_blank">mishra.vivs@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Also to add. It works absolutely fine on \
single node.<span><font color="#888888"><div><br> \
</div><div>-Vivek</div></font></span></div><div><div class="gmail_extra"><br><br><div \
class="gmail_quote">On Tue, Dec 24, 2013 at 12:15 PM, Vivek Mishra <span \
dir="ltr"><<a href="mailto:mishra.vivs@gmail.com" \
target="_blank">mishra.vivs@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi,<div>I have a 6 node, 2DC cluster setup. I \
have configured consistency level to QUORUM. But very often i am getting \
"Broken pipe"</div>
<div><span style="font-size:16px;font-family:monospace">com.impetus.client.cassandra.CassandraClientBase</span><br \
style="font-size:16px;font-family:monospace"> <span \
style="font-size:16px;font-family:monospace">(CassandraClientBase.java:1926) - Error \
while executing native CQL</span><br \
style="font-size:16px;font-family:monospace"><span \
style="font-size:16px;font-family:monospace">query Caused by: .</span><br \
style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace">org.apache.thrift.transport.TTransportException</span><span \
style="font-size:16px;font-family:monospace">java.net.SocketException: Broken \
pipe</span><br style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace"> \
at</span><br style="font-size:16px;font-family:monospace"><span \
style="font-size:16px;font-family:monospace">org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport</span><span \
style="font-size:16px;font-family:monospace">java:147)</span><br \
style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace">at </span><span \
style="font-size:16px;font-family:monospace">org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.jav</span><span \
style="font-size:16px;font-family:monospace">a</span><span \
style="font-size:16px;font-family:monospace">:156)</span><br \
style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace">at</span><br \
style="font-size:16px;font-family:monospace"><span \
style="font-size:16px;font-family:monospace">org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)</span><br \
style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace">at</span><br \
style="font-size:16px;font-family:monospace"><span \
style="font-size:16px;font-family:monospace">org.apache.cassandra.thrift.Cassandra$Client.send_execute_cql3_query(Ca</span><span \
style="font-size:16px;font-family:monospace">s</span><span \
style="font-size:16px;font-family:monospace">sandra.java:1556)</span><br \
style="font-size:16px;font-family:monospace">
<span style="font-size:16px;font-family:monospace">at</span></div><div><span \
style="font-size:16px;font-family:monospace">org.apache.cassandra.thrift.Cassandra$Client.execute_cql3_query(Cassand</span><span \
style="font-size:16px;font-family:monospace">r</span><span \
style="font-size:16px;font-family:monospace">a.java:1546)</span><br>
</div><div><span style="font-size:16px;font-family:monospace"><br></span></div><div><span \
style="font-size:16px;font-family:monospace"><br></span></div><div><font \
face="monospace" size="3">I am simply reading few records from a column family(not \
huge amount of data)</font></div>
<div><font face="monospace" size="3"><br></font></div><div><font face="monospace" \
size="3">Connection pooling and socket time out is properly configured. I have even \
modified </font></div>
<div><font face="monospace" \
size="3">read_request_timeout_in_ms<br></font></div><div><font face="monospace" \
size="3">request_timeout_in_ms<br></font></div><div><font face="monospace" \
size="3">write_request_timeout_in_ms in cassandra.yaml to higher value.<br>
</font></div><div><font face="monospace" size="3"><br></font></div><div><font \
face="monospace" size="3"><br></font></div><div><font face="monospace" size="3">any \
idea? Is it an issue at server side or with client API?</font></div>
<span><font color="#888888">
<div><font face="monospace" size="3"><br></font></div><div><font face="monospace" \
size="3">-Vivek</font></div></font></span></div> </blockquote></div><br></div>
</div></blockquote></div><br><br clear="all"><div><br></div></div><span><font \
color="#888888">-- <br><div dir="ltr"><div><font face="verdana, sans-serif">Steve \
Robenalt<br></font></div><div><font face="verdana, sans-serif">Software Architect<br>
</font></div>
<div><font face="verdana, sans-serif">HighWire | Stanford \
University </font></div>
<div><span style="font-family:verdana,sans-serif">425 Broadway St, Redwood City, CA \
94063</span><font face="verdana, sans-serif"> <br></font><div \
style="font-family:arial"><br></div></div><font face="verdana, sans-serif"><a \
href="mailto:srobenal@stanford.edu" target="_blank">srobenal@stanford.edu</a> <br>
</font><div><font face="verdana, sans-serif"><a href="http://highwire.stanford.edu/" \
target="_blank">http://highwire.stanford.edu</a> </font></div>
<div><font face="verdana, sans-serif"><br></font></div><div><font face="verdana, \
sans-serif"><i><br></i></font></div>
<div><br><br><div style="font-family:arial">
<span style="font-family:verdana,sans-serif;font-size:13px"><br></span></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
dir="ltr"><div><font face="verdana, sans-serif">Steve \
Robenalt<br></font></div><div><font face="verdana, sans-serif">Software \
Architect<br></font></div> <div><font face="verdana, sans-serif">HighWire | Stanford \
University </font></div>
<div><span style="font-family:verdana,sans-serif">425 Broadway St, Redwood City, CA \
94063</span><font face="verdana, sans-serif"> <br></font><div \
style="font-family:arial"><br></div></div><font face="verdana, sans-serif"><a \
href="mailto:srobenal@stanford.edu" target="_blank">srobenal@stanford.edu</a> <br> \
</font><div><font face="verdana, sans-serif"><a href="http://highwire.stanford.edu/" \
target="_blank">http://highwire.stanford.edu</a> </font></div>
<div><font face="verdana, sans-serif"><br></font></div><div><font face="verdana, \
sans-serif"><i><br></i></font></div>
<div><br><br><div style="font-family:arial">
<span style="font-family:verdana,sans-serif;font-size:13px"><br></span></div></div></div>
</div>
</blockquote></div><br></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic