[prev in list] [next in list] [prev in thread] [next in thread]
List: httpclient-users
Subject: RE: Connection Reset errors
From: Mark Claassen <mac01 () donnell ! com>
Date: 2013-01-31 20:30:15
Message-ID: 003b01cdfff1$c76f1700$564d4500$ () com
[Download RAW message or body]
This is the longest she has gone without getting the error, so I am calling it fixed. \
I learned a lot. And, in going to 4.2.3, it is going noticeable faster now. It was \
hard getting here, but everything is good now. Thanks again!
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 4:55 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
> It looks like all this misery was self inflicted.
Thank you so much for your help. (Setting the linger to 0 was not my idea...)
> I had thought I had tried to change the linger in an earlier test
If only I would have double checked myself on this one.
The user in question has not had time to test it yet; tomorrow I will know for sure. \
Hopefully what caused the error in my test cases also caused the problem for her.
Mark
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 4:30 PM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Tue, 2013-01-29 at 15:27 -0500, Mark Claassen wrote:
> Wow, that certainly helped my test case! Now to try it for real.
>
> This leaves me with 2 big questions:
> 1) Is there a reason for Tomcat to behave this way?
In the early days of HTTP/1.1 many HTTP agents had issues with persistent connection \
support. This feature was most likely intended to help broken clients recover by \
forcing them to open a new connection after a certain number of requests.
> 2) Why is this not WAY more common?
>
> I think the answer to #2 might be (at least partially) that we have our \
> connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, when \
> Tomcat closed the connection, if the client was a bit too slow, the socket would \
> close before the client could finish reading all the data.
That explains it. By setting SO_LINGER to zero you are basically instructing Tomcat \
to terminate connections abnormally by sending a RST (connection reset) command \
instead of a normal TCP termination procedure.
It looks like all this misery was self inflicted.
Oleg
> I had thought I had tried to change the linger in an earlier test, but maybe I \
> forgot to save the change or something.
> In any event, she is trying this and we will see how it goes.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 3:12 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Thanks! I will check it out!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 3:09 PM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> > CLUE!
> >
> > I opened up the cipher suites on Tomecat like I mentioned and then made my own \
> > socketfactory (limiting the cipher suites) as I saw in an example. Now when I \
> > run the program it dies at the same place every time...on the 101st connection. \
> > This has to be significant in some way...although I don't know how.
> > I about doubled the size of the message I send, and it makes no difference.
> >
> > I put the iteration count in the HttpHeader so I could track it in
> > the Tomcat access log. Tomcat shows that it receives this message
> > (message 100) and that it thinks it succeeded (status == 200)
> >
> > The SocketTimeoutException I noticed before is everywhere in the logs. This is \
> > from the StaleConnectionCheck and is likely irrelevant.
> >
>
>
> Tomcat by default closes persistent connections once it has been used to serve 100 \
> requests.
> See 'maxKeepAliveRequests' parameter:
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
> Try setting this value to -1 and see if that makes the problem go away.
>
> Oleg
>
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 2:01 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I saw some things on the web with this error, but those seemed to be errors on \
> > startup because particular protocols where not enabled. I changed my Tomcat \
> > configuration to use SSL instead of TLS, and removed the cipherSuites attribute. \
> > Similar result. However I saw the below in the debug output.
> > What is the socket timeout doing there, and why is there a SocketTimeoutException \
> > when this is all occurring in the same second.
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> > FINE: << " <p>If you page is not redirected, click <a \
> > href="JViewer/">here</a>[\n]" Thread-00:Jan 29, 2013 1:48:32 PM \
> > org.apache.http.impl.conn.Wire wire Thread-00, handling exception: \
> > java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) \
> > called
> > FINE: << " </BODY>[\n]"
> > Thread-00, setSoTimeout(45000) called
> >
> > [snip]
> >
> > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> > length = 176
> >
> > [snip]
> >
> > [Raw read]: length = 5
> > 0000: 17 03 01 03 0A .....
> > Thread-00, handling exception: java.net.SocketException: Connection reset %% \
> > Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: \
> > fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len \
> > = 18
> > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> > 0010: C8 08 ..
> > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> > sending alert: java.net.SocketException: Connection reset by peer:
> > socket write error Thread-00, called closeSocket() Thread-00, called
> > close() Thread-00:Jan 29, 2013 1:48:32 PM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 12:39 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Don't know if this will help either...but here is hoping!
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Tuesday, January 29, 2013 9:03 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > > I ran my test program using 4.01 and it seems to fail even faster.
> > > (Switching a few things around to get it to compile, or course.
> > > Using
> > > ThreadSafeClientConnManager.)
> > >
> > > This is odd, since the user said she didn't experience the problem until the \
> > > upgrade which gave her the newer version. Must also be something else going \
> > > on.
> > > Still, the Java.net stuff is completely error free. (I can't say
> > > for sure if she would notice automatic retries, if they occurred.)
> > >
> > > Is there any way I can reduce the occurrences of the error? Something I could \
> > > set for specific users that are experiencing this problem?
> >
> > Mark,
> >
> > This log unfortunately does not help. Interesting bit, though, it that a \
> > connection reset happens while trying to read a head of a new message. So, \
> > clearly the stale connection check did not work. In the former days (Java 1.2 and \
> > possibly 1.3) state connection checking was simply useless with SSL encrypted \
> > connections. Apparently it can still be ineffective even with modern JREs.
> > So, running your application with SSL debugging turned on should be
> > more
> > informative:
> >
> > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDeb
> > ug
> > .html
> >
> > Oleg
> >
> > > ------------------------------------
> > > Stack trace and debug using 4.0.1
> > >
> > > FINE: Stale connection check
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 28, 2013 12:08:45
> > > PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.client.DefaultRequestDirector execute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection shut down
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > > releaseConnection
> > > FINE: Released connection is not reusable.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > > FINE: Releasing connection
> > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute
> > > notifyWaitingThread
> > > FINE: Notifying no-one, there are no waiting threads
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 12:30 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I was able to put a break point in DefaultClientConnection.close(). It \
> > > sometimes gets called when a connection is releasedManagedConnection is called. \
> > > When the Connection Reset happens, it gets called 3 times in quick succession. \
> > > However, looks like this is just normal processing for the failed connection. \
> > > Whatever error is the root cause seems to have already occurred.
> > > The Connection Reset calls do not seem to be correlated to the normal \
> > > releasedManagedConnection calls that work.
> > > First call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > HttpRequestExecutor.closeConnection:144
> > > HttpRequestExecutor.execute:131
> > > DefaultRequestDirector.tryExecute:717
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Second call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > DefaultRequestDirector.tryExecute:723
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Third call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > HttpPoolEntry.close:89
> > > AbstractConnPool.release:323
> > > PoolingClientConnectionManager.releaseConnection:278
> > > ManagedClientConnectionImpl.abortConnection:463
> > > DefaultRequestDirector.abortConnection:1157
> > > DefaultRequestDirector.execute:621
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:41 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 \
> > > (pool size). So, I had just 1 thread making the request over and over again \
> > > without pausing.
> > > Below is the last part of what I got.
> > >
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > closeExpiredConnections
> > > FINE: Closing expired connections
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > requestConnection
> > > FINE: Connection request: [route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > leaseConnection
> > > FINE: Connection leased: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Stale connection check
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache
> > > process
> > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestTargetAuthentication
> > > process
> > > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestProxyAuthentication process
> > > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > tryExecute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 25, 2013 11:38:01
> > > AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 0 of 1; total allocated: 0 of 1]
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > > at java.lang.Thread.run(Thread.java:662)
> > > Java Result: 1
> > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original
> > > Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:25 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am running the test using a non-SSL connection and the Connection Reset is \
> > > not happening. I am going to continue to run the test, but none of the SSL \
> > > tests ran this long without failing.
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:07 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Before the upgrade, she was using HttpClient 4.01.
> > > ---
> > > Not sure if it is kosher to send an attachment to the list, but here is a test \
> > > program. This errors for me after just a minute or two.
> > > The website it is connecting to is just a Tomcat server running
> > > the default ROOT webapp. It is delivering the default HTML page
> > > we have set up on that. I would imagine that it is ignoring the
> > > POST and treating it like a GET
> > >
> > > In production, where she is having the problem, it is using the default \
> > > HttpsURLConnection socket factory. However, for testing, I am using this one \
> > > to bypass the normal certificate checks and stuff. It happens to both of us, \
> > > so I don't think the error has anything to do with the socket factory. The \
> > > message that we send in production are also a lot longer, so I don't think it \
> > > has anything to do with message size either.
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 25, 2013 7:46 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > Mark
> > >
> > > You do not need to do anything special other than closing the content input \
> > > stream of the response entity (if it is present) to ensure proper resource \
> > > deallocation. What is important that it has to be done no matter if you need \
> > > the content or not. I do not think the problem you are dealing with has \
> > > anything to do with resource management.
> > > There are probably several things you might want to try out
> > >
> > > (1) if you can confirm that this issue is HttpClient version specific and tell \
> > > me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 \
> > > to 4.1) I could analyze the differences and try to figure out what could \
> > > potentially be the cause of connection resets.
> > > (2) This is entirely possible this is somehow SSL related and connection gets \
> > > reset by the SSL stack. You might want to run your app with SSL debug enabled \
> > > and see if there is anything in the SSL logs that could help explain connection \
> > > being reset.
> > > (3) Otherwise there is no way around getting very close and personal with \
> > > Wireshark.
> > > Oleg
> > >
> > >
> > >
> > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > > Something else I thought of is this: We have some transactions where the \
> > > > response is not too important. For instance, if we send a pulse to the \
> > > > server to let it know we are still here. We don't really care what the \
> > > > response is, if we didn't get an error then we did our part. If we did get \
> > > > and error, we will try again soon anyway, so no big deal.
> > > > Is there something that needs to be done on a connection that is closed \
> > > > before reading all the data? I have never experienced reading stale data, so \
> > > > this seems a bit unlikely to me. However, if there is implicit cleanup that \
> > > > is being done, could this be, in some way, causing my problems?
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Thursday, January 24, 2013 10:29 AM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > I am just not convinced this is a network thing. It happens too often for \
> > > > her, and never with the java.net connection. She has a newer machine, maybe \
> > > > threading issues are just more apparent on her hardware.
> > > > So, maybe the problem is somewhere in my code. Can you tell me what are the \
> > > > methods that might release a connection back to the pool? I am wondering if \
> > > > there is a way I could be releasing the connection back to the pool, it gets \
> > > > picked up by something else, and then the original thread tries to close it \
> > > > again or something.
> > > > The docs say the closing the InputStream will do it, but will anything else \
> > > > do it implicitly?
> > > > --
> > > >
> > > > Another avenue I am investigating is trying to force the issue by making the \
> > > > connections work harder. I am trying to force the issue doing a lot of \
> > > > requests all at once. I reduced my MAX down to 1 to force contention. \
> > > > Everything seems to work pretty well. However, I did get the Connection
> > > > Reset error twice on my machine! The logs were fairly regular
> > > > looking since there is only one connection at a time. However,
> > > > when this happened, it looks like it is sending 2 request at
> > > > once. This shouldn't be happening because there is only one connection.
> > > > (Granted, it is hard to tell if the logging is indicating a
> > > > threading issue, or if the "Connection Reset" error caused the
> > > > logging abnormality.)
> > > >
> > > > I don't need to block around the "httpClient.execute(request);" do I?
> > > >
> > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > > --
> > > >
> > > > I do this to establish the connection:
> > > >
> > > > This is run once in the setup
> > > > -----------------------------
> > > > connectionManager = new
> > > > PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > >
> > > >
> > > >
> > > > This is called when I want to make a request
> > > > --------------------------------------------
> > > > response = httpClient.execute(request);
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 5:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Well, the user went all day without any errors while using the Java.net \
> > > > connector.
> > > > The error using the HttpClient connector has been happening with HttpClient \
> > > > 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is \
> > > > continuing under 4.2.3). I will have to check on the upgrade path and see if \
> > > > she had been using a 4.0.x version before this, or if they jumped from a \
> > > > version that used a 3.x version.
> > > > Mark
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:47 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Oh, I forgot to add that all my requests are POST requests.
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:02 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > While she is testing the java.net stuff, I decided to look into the \
> > > > possibility of retrying the request. Looking at this more closely, I see \
> > > > that the error is on the reading of the data, and the write seems to succeed. \
> > > > This gives me something to check on, since I should be able to see whether or \
> > > > not the servlet actually got her request or not.
> > > > However, is this a common manifestation of the Connection Reset error? The \
> > > > write succeeds (or fails silently) and then it is on the read the problem can \
> > > > be detected? Or does this mean that the socket was fine during the write \
> > > > operation, but then was somehow severed before the read?
> > > > Mark
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 12:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Looks like the stale connection check is enabled by default anyway, so I must \
> > > > have been using it all along.
> > > > > It may well be that HttpURLConnection silently retries failed
> > > > > requests
> > > > Is this a common thing to do, like for browsers? Granted, she is in our app \
> > > > more than most others, but I don't think she experiences issues with other \
> > > > applications.
> > > > She is not getting a Connection Refused or some other unable to read \
> > > > exception, but a Connection Reset. The stale connection check passed, so \
> > > > HttpClient thinks this is a good connection, right? Or are there limits to \
> > > > how reliable a stale connection check can be?
> > > > public boolean isStale() {
> > > > if (!isOpen()) {
> > > > return true;
> > > > }
> > > > if (isEof()) {
> > > > return true;
> > > > }
> > > > try {
> > > > this.inbuffer.isDataAvailable(1);
> > > > return isEof();
> > > > } catch (SocketTimeoutException ex) {
> > > > return false;
> > > > } catch (IOException ex) {
> > > > return true;
> > > > }
> > > > }
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 11:24 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. \
> > > > > Not sure how things were going to turn out with HttpClient, I kept that \
> > > > > code in place and made it so I could use either API. I had her switch back \
> > > > > to using the java.net stuff the other day and the problem appeared to go \
> > > > > away. I have configured her system to use the java.net connector again \
> > > > > today. Before I make any conclusions I need some more data from her on \
> > > > > each connector. However, the previous data seems to indicate that it is \
> > > > > only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > >
> > > > It may well be that HttpURLConnection silently retries failed requests \
> > > > without your knowing about it. It certainly did that when I last touched it \
> > > > (which admittedly was a looooooong time ago), and that was precisely the \
> > > > reason I had to stop using it and to look for a better alternative.
> > > >
> > > > > Some other things of note:
> > > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > > this.)
> > > > > - The application is launched via Webstart
> > > > > - The SocketFactory is created with: fact = new
> > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory
> > > > > ()
> > > > > ,v
> > > > > er
> > > > > if
> > > > > ier);
> > > > >
> > > >
> > > > I do not thing any of these should matter.
> > > >
> > > > Oleg
> > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > > Thanks for the explanation.
> > > > > > > pro-actively evicting expired connections and connections
> > > > > > > that have been idle
> > > > > > Seems like just doing the stale check would more reliable,
> > > > > > and speed does not seem to be a factory (especially with
> > > > > > 4.2.3)
> > > > > >
> > > > > > I got in configured and distributed to this user and she is
> > > > > > still having the same problem. Any other ideas? I double
> > > > > > checked my code, and it looks like everything is correct.
> > > > > > (She is not using a
> > > > > > proxy.)
> > > > > >
> > > > >
> > > > > This can also be genuine network stability issues. Essentially, connection \
> > > > > resets are inevitable and robust HTTP services have to be prepared to deal \
> > > > > with them (most likely by re-trying idempotent methods automatically and \
> > > > > reporting the problem back to the caller otherwise).
> > > > > Oleg
> > > > >
> > > > >
> > > > > >
> > > > > > Method that makes the request (from multiple threads)
> > > > > > {
> > > > > > response = httpClient.execute(request);
> > > > > > }
> > > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = \
> > > > > > 10) {
> > > > > > HttpParams params = new BasicHttpParams();
> > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new \
> > > > > > TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS)); \
> > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, \
> > > > > > CONNECTION_ESTABLISH_TIMEOUT * 1000); \
> > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, \
> > > > > > DEFAULT_HTTP_SOCKET_BUFFER_SIZE); \
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, \
> > > > > > Boolean.TRUE); configureProxy(params,protocol);
> > > > > >
> > > > > > connectionManager = new \
> > > > > > PoolingClientConnectionManager(schemeRegistry); \
> > > > > > connectionManager.setDefaultMaxPerRoute(MAX); \
> > > > > > connectionManager.setMaxTotal(MAX);
> > > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > > }
> > > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > > Proxy proxy = getProxy();
> > > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > > //SSL or not, we set the proxy up as HTTP
> > > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), \
> > > > > > address.getPort(), SCHEME_HTTP); \
> > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy); \
> > > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO); \
> > > > > > if (!scheme.equals(SCHEME_HTTP)) { //If the scheme is HTTPS, we need to \
> > > > > > register an HTTP scheme for the proxy. schemeRegistry.register(new \
> > > > > > Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > > To: HttpClient User Discussion
> > > > > > Subject: Re: Connection Reset errors
> > > > > >
> > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > > >
> > > > > > Basically 'connection reset' errors are a direct result of a general \
> > > > > > limitation of the blocking I/O model: when not blocked in an I/O \
> > > > > > operation there is no way for the socket to get notified of the opposite \
> > > > > > endpoint terminating the connection. While kept alive in the pool \
> > > > > > blocking HTTP connections can get stale. However, the only way to find it \
> > > > > > out is to try to attempt an I/O operation on such connection.
> > > > > > > I noticed that there is a stale connection check:
> > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTIO
> > > > > > > N_
> > > > > > > CH
> > > > > > > EC
> > > > > > > K,
> > > > > > > Boolean.TRUE);
> > > > > > >
> > > > > > > Is this something that I should be using? I am connecting to the same \
> > > > > > > source over and over again.
> > > > > > > Mark
> > > > > > >
> > > > > >
> > > > > > Yes, this is what you should be using. Another alternative would be \
> > > > > > pro-actively evicting expired connections and connections that have been \
> > > > > > idle longer than a given period of time.
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > > To: 'HttpClient User Discussion'
> > > > > > > Subject: Connection Reset errors
> > > > > > >
> > > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > > did not think this had do to with HttpClient, so much as
> > > > > > > other factors on her machine and the server. However, I did a bit of \
> > > > > > > searching and see that people have commented on this error with \
> > > > > > > HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that \
> > > > > > > would help at all, and it didn't.
> > > > > > > This user is fine for a while, and then will get the
> > > > > > > Connection Reset error. She then just retries the request, and it \
> > > > > > > works. Any tips on how to resolve this would be greatly appreciated.
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > > <Connector
> > > > > > > port="5502"
> > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > > server="Unknown"
> > > > > > > connectionLinger="0"
> > > > > > > socket.soTimeout="300000"
> > > > > > > SSLEnabled="true"
> > > > > > > maxThreads="150"
> > > > > > > scheme="https"
> > > > > > > secure="true"
> > > > > > > clientAuth="false"
> > > > > > > keystoreFile="-----"
> > > > > > > keystoreType="PKCS12"
> > > > > > > keystorePass="-----"
> > > > > > > sslProtocol="TLS"
> > > > > > >
> > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC
> > > > > > > _S
> > > > > > > HA ,T LS _D HE
> > > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WIT
> > > > > > > H_ 3D ES _E DE _C
> > > > > > > BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > > >
> > > > > > >
> > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > > Message (Connection reset)
> > > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown \
> > > > > > > Source) at \
> > > > > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown \
> > > > > > > Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown \
> > > > > > > Source) at \
> > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > > at
> > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > > ja
> > > > > > > va:138)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic