[prev in list] [next in list] [prev in thread] [next in thread]
List: httpclient-users
Subject: Re: Response timeout issue
From: Oleg Kalnichevski <olegk () apache ! org>
Date: 2019-05-30 9:39:39
Message-ID: ff4b9cd5c1151fb3bd0650529747ab8f6532b5c9.camel () apache ! org
[Download RAW message or body]
On Wed, 2019-05-29 at 18:56 +0200, Joan grupoventus wrote:
> Hi Oleg,
>
> I'm still trying to manage this with our client ... we are not able
> to reproduce it in our environment.
>
> In the meantime we have come across with another issue. I explain it
> just to know if you are aware that someone else has experienced
> something similar with HttpClient5. After 7 days working correctly,
> suddenly the http pool becomes exhausted and all the asynchronous
> tasks hang until the container asynchronous timeout is reached. What
> we see is the request is sent to the backend but no response is
> obtained, and finally the asynchronous task finishes due to the
> container timeout after 150s ... no response timeout is thrown (it's
> set at 90s).
>
Hi Joan
No, I am not aware of anyone else experiencing similar issues.
Oleg
> Our client restarted Tomcat, we could only get a thread stack ... not
> sure if this can be of any utility but I could attach it ...
>
> Thanks,
>
> Joan.
>
>
> -----Original Message-----
> From: Oleg Kalnichevski <olegk@apache.org>
> Sent: Monday, May 27, 2019 11:10 PM
> To: HttpClient User Discussion <httpclient-users@hc.apache.org>
> Subject: Re: Response timeout issue
>
> Joan
>
> Please raise a JIRA for this issue and attach a wire log with time
> stamps of a _complete_ message exchange demonstrating the incorrect
> behavior.
>
> I will review connection management code tomorrow.
>
> Oleg
>
>
> On Mon, 2019-05-27 at 19:33 +0200, Joan ventusproxy wrote:
> > Hello,
> >
> >
> >
> > We are having an issue with HttpClient 5 beta4 / HttpCore 5 Beta7
> > and
> > socket timeout after switching from asyncClient 1.3.
> >
> >
> >
> > Our requests have a response timeout of 4 seconds. When a response
> > timeout occurs we see this in our log:
> >
> > 2019-05-27 13:05:35 [62.73.191.161]
> > GetAccomodationAvail
> > Response timeout java.net.SocketTimeoutException:
> > 4,000 MILLISECONDS
> >
> > at
> > org.apache.hc.core5.io.SocketTimeoutExceptionFactory.create(SocketT
> > im
> > eoutExceptionFactory.java:50)
> >
> > at
> > org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onTim
> > eo
> > ut(AbstractHttp1StreamDuplexer.java:378)
> >
> > at
> > org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.timeo
> > ut
> > (AbstractHttp1IOEventHandler.java:81)
> >
> > at
> > org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.timeout
> > (C
> > lientHttp1IOEventHandler.java:39)
> >
> > at
> > org.apache.hc.core5.reactor.InternalDataChannel.onTimeout(InternalD
> > at
> > aChannel.java:188)
> >
> > at
> > org.apache.hc.core5.reactor.InternalChannel.checkTimeout(InternalCh
> > an
> > nel.java:67)
> >
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.checkTimeout(Single
> > Co
> > reIOReactor.java:232)
> >
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.validateActiveChann
> > el
> > s(SingleCoreIOReactor.java:165)
> >
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCor
> > eI
> > OReactor.java:127)
> >
> > at
> > org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(Abs
> > tr
> > actSingleCoreIOReactor.java:81)
> >
> > at
> > org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.jav
> > a:
> > 44)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> >
> >
> > That's ok, but when we check our statistics the average time in
> > case
> > of a response timeout is around 1s. For example the above request
> > has
> > been logged with response timeout:
> >
> > 13:05:35.423,748,http.responseTimeout,13:05:36.171
> >
> >
> >
> > But the total time consumed by this request, since it enters to
> > our
> > app (at 13:05:35.423) until the response is returned (at
> > 13:05:36.171), is just 748ms, not more than 4s.
> >
> >
> >
> > After making a lot of tests, the solution came disabling keep
> > alive.
> > Our pool had keep alive enabled (with a value of 20s), disabling
> > keep
> > alive then everything works fine. Response timeouts almost
> > disappear
> > and when one occurs, the logged time is around 4s. If we enable
> > keepalive, the issue occurs again.
> >
> >
> >
> > Our KeepAliveStrategy is the same used in asyncClient (changing to
> > the
> > http5 cclasses). It's set here:
> >
> > HttpAsyncClientBuilder hcBuilder =
> > HttpAsyncClients.custom().setKeepAliveStrategy(this.setKeepAliveStr
> > at
> > egy(pool.getKeepAliveDuration())) …
> >
> >
> >
> > And the class is:
> >
> >
> >
> > ( … )
> >
> > public TimeValue getKeepAliveDuration(HttpResponse response,
> > HttpContext context)
> >
> >
> >
> > {
> >
> > // If we have set a keepalive in our pool, it's used.
> >
> > if (this.keepAliveDuration > 0) return
> > (TimeValue.ofMilliseconds(this.keepAliveDuration));
> >
> >
> >
> > // Otherwise search for it in the client response.
> >
> > BasicHeaderElementIterator it = new
> > BasicHeaderElementIterator(response.headerIterator(HTTP.CONN_KEEP_A
> > LI
> > VE));
> >
> >
> >
> > while (it.hasNext())
> >
> > {
> >
> > HeaderElement he = it.next();
> >
> > String value = he.getValue();
> >
> >
> >
> > if ( (value != null) &&
> > (he.getName().equalsIgnoreCase("timeout")) )
> >
> > {
> >
> > try { return (TimeValue.ofSeconds(Long.parseLong(value))); }
> >
> > catch (NumberFormatException ignore) { break; }
> >
> > }
> >
> > }
> >
> >
> >
> > // Otherwise return '100ms'
> >
> > return (TimeValue.ofMilliseconds(100L));
> >
> > }
> >
> >
> >
> >
> >
> > It will be difficult to put traces, we are already in production
> > with
> > this client. Just let me know what you need, and we will do our
> > best
> > to collect the info you need.
> >
> >
> >
> > Thanks,
> >
> > Joan,
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic