[prev in list] [next in list] [prev in thread] [next in thread]
List: httpclient-users
Subject: Re: I/O dispatch worker terminated abnormally, what to do?
From: Oleg Kalnichevski <olegk () apache ! org>
Date: 2016-05-31 8:37:26
Message-ID: 1464683846.11902.12.camel () apache ! org
[Download RAW message or body]
On Mon, 2016-05-30 at 21:37 +0000, John Price wrote:
> We're using the HttpAsyncClient to make calls from within our own server
> application. Periodically (once a week or less), our service stops being
> able to make requests and we see errors of the form:
>
> java.lang.IllegalStateException: Request cannot be executed; I/O reactor
> status: STOPPED
> at org.apache.http.util.Asserts.check(Asserts.java:46)
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
> at
> org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
>
> ...
>
> Digging further into the logs, it seems these begin after we see an
> exception that I've included at the bottom but basically seems to boil down
> to "I/O dispatch worker terminated abnormally" caused by "IOException:
> Connection reset by peer" in the depths of SSLIOSession.
>
> I guess I've got two questions: first, is this a bug? I'm not sure how much
> additional info I can provide into the root cause of the IOException since
> it happens so infrequently, but I'll try to provide what I can if it's
> relevant.
>
> Second, how should we be handling this issue? Some of the Apache
> documentation and older mailing list posts talk about "restarting the I/O
> Reactor" but I'm not clear what means in this context since I'm not
> interacting with it directly. Should I just be throwing away my instance of
> HttpAsyncClient and creating a new one?
>
> We're using:
> httpclient 4.5.2
> httpasyncclient 4.1.1
> httpcore.nio4.4.4
>
> John.
>
> org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker
> terminated abnormally
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:358)
>
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
>
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: rx.exceptions.OnErrorNotImplementedException: Connection reset
> by peer
The original 'java.io.IOException: Connection reset by peer' exception
was re-thrown as OnErrorNotImplementedException from inside the
FutureCallback#failed method, which killed the i/o dispatch thread and
caused the i/o reactor to shut down.
This issue needs to be addressed either in your application code or in
the RX library.
Hope this helps
Oleg
> at rx.Observable$27.onError(Observable.java:7923)
> at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
> at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at OurCode.SessionManager.lambda$null$10(SessionManager.java:174)
> at rx.Observable$10.onError(Observable.java:4407)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:65)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.onError(OperatorMerge.java:250)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at rx.apache.http.ObservableHttp$2$1.failed(ObservableHttp.java:195)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
> at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
>
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:152)
>
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:76)
>
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:39)
>
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:127)
>
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
>
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
>
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
>
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
>
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_65]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_65]
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_65]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_65]
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:449)
> ~[antelope-server.jar:na]
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)
> ~[antelope-server.jar:na]
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122)
> ~[antelope-server.jar:na]
> ... 7 common frames omitted
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic