[prev in list] [next in list] [prev in thread] [next in thread] 

List:       httpclient-users
Subject:    Re: question about Unhandled IOException in HttpClient.execute with ResponseHandler
From:       Duke DAI <duke.dai.007 () gmail ! com>
Date:       2019-03-21 13:19:39
Message-ID: CAPYoNwfwPLCF14qNMNjeus50hw0bQT0KKKMhsf85qszhmx2F2w () mail ! gmail ! com
[Download RAW message or body]


Finally, it proved to be an issue with glibc, similar with
https://issues.apache.org/jira/browse/HADOOP-7154.
After setting MALLOC_ARENA_MAX=4, native memory consumption got controlled.


Best regards,
Duke
If not now, when? If not me, who?


On Sun, Mar 17, 2019 at 10:02 PM Duke DAI <duke.dai.007@gmail.com> wrote:

> Hi Gurus,
>
> I'm struggling with native memory leak issue on my application. It works
> like a crawler, the only significant thirdparty is httpclient 4.5.6.
> pmap shows there are a lot of "anon" allocations but no hint where they
> come from. "anon" will become more and more along with the application
> running, slowly and gradually. It may be coincident with threads number
> spike.
>
> It's sure it's not a heap issue because gc.log only showed young GCs with
> G1GC, quite good performance with less than 100ms. There is NO mixed GC
> because young GC can collect most of garbages in heap, 98% of heap. I even
> suspected that G1GC young GC will never trigger the garbage collecting
> process in native memory(by DirectByteBuffer). Some articles say only full
> GC can trigger native memory garbage collecting, but they are talking about
> traditional GC algorithms, not G1GC. I'm not sure about the suspect.
>
> I tried below JVM options to limit native memory consumption, but there is
> no help. And jstat indicated there is no problem with Metaspace.
> -XX:MaxMetaspaceSize=512m -Djdk.nio.maxCachedBufferSize=5000000
>
> Before I try -XX:MaxDirectMemorySize(will try next), I inspected all code
> to make sure there is no leak from code level. When I read the code, it
> seems it does not handle IOException thrown by ResponseHandler. If
> IOException(SocketTimeOutException when reading the
> response.getEntity().getContent()) is hit in ResponseHandler, the framework
> only handles ClientProtocolException but not IOException, so the
> InputStream may get leaked there. Response.close will call
> socket.close(socket input/output stream), but I wonder if the intermediate
> DirectByteBuffer will be closed or not.
>
> Can some one shed a light here?
>
> @Override
> public <T> T execute(final HttpHost target, final HttpRequest request,
>         final ResponseHandler<? extends T> responseHandler, final HttpContext context)
>         throws IOException, ClientProtocolException {
>     Args.notNull(responseHandler, "Response handler");
>
>     final CloseableHttpResponse response = execute(target, request, context);
>     try {
>         final T result = responseHandler.handleResponse(response);
>         final HttpEntity entity = response.getEntity();
>         EntityUtils.consume(entity);
>         return result;
>     } catch (final ClientProtocolException t) {
>         // Try to salvage the underlying connection in case of a protocol exception
>         final HttpEntity entity = response.getEntity();
>         try {
>             EntityUtils.consume(entity);
>         } catch (final Exception t2) {
>             // Log this exception. The original exception is more
>             // important and will be thrown to the caller.
>             this.log.warn("Error consuming content after an exception.", t2);
>         }
>         throw t;
>     } finally {
>         response.close();
>     }
> }
>
>
>
>
> Best regards,
> Duke
> If not now, when? If not me, who?
>


[prev in list] [next in list] [prev in thread] [next in thread]