[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: maven-surefire-plugin configuration for Hadoop 2.8.5
From:       Steve Loughran <stevel () cloudera ! com ! INVALID>
Date:       2019-05-02 9:57:56
Message-ID: CAL7CpJz_cKaLgBXjFafBOWEjR2BKqxSXU9yaBKSzJA21P9xEBw () mail ! gmail ! com
[Download RAW message or body]

On Wed, May 1, 2019 at 6:29 PM Elaine Ang <elaineang95@gmail.com> wrote:

> Hi all,
>
> I'm running unit tests for Hadoop 2.8.5. I had a script that basically
> repeatedly run command
>    mvn -Dtest=t1,t2,t3,... test
> with different test selections. And after each test command finishes, I
> need to get test running stats for other purposes.
>
> The problem is that I saw lingering java processes running
> surefirebooterxxxxx.jar that consumes a lot of memory and CPU resources. Is
> this owing to setting  <forkedProcessTimeoutInSeconds> to a timeout value
> under maven-surefire-plugin configuration (in hadoop-project/pom.xml)?
>

the timeout is there to kill hung tests so that jenkins doesn't get blocked.


>
> If so, I'm wondering if there are specific reasons (e.g. required by
> certain unit tests) for using a timeout value and delay the killing of
> forked JVMs, instead of doing something like <shutdown>exit</shutdown>?
> I changed my pom configuration to just using shutdown exit and I didn't see
> lingering java surefirebooter processes anymore, but I'm not sure if this
> change would break any Hadoop unit test.
>
> Looking at the maven docs, the shutdown operation seems to be for handling
the explicit kill of the maven process. I don't see any reason not to add
that, so please file a JIRA and then link to a github PR for the change.

If you really are doing complex test work, then the limitations of the
maven test runner (it's hard-coded generation of the XML DOM test report
originally written for Ant's <junitreport> surface: it builds up the entire
DOM in memory and only saves it when the test suites finish, just so that
the summary xml attributes can be added. You are free to implement your own
JUnit test runner which generates different reports (indeed, I Ant's
<junit> runnier will let you plug in different listeners to build different
reports). If you find yourself waiting for a full surefire test run to
complete before getting the results, know that you have the option not to
do that.

-Steve




> Any thoughts on this? Thanks a lot in advance.
>
> Best,
> Elaine
>

[Attachment #3 (text/html)]

<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Wed, May 1, 2019 at 6:29 PM Elaine Ang &lt;<a \
href="mailto:elaineang95@gmail.com">elaineang95@gmail.com</a>&gt; \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br> <br>
I&#39;m running unit tests for Hadoop 2.8.5. I had a script that basically<br>
repeatedly run command<br>
     mvn -Dtest=t1,t2,t3,... test<br>
with different test selections. And after each test command finishes, I<br>
need to get test running stats for other purposes.<br>
<br>
The problem is that I saw lingering java processes running<br>
surefirebooterxxxxx.jar that consumes a lot of memory and CPU resources. Is<br>
this owing to setting   &lt;forkedProcessTimeoutInSeconds&gt; to a timeout value<br>
under maven-surefire-plugin configuration (in \
hadoop-project/pom.xml)?<br></blockquote><div><br></div><div>the timeout is there to \
kill hung tests so that jenkins doesn&#39;t get blocked.</div><div>  \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> <br>
If so, I&#39;m wondering if there are specific reasons (e.g. required by<br>
certain unit tests) for using a timeout value and delay the killing of<br>
forked JVMs, instead of doing something like \
&lt;shutdown&gt;exit&lt;/shutdown&gt;?<br> I changed my pom configuration to just \
using shutdown exit and I didn&#39;t see<br> lingering java surefirebooter processes \
anymore, but I&#39;m not sure if this<br> change would break any Hadoop unit \
test.<br> <br></blockquote><div>Looking at the maven docs, the shutdown operation \
seems to be for handling the explicit kill of the maven process. I don&#39;t see any \
reason not to add that, so please file a JIRA and then link to a github PR for the \
change.</div><div><br></div><div>If you really are doing complex test work, then the \
limitations of the maven test runner (it&#39;s hard-coded generation of the XML DOM \
test report originally written for Ant&#39;s &lt;junitreport&gt; surface: it builds \
up the entire DOM in memory and only saves it when the test suites finish, just so \
that the summary xml attributes can be added. You are free to implement your own \
JUnit test runner which generates different reports (indeed, I Ant&#39;s \
&lt;junit&gt; runnier will let you plug in different listeners to build different \
reports). If you find yourself waiting for a full surefire test run to complete \
before getting the results, know that you have the option not to do \
that.</div><div><br></div><div>-Steve</div><div><br></div><div><br></div><div>  \
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px \
solid rgb(204,204,204);padding-left:1ex"> Any thoughts on this? Thanks a lot in \
advance.<br> <br>
Best,<br>
Elaine<br>
</blockquote></div></div>



[prev in list] [next in list] [prev in thread] [next in thread]